Report for ahmedrachid/FinancialBERT-Sentiment-Analysis

#84
by giskard-bot - opened

Hi Team,

This is a report from Giskard Bot Scan 🐢.

We have identified 3 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).

👉Robustness issues (3)
Vulnerability Level Data slice Metric Transformation Deviation
Robustness major 🔴 Fail rate = 0.416 Transform to title case 416/1000 tested samples (41.6%) changed prediction after perturbation
🔍✨Examples When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 41.6% of the cases. We expected the predictions not to be affected by this transformation.
text Transform to title case(text) Original prediction Prediction after perturbation
408 Ragutis , which is based in Lithuania 's second-largest city Kaunas , boosted its sales last year 22.3 per cent to 36.4 million liters . Ragutis , Which Is Based In Lithuania 'S Second-Largest City Kaunas , Boosted Its Sales Last Year 22.3 Per Cent To 36.4 Million Liters . positive (p = 1.00) neutral (p = 1.00)
785 The agreement must be approved by the Russian competition authorities before it enters into force . The Agreement Must Be Approved By The Russian Competition Authorities Before It Enters Into Force . positive (p = 0.98) neutral (p = 1.00)
117 Finnish department store chain Stockmann Oyj Abp net profit rose to 39.8 mln euro ( $ 56.8 mln ) for the first nine months of 2007 from 37.4 mln euro ( $ 53.4 mln ) for the same period of 2006 . Finnish Department Store Chain Stockmann Oyj Abp Net Profit Rose To 39.8 Mln Euro ( $ 56.8 Mln ) For The First Nine Months Of 2007 From 37.4 Mln Euro ( $ 53.4 Mln ) For The Same Period Of 2006 . positive (p = 1.00) neutral (p = 1.00)
Vulnerability Level Data slice Metric Transformation Deviation
Robustness major 🔴 Fail rate = 0.408 Transform to uppercase 408/1000 tested samples (40.8%) changed prediction after perturbation
🔍✨Examples When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 40.8% of the cases. We expected the predictions not to be affected by this transformation.
text Transform to uppercase(text) Original prediction Prediction after perturbation
785 The agreement must be approved by the Russian competition authorities before it enters into force . THE AGREEMENT MUST BE APPROVED BY THE RUSSIAN COMPETITION AUTHORITIES BEFORE IT ENTERS INTO FORCE . positive (p = 0.98) neutral (p = 0.99)
118 Finnish lifting equipment maker Konecranes Oyj said on July 30 , 2008 that its net profit rose to 71.2 mln euro ( $ 111.1 mln ) for the first half of 2008 from 57.1 mln euro ( $ 89.1 mln ) for the same period of 2007 . FINNISH LIFTING EQUIPMENT MAKER KONECRANES OYJ SAID ON JULY 30 , 2008 THAT ITS NET PROFIT ROSE TO 71.2 MLN EURO ( $ 111.1 MLN ) FOR THE FIRST HALF OF 2008 FROM 57.1 MLN EURO ( $ 89.1 MLN ) FOR THE SAME PERIOD OF 2007 . positive (p = 1.00) neutral (p = 0.99)
875 Net sales will , however , increase from 2005 . NET SALES WILL , HOWEVER , INCREASE FROM 2005 . positive (p = 1.00) neutral (p = 1.00)
Vulnerability Level Data slice Metric Transformation Deviation
Robustness medium 🟡 Fail rate = 0.071 Add typos 71/1000 tested samples (7.1%) changed prediction after perturbation
🔍✨Examples When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 7.1% of the cases. We expected the predictions not to be affected by this transformation.
text Add typos(text) Original prediction Prediction after perturbation
1919 Cash flow from operations totalled EUR 7.4 mn , compared to a negative EUR 68.6 mn in the second quarter of 2008 . Cash dlow from operations totalled WUR 7.4 mhn , comlared to a negative EUR 68.6 mn in the second wquarter of 2008 . positive (p = 1.00) negative (p = 0.99)
2217 Talentum 's net sales in September were smaller than expected . Talentum ' net sales in September were skaler tham expeted . negative (p = 0.99) neutral (p = 1.00)
235 Olvi , which controls a 94 percent stake in Ragutis through A. Le Coq , said in its annual report published earlier this year that the Lithuanian brewery 's sales reached 15.04 million euros last year , a rise of 20.4 percent from 12.49 million euros in 2004 . Olvi , which controls a 94 percent stake in Raygutis through A. Le Coq , said in its annual report published earlier tbhis year taht the Lithuanian brewery 's sales reached 15.04 mlilion euros laxt year , a ise fo 20.4 percent from 12.49 milklion euros in 2004 . positive (p = 1.00) neutral (p = 0.89)

Checkout out the Giskard Space and test your model.

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.

Sign up or log in to comment