Contradiction Detection in Financial Reports
DOI:
https://doi.org/10.7557/18.6799Keywords:
contradiction detection, natural language processing, text mining, financial reports, deep learningAbstract
Finding and amending contradictions in a financial report is crucial for the publishing company and its financial auditors. To automate this process, we introduce a novel approach that incorporates informed pre-training into its transformer-based architecture to infuse this model with additional Part-Of-Speech knowledge. Furthermore, we fine-tune the model on the public Stanford Natural Language Inference Corpus and our proprietary financial contradiction dataset. It achieves an exceptional contradiction detection F1 score of 89.55% on our real-world financial contradiction dataset, beating our several baselines by a considerable margin. During the model selection process we also test various financial-document-specific transformer models and find that they underperform the more general embedding approaches.
References
D. Araci. Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063, 2019. doi: 10.48550/arXiv.1908.10063
D. Biesner, R. Ramamurthy, R. Stenzel, M. Lübbering, L. Hillebrand, A. Ladi, M. Pielka, R. Stenzel, R. Loitz, C. Bauckhage, and R. Sifa. Anonymization of german financial documents using neural network-based language models with contextual word representations. Springer International Journal of Data Science and Analytics, 2021. doi: 10.1007/s41060-021-00285-x
S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning. A large annotated corpus for learning natural language inference. In Proc. EMNLP, 2015. doi: 10.18653/v1/D15-1075
Y. Cao, H. Li, P. Luo, and J. Yao. Towards automatic numerical cross-checking: Extracting formulas from text. In Proc. WWW, 2018. doi: 10.1145/3178876.3186166
C. L. Chapman, L. Hillebrand, M. R. Stenzel, T. Deusser, D. Biesner, C. Bauckhage, and R. Sifa. Towards generating financial reports from tabular data using transformers. In Proc. CD-MAKE, pages 221–232. Springer, 2022. doi: 10.1007/978-3-031-14463-9 14
A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzm ́an, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proc. ACL, 2020. doi: 10.18653/v1/2020.acl-main.747
M.-C. De Marneffe, A. N. Rafferty, and C. D. Manning. Finding contradictions in text. In Proc. ACL-HLT, pages 1039–1047, 2008
T. Deußer, S. M. Ali, L. Hillebrand, D. Nurchalifah, B. Jacob, C. Bauckhage, and R. Sifa. KPI-EDGAR: A novel dataset and accompanying metric for relation extraction from financial documents. In Proc. ICMLA, 2022. doi: 10.48550/arXiv.2210.09163
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT, 2019. doi: 10.18653/v1/N19-1423
D. Gordeev, A. Davletov, A. Rey, and N. Arefiev. LIORI at the FinCausal 2020 shared task. In Proc. FNP, pages 45–49, 2020
S. Harabagiu, A. Hickl, and F. Lacatusu. Negation, contrast and contradiction in text processing. In Proc. AAAI, volume 6, pages 755–762, 2006
A. Hazourli. FinancialBERT - a pretrained language model for financial text mining. 2022. doi: 10.13140/RG.2.2.34032.12803
L. Hillebrand, T. Deußer, T. Dilmaghani, B. Kliem, R. Loitz, C. Bauckhage, and R. Sifa. Towards automating numerical consistency checks in financial reports. In Proc. BigData, 2022. doi: 10.48550/arXiv.2211.06112
L. Hillebrand, T. Deußer, T. Dilmaghani, B. Kliem, R. Loitz, C. Bauckhage, and R. Sifa. KPI-BERT: A joint named entity recognition and relation extraction model for financial reports. In Proc. ICPR, pages 606–612, 2022. doi: 10.1109/ICPR56361.2022.9956191
M. Honnibal, I. Montani, S. Van Landeghem, and A. Boyd. spacy: Industrial-strength natural language processing in python. 2020
C. Hsu, C.-T. Li, D. Saez-Trumper, and Y.-Z. Hsu. WikiContradiction: Detecting self-contradiction articles on wikipedia. In Proc. Big Data, pages 427–436. IEEE, 2021. doi: 10.1109/BigData52589.2021.9671319
D. Jin, S. Liu, Y. Liu, and D. Hakkani-Tur. Improving bot response contradiction detection via utterance rewriting. arXiv preprint arXiv:2207.11862, 2022. doi: 10.48550/arXiv.2207.11862
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692, 2019. doi: 10.48550/arXiv.1907.11692
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In Proc. ICLR, 2019. doi: arXiv.1711.05101
P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. J. of the Association for Information Science and Technology, 65(4):782–796, 2014. doi: 10.1002/asi.23062
M. Q. N. Pham, M. Le Nguyen, and A. Shimazu. Using shallow semantic parsing and relation extraction for finding contradiction in text. In Proc. ACL-IJCNLP, pages 1017–1021, 2013.
M. Pielka, R. Sifa, L. P. Hillebrand, D. Biesner, R. Ramamurthy, A. Ladi, and C. Bauckhage. Tackling contradiction detection in german using machine translation and end-to-end recurrent neural networks. In Proc. ICPR, pages 6696–6701, 2021. doi: 10.1109/ICPR48806.2021.9413257
M. Pielka, S. Schmidt, L. Pucknat, and R. Sifa. Towards linguistically informed multi-objective pre-training for natural language inference. In Proc ECIR (in press), 2023. doi: 10.48550/arXiv.2212.07428
L. Pucknat, M. Pielka, and R. Sifa. Detecting contradictions in german text: A comparative study. In Proc. SSCI, pages 01–07, 2021. doi: 10.1109/SSCI50451.2021.9659881
A. Radford and K. Narasimhan. Improving language understanding by generative pre-training. 2018. URL https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. In OpenAI Blog, Accessed: 2022-08-29, 2019. URL https://openai.com/blog/better-language-models
Z. Rahimi and M. ShamsFard. Contradiction detection in persian text. arXiv preprint arXiv:2107.01987, 2021. doi: 10.48550/ARXIV.2107.01987
R. Ramamurthy, M. Lübbering, T. Bell, M. Gebauer, B. Ulusay, D. Uedelhoven, T. D. Khameneh, R. Loitz, M. Pielka, C. Bauckhage, and R. Sifa. Automatic indexing of financial documents via information extraction. In Proc. SSCI, pages 01–05, 2021. doi: 10.1109/SSCI50451.2021.9659977
R. Ramamurthy, M. Pielka, R. Stenzel, C. Bauckhage, R. Sifa, T. D. Khameneh, U. Warning, B. Kliem, and R. Loitz. ALiBERT: improved automated list inspection (ALI) with BERT. In Proc. DocEng, pages 1–4, 2021. doi: 10.1145/3469096.3474928
K. Russo. What are the risks of inaccurate financial reporting?, March 2022. URL https://www.netsuite.com/portal/resource/articles/accounting/inaccurate-financial-reporting.shtml. [Online; posted 21/03/2022; retrieved 22/08/2022]
R. Sep ́ulveda-Torres, A. Bonet-Jover, and E. Saquete. “Here are the rules: Ignore all rules”: Automatic contradiction detection in spanish. Applied Sciences, 11(7):3060, 2021. doi: 10.3390/app11073060
R. Sifa, A. Ladi, M. Pielka, R. Ramamurthy, L. Hillebrand, B. Kirsch, D. Biesner, R. Stenzel, T. Bell, M. Lübbering, et al. Towards automated auditing with machine learning. In Proc. DocEng, 2019. doi: 10.1145/3342558.3345421
R. Sifa, M. Pielka, R. Ramamurthy, A. Ladi, L. Hillebrand, and C. Bauckhage. Towards contradiction detection in german: a translation-driven approach. In Proc. SSCI, pages 2497–2505. IEEE, 2019. doi: 10.1109/SSCI44817.2019.9003090
Y. Takabatake, H. Morita, D. Kawahara, S. Kurohashi, R. Higashinaka, and Y. Matsuo. Classification and acquisition of contradictory event pairs using crowdsourcing. In Proc. Workshop on EVENTS at NAACL-HLT, pages 99–107, 2015. doi: 10.3115/v1/W15-0813
N. S. Tawfik and M. R. Spruit. Automated contradiction detection in biomedical literature. In Proc. MLDM, pages 138–148. Springer, 2018. doi: 10.1007/978-3-319-96136-1 12
F. Zhu, D. Ning, Y. Wang, and S. Liu. A novel cost-sensitive capsule network for audit fraud detection. In Proc. IUCC, pages 549–556, 2021. doi: 10.1109/IUCC-CIT-DSCI-SmartCNS55181.2021.00091
Downloads
Published
Issue
Section
License
Copyright (c) 2023 Tobias Deußer, Maren Pielka, Lisa Pucknat, Basil Jacob, Tim Dilmaghani, Mahdis Nourimand, Bernd Kliem, Rüdiger Loitz, Christian Bauckhage, Rafet Sifa
This work is licensed under a Creative Commons Attribution 4.0 International License.
