New developments in deep learning for natural language processing and explanation

Abstract

This PhD thesis introduces LamBERTa, a pioneering BERT-based language understanding framework tailored for law article retrieval as a prediction task. LamBERTa addresses the challenges posed by a multi-class classification scenario featuring hundreds or thousands of classes and minimal per-class training instances, generated in an unsupervised manner. The study explores domain-adaptation coupled with task-adaptation to train LamBERTa models for the Italian Civil Code (ICC) article prediction, incorporating a recently defined Italian legal BERT and investigating the impact of enhancing the tokenizer with terms from the target legal corpus. The work’s contributions extend to the exploration of LamBERTa models’ explainability, employing model-level and instance-level explainability techniques. While initially focused on the Italian Civil Code, the LamBERTa architecture proves adaptable to other law code corpora such as the General Data Protection Regulation (GDPR), further powered by selfsupervised task-adaptive pre-training stages, with or without data enrichment based on recitals, marking the first comprehensive study of domain-adaptive and task-adaptive legal pre-trained language models for GDPR article retrieval. Furthermore, the thesis conducts a network analysis and mining study, presenting the first examination of citation networks inferred from ICC articles. Insights into structural features reveal hidden patterns, facilitating new interpretations and studies of the ICC. This methodology is adaptable to other civil law code systems organized similarly to the ICC, supporting tasks like statute law retrieval, entailment, and question answering. The study also introduces LawNet-Viz, a web-based tool for modeling, analyzing, and visualizing law reference networks, aiming to assist legal experts and citizens in legal information processing and understanding. Finally, the thesis outlines the TrustSearch project’s ambitious goal to develop an AI tool that enhances the search experience promoting critical thinking. Two stance detection approaches, Definition-based and User-driven, are presented, showcasing the effectiveness of AI in transforming the search and data discovery experience. In particular, Chapter 1 introduces the motivations behind this research work, then it overviews recent works that address legal classification and retrieval problems based on deep learning methods and network analysis techniques as well as entailment resolution using Transformers and LLMs. Chapter 2 provides the reader with essential information about the research topic and it serves to establish the context around The Italian Civil Code (ICC) and The General Data Protection Regulation (GDPR). Chapter 3 presents our proposed framework for the civil-law article retrieval problem named as LamBERTa with the application on the ICC corpus. Within it, we investigate the performance of the models through an extensive quantitative and qualitative set of experiments. Furthermore, we study the effects of domain adaptive pre-training strategies and task adaptive training strategies and we scrutinize the concept of Explainability leveraging on the use of different techniques. Chapter 4 presents the LamBERTa models applied on the General Data Protection Regulation (GDPR) corpus, also powering them through self-supervised task-adaptive pre-training schemes. Chapter 5 explores the Italian Civil Code (ICC) from an unprecedented perspective based on network analysis. Furthermore, we present LawNet-Viz, a web-based tool for the modeling, analysis and visualization of law reference networks extracted from a statute law corpus. Chapter 6 is based on the research I carried out during the PhD visiting period abroad at (UPV) University Polytechnic of Valencia, Spain, where I actively participated in the TrustSearch project, a research project funded by the European Union, developing an AI tool that offers a new search experience to promote critical thinking. In this regard, we developed two different approaches, the first one which is based on the state of art LLMs prompt engineer, the second one consists is an application of an encoder-decoder Transformer-based model for the stance detection leveraging on the entailment task. Chapter 7 summarizes findings and discusses future studies which could extend the research.