Publications

2024

1. preprint

   General Purpose Verification for Chain of Thought Prompting

   Robert Vacareanu, Anurag Pratik, Evangelia Spiliopoulou, and 6 more authors

   ArXiv, 2024

   preprint

   arXiv
2. preprint

   From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

   Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, and 1 more author

   In , 2024

   preprint

   arXiv
3. LREC-COLING2024

   Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation

   Fahmida Alam, Md Asiful Islam, Robert Vacareanu, and 1 more author

   In Proceedings of the Fourteenth Language Resources and Evaluation Conference , May 2024

   Abs arXiv

   We introduce a meta dataset for few-shot relation extraction, which includes two datasets derived from existing supervised relation extraction datasets – NYT29 (Takanobu et al. , 2019 ; Nayak and Ng , 2020) and WIKIDATA (Sorokin and Gurevych, 2017) – as well as a few-shot form of the TACRED dataset (Sabo et al., 2021). Importantly, all these few-shot datasets were generated under realistic assumptions such as: the test relations are different from any relations a model might have seen before, limited training data, and a preponderance of candidate relation mentions that do not correspond to any of the relations of interest. Using this large resource, we conduct a comprehensive evaluation of six recent few-shot relation extraction methods, and observe that no method comes out as a clear winner. Further, the overall performance on this task is low, indicating substantial need for future research. We release all versions of the data, i.e., both supervised and few-shot, for future research.

4. LREC-COLING2024

   Active Learning Design Choices for NER with Transformers

   Robert Vacareanu, Enrique Noriega-Atala, Gus Hahn-Powell, and 2 more authors

   In Proceedings of the Joint International Conference on Computational Linguistics, Language Resources and Evaluation , May 2024

   LREC-COLING 2024 (Poster)

   Abs

   We explore multiple important choices that have not been analyzed in conjunction regarding active learning for token classification using transformer networks. These choices are: (i) how to select what to annotate, (ii) decide whether to annotate entire sentences or smaller sentence fragments, (iii) how to train with incomplete annotations at token-level, and (iv) how to select the initial seed dataset. We explore whether annotating at sub-sentence level can translate to an improved downstream performance by considering two different sub-sentence annotation strategies: (i) entity-level, and (ii) token-level. These approaches result in some sentences being only partially annotated. To address this issue, we introduce and evaluate multiple strategies to deal with partially-annotated sentences during the training process. We show that annotating at the sub-sentence level achieves comparable or better performance than sentence-level annotations with a smaller number of annotated tokens. We then explore the extent to which the performance gap remains once accounting for the annotation time and found that both annotation schemes perform similarly.

5. Findings of NAACL2024

   Best of Both Worlds: A Pliable and Generalizable Neuro-Symbolic Approach for Relation Classification

   Robert Vacareanu, Fahmida Alam, Md Asiful Islam, and 2 more authors

   In Findings of the Association for Computational Linguistics: NAACL 2024 , Jun 2024

   Findings of NAACL 2024 (Poster)

   Abs arXiv

   This paper introduces a novel neuro-symbolic architecture for relation classification (RC) that combines rule-based methods with contemporary deep learning techniques. This approach capitalizes on the strengths of both paradigms: the adaptability of rule-based systems and the generalization power of neural networks. Our architecture consists of two components: a declarative rule-based model for transparent classification and a neural component to enhance rule generalizability through semantic text matching. Notably, our semantic matcher is trained in an unsupervised domain-agnostic way, solely with synthetic data. Further, these components are loosely coupled, allowing for rule modifications without retraining the semantic matcher. In our evaluation, we focused on two few-shot relation classification datasets: Few-Shot TACRED and a Few-Shot version of NYT29. We show that our proposed method outperforms previous state-of-the-art models in three out of four settings, despite not seeing any human-annotated training data. Further, we show that our approach remains modular and pliable, i.e., the corresponding rules can be locally modified to improve the overall model. Human interventions to the rules for the TACRED relation \textttorg:parents boost the performance on that relation by as much as 26% relative improvement, without negatively impacting the other relations, and without retraining the semantic matching component.

6. EACL2024 (Oral)

   A Weak Supervision Approach for Few-Shot Aspect Based Sentiment Analysis

   Robert Vacareanu, Siddharth Varia, Kishaloy Halder, and 5 more authors

   In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , Mar 2024

   EACL 2024 (Oral)

   Abs arXiv

   We explore how weak supervision on abundant unlabeled data can be leveraged to improve few-shot performance in aspect-based sentiment analysis (ABSA) tasks. We propose a pipeline approach to construct a noisy ABSA dataset, and we use it to adapt a pre-trained sequence-to-sequence model to the ABSA tasks. We test the resulting model on three widely used ABSA datasets, before and after fine-tuning. Our proposed method preserves the full fine-tuning performance while showing significant improvements (15.84 absolute F1) in the few-shot learning scenario for the harder tasks. In zero-shot (i.e., without fine-tuning), our method outperforms the previous state of the art on the aspect extraction sentiment classification (AESC) task and is, additionally, capable of performing the harder aspect sentiment triplet extraction (ASTE) task.

2023

1. WASSA @ ACL2023

   Instruction Tuning for Few-Shot Aspect-Based Sentiment Analysis

   Siddharth Varia, Shuai Wang, Kishaloy Halder, and 7 more authors

   In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis , Jul 2023

   Abs arXiv

   Aspect-based Sentiment Analysis (ABSA) is a fine-grained sentiment analysis task which involves four elements from user-generated texts:aspect term, aspect category, opinion term, and sentiment polarity. Most computational approaches focus on some of the ABSA sub-taskssuch as tuple (aspect term, sentiment polarity) or triplet (aspect term, opinion term, sentiment polarity) extraction using either pipeline or joint modeling approaches. Recently, generative approaches have been proposed to extract all four elements as (one or more) quadrupletsfrom text as a single task. In this work, we take a step further and propose a unified framework for solving ABSA, and the associated sub-tasksto improve the performance in few-shot scenarios. To this end, we fine-tune a T5 model with instructional prompts in a multi-task learning fashion covering all the sub-tasks, as well as the entire quadruple prediction task. In experiments with multiple benchmark datasets, we show that the proposed multi-task prompting approach brings performance boost (by absolute 8.29 F1) in the few-shot learning setting.

2022

1. PaN-DL @ COLING2022

   Neural-Guided Program Synthesis of Information Extraction Rules Using Self-Supervision

   Enrique Noriega-Atala, Robert Vacareanu, Gus Hahn-Powell, and 1 more author

   In Proceedings of the First Workshop on Pattern-based Approaches to NLP in the Age of Deep Learning , Oct 2022

   Abs

   We propose a neural-based approach for rule synthesis designed to help bridge the gap between the interpretability, precision and maintainability exhibited by rule-based information extraction systems with the scalability and convenience of statistical information extraction systems. This is achieved by avoiding placing the burden of learning another specialized language on domain experts and instead asking them to provide a small set of examples in the form of highlighted spans of text. We introduce a transformer-based architecture that drives a rule synthesis system that leverages a self-supervised approach for pre-training a large-scale language model complemented by an analysis of different loss functions and aggregation mechanisms for variable length sequences of user-annotated spans of text. The results are encouraging and point to different desirable properties, such as speed and quality, depending on the choice of loss and aggregation method.

2. PaN-DL @ COLING2022

   PatternRank: Jointly Ranking Patterns and Extractions for Relation Extraction Using Graph-Based Algorithms

   Robert Vacareanu, Dane Bell, and Mihai Surdeanu

   In PANDL , Oct 2022

   PaN-DL Workshop At COLING

   Abs

   In this paper we revisit the direction of using lexico-syntactic patterns for relation extraction instead of today’s ubiquitous neural classifiers. We propose a semi-supervised graph-based algorithm for pattern acquisition that scores patterns and the relations they extract jointly, using a variant of PageRank. We insert light supervision in the form of seed patterns or relations, and model it with several custom teleportation probabilities that bias random-walk scores of patterns/relations based on their proximity to correct information. We evaluate our approach on Few-Shot TACRED, and show that our method outperforms (or performs competitively with) more expensive and opaque deep neural networks. Lastly, we thoroughly compare our proposed approach with the seminal RlogF pattern acquisition algorithm of, showing that it outperforms it for all the hyper parameters tested, in all settings.

3. NAACL2022 (Demo)

   A Human-machine Interface for Few-shot Rule Synthesis for Information Extraction

   Robert Vacareanu, George Caique Gouveia Barbosa, Enrique Noriega-Atala, and 4 more authors

   Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations, Oct 2022

   Abs

   We propose a system that assists a user in constructing transparent information extraction models, consisting of patterns (or rules) written in a declarative language, through program synthesis.Users of our system can specify their requirements through the use of examples,which are collected with a search interface.The rule-synthesis system proposes rule candidates and the results of applying them on a textual corpus; the user has the option to accept the candidate, request another option, or adjust the examples provided to the system.Through an interactive evaluation, we show that our approach generates high-precision rules even in a 1-shot setting. On a second evaluation on a widely-used relation extraction dataset (TACRED), our method generates rules that outperform considerably manually written patterns.Our code, demo, and documentation is available at https://clulab.github.io/odinsynth.

4. LREC2022

   From Examples to Rules: Neural Guided Rule Synthesis for Information Extraction

   Robert Vacareanu, Marco A. Valenzuela-Escárcega, George Barbosa, and 2 more authors

   In Proceedings of the 13th Language Resources and Evaluation Conference (LREC) , Oct 2022

   arXiv

2021

1. Patent

   Methods For Extracting And Assessing Information From Literature Documents

   Mihai Surdeanu, Marco A. Valenzuela Escarcega, Gustave Hahn-Powell, and 7 more authors

   Oct 2021

2020

1. COLING2020

   An Unsupervised Method for Learning Representations of Multi-word Expressions for Semantic Classification

   Robert Vacareanu, Marco A. Valenzuela-Escarcega, Rebecca Sharp, and 1 more author

   In The 28th International Conference on Computational Linguistics in Barcelona (COLING 2020) , Oct 2020
2. LREC2020

   Parsing as Tagging

   Robert Vacareanu, George C. G. Barbosa, Marco A. Valenzuela-Escarcega, and 1 more author

   In Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC) , Oct 2020