publications

2024

1. Analysis of Plan-based Retrieval for Grounded Text Generation

   Ameya Godbole, Nicholas Monath, Seungyeon Kim, Ankit Singh Rawat, Andrew McCallum, and Manzil Zaheer

   In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Nov 2024

   Abs Bib HTML

   In text generation, hallucinations refer to the generation of seemingly coherent text that contradicts established knowledge. One compelling hypothesis is that hallucinations occur when a language model is given a generation task outside its parametric knowledge (due to rarity, recency, domain, etc.). A common strategy to address this limitation is to infuse the language models with retrieval mechanisms, providing the model with relevant knowledge for the task. In this paper, we leverage the planning capabilities of instruction-tuned LLMs and analyze how planning can be used to guide retrieval to further reduce the frequency of hallucinations. We empirically evaluate several variations of our proposed approach on long-form text generation tasks. By improving the coverage of relevant facts, plan-guided retrieval and generation can produce more informative responses while providing a higher rate of attribution to source documents.

   @inproceedings{godbole-etal-2024-analysis,
     title = {Analysis of Plan-based Retrieval for Grounded Text Generation},
     author = {Godbole, Ameya and Monath, Nicholas and Kim, Seungyeon and Rawat, Ankit Singh and McCallum, Andrew and Zaheer, Manzil},
     editor = {Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung},
     booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
     month = nov,
     year = {2024},
     address = {Miami, Florida, USA},
     publisher = {Association for Computational Linguistics},
     pages = {13101--13119},
   }

2023

1. SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples

   Deqing Fu, Ameya Godbole, and Robin Jia

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Dec 2023

   Abs DOI Bib HTML

   Detecting negatives (such as non-entailment relationships, unanswerable questions, and false claims) is an important and challenging aspect of many natural language understanding tasks. Though manually collecting challenging negative examples can help models detect them, it is both costly and domain-specific. In this work, we propose Self-labeled Counterfactuals for Extrapolating to Negative Examples (SCENE), an automatic method for synthesizing training data that greatly improves models’ ability to detect challenging negative examples. In contrast with standard data augmentation, which synthesizes new examples for existing labels, SCENE can synthesize negative examples zero-shot from only positive ones. Given a positive example, SCENE perturbs it with a mask infilling model, then determines whether the resulting example is negative based on a self-training heuristic. With access to only answerable training examples, SCENE can close 69.6% of the performance gap on SQuAD 2.0, a dataset where half of the evaluation examples are unanswerable, compared to a model trained on SQuAD 2.0. Our method also extends to boolean question answering and recognizing textual entailment, and improves generalization from SQuAD to ACE-whQA, an out-of-domain extractive QA benchmark.

   @inproceedings{fu-etal-2023-scene,
     title = {{SCENE}: Self-Labeled Counterfactuals for Extrapolating to Negative Examples},
     author = {Fu, Deqing and Godbole, Ameya and Jia, Robin},
     editor = {Bouamor, Houda and Pino, Juan and Bali, Kalika},
     booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
     month = dec,
     year = {2023},
     address = {Singapore},
     publisher = {Association for Computational Linguistics},
     doi = {10.18653/v1/2023.emnlp-main.485},
     pages = {7832--7848},
   }

2. Benchmarking Long-tail Generalization with Likelihood Splits

   Ameya Godbole, and Robin Jia

   In Findings of the Association for Computational Linguistics: EACL 2023, May 2023

   Abs DOI Bib HTML

   In order to reliably process natural language, NLP systems must generalize to the long tail of rare utterances. We propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets. We create ‘Likelihood Splits’ where examples that are assigned lower likelihood by a pre-trained language model (LM) are placed in the test set, and more likely examples are in the training set. This simple approach can be customized to construct meaningful train-test splits for a wide range of tasks. Likelihood Splits surface more challenges than random splits: relative error rates of state-of-the-art models increase by 59% for semantic parsing on Spider, 93% for natural language inference on SNLI, and 33% for yes/no question answering on BoolQ, on our splits compared with the corresponding random splits. Moreover, Likelihood Splits create fairer benchmarks than adversarial filtering; when the LM used to create the splits is also employed as the task model, our splits do not unfairly penalize the LM.

   @inproceedings{godbole-jia-2023-benchmarking,
     title = {Benchmarking Long-tail Generalization with Likelihood Splits},
     author = {Godbole, Ameya and Jia, Robin},
     editor = {Vlachos, Andreas and Augenstein, Isabelle},
     booktitle = {Findings of the Association for Computational Linguistics: EACL 2023},
     month = may,
     year = {2023},
     address = {Dubrovnik, Croatia},
     publisher = {Association for Computational Linguistics},
     doi = {10.18653/v1/2023.findings-eacl.71},
     pages = {963--983},
   }

2022

1. Knowledge Base Question Answering by Case-based Reasoning over Subgraphs

   Rajarshi Das, Ameya Godbole, Ankita Naik, Elliot Tower, Manzil Zaheer, Hannaneh Hajishirzi, Robin Jia, and Andrew Mccallum

   In Proceedings of the 39th International Conference on Machine Learning, 17–23 jul 2022

   Abs Bib HTML PDF

   Question answering (QA) over knowledge bases (KBs) is challenging because of the diverse, essentially unbounded, types of reasoning patterns needed. However, we hypothesize in a large KB, reasoning patterns required to answer a query type reoccur for various entities in their respective subgraph neighborhoods. Leveraging this structural similarity between local neighborhoods of different subgraphs, we introduce a semiparametric model (CBR-SUBG) with (i) a nonparametric component that for each query, dynamically retrieves other similar k-nearest neighbor (KNN) training queries along with query-specific subgraphs and (ii) a parametric component that is trained to identify the (latent) reasoning patterns from the subgraphs of KNN queries and then apply them to the subgraph of the target query. We also propose an adaptive subgraph collection strategy to select a query-specific compact subgraph, allowing us to scale to full Freebase KB containing billions of facts. We show that CBR-SUBG can answer queries requiring subgraph reasoning patterns and performs competitively with the best models on several KBQA benchmarks. Our subgraph collection strategy also produces more compact subgraphs (e.g. 55% reduction in size for WebQSP while increasing answer recall by 4.85%)\footnoteCode, model, and subgraphs are available at \htmlhttps://github.com/rajarshd/CBR-SUBG.

   @inproceedings{pmlr-v162-das22a,
     title = {Knowledge Base Question Answering by Case-based Reasoning over Subgraphs},
     author = {Das, Rajarshi and Godbole, Ameya and Naik, Ankita and Tower, Elliot and Zaheer, Manzil and Hajishirzi, Hannaneh and Jia, Robin and Mccallum, Andrew},
     booktitle = {Proceedings of the 39th International Conference on Machine Learning},
     pages = {4777--4793},
     year = {2022},
     editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan},
     volume = {162},
     series = {Proceedings of Machine Learning Research},
     month = {17--23 Jul},
     publisher = {PMLR},
   }

2021

1. Case-based Reasoning for Natural Language Queries over Knowledge Bases

   Rajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole, Ethan Perez, Jay Yoon Lee, Lizhen Tan, Lazaros Polymenakos, and Andrew McCallum

   In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Nov 2021

   Abs DOI Bib HTML

   It is often challenging to solve a complex problem from scratch, but much easier if we can access other similar problems with their solutions — a paradigm known as case-based reasoning (CBR). We propose a neuro-symbolic CBR approach (CBR-KBQA) for question answering over large knowledge bases. CBR-KBQA consists of a nonparametric memory that stores cases (question and logical forms) and a parametric model that can generate a logical form for a new question by retrieving cases that are relevant to it. On several KBQA datasets that contain complex questions, CBR-KBQA achieves competitive performance. For example, on the CWQ dataset, CBR-KBQA outperforms the current state of the art by 11% on accuracy. Furthermore, we show that CBR-KBQA is capable of using new cases \textitwithout any further training: by incorporating a few human-labeled examples in the case memory, CBR-KBQA is able to successfully generate logical forms containing unseen KB entities as well as relations.

   @inproceedings{das-etal-2021-case,
     title = {Case-based Reasoning for Natural Language Queries over Knowledge Bases},
     author = {Das, Rajarshi and Zaheer, Manzil and Thai, Dung and Godbole, Ameya and Perez, Ethan and Lee, Jay Yoon and Tan, Lizhen and Polymenakos, Lazaros and McCallum, Andrew},
     editor = {Moens, Marie-Francine and Huang, Xuanjing and Specia, Lucia and Yih, Scott Wen-tau},
     booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
     month = nov,
     year = {2021},
     address = {Online and Punta Cana, Dominican Republic},
     publisher = {Association for Computational Linguistics},
     doi = {10.18653/v1/2021.emnlp-main.755},
     pages = {9594--9611},
   }

2020

1. Probabilistic Case-based Reasoning for Open-World Knowledge Graph Completion

   Rajarshi Das, Ameya Godbole, Nicholas Monath, Manzil Zaheer, and Andrew McCallum

   In Findings of the Association for Computational Linguistics: EMNLP 2020, Nov 2020

   Abs DOI Bib HTML

   A case-based reasoning (CBR) system solves a new problem by retrieving ‘cases’ that are similar to the given problem. If such a system can achieve high accuracy, it is appealing owing to its simplicity, interpretability, and scalability. In this paper, we demonstrate that such a system is achievable for reasoning in knowledge-bases (KBs). Our approach predicts attributes for an entity by gathering reasoning paths from similar entities in the KB. Our probabilistic model estimates the likelihood that a path is effective at answering a query about the given entity. The parameters of our model can be efficiently computed using simple path statistics and require no iterative optimization. Our model is non-parametric, growing dynamically as new entities and relations are added to the KB. On several benchmark datasets our approach significantly outperforms other rule learning approaches and performs comparably to state-of-the-art embedding-based approaches. Furthermore, we demonstrate the effectiveness of our model in an “open-world” setting where new entities arrive in an online fashion, significantly outperforming state-of-the-art approaches and nearly matching the best offline method.

   @inproceedings{das-etal-2020-probabilistic,
     title = {Probabilistic Case-based Reasoning for Open-World Knowledge Graph Completion},
     author = {Das, Rajarshi and Godbole, Ameya and Monath, Nicholas and Zaheer, Manzil and McCallum, Andrew},
     editor = {Cohn, Trevor and He, Yulan and Liu, Yang},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
     month = nov,
     year = {2020},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     doi = {10.18653/v1/2020.findings-emnlp.427},
     pages = {4752--4765},
   }

2. A Simple Approach to Case-Based Reasoning in Knowledge Bases

   Rajarshi Das, Ameya Godbole, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum

   In Automated Knowledge Base Construction, Nov 2020

   Abs Bib HTML

   Consider the task of finding a target entity given a source entity and a binary relation. Our approach finds multiple \textitgraph path patterns that connect similar source entities through the given relation, and looks for pattern matches starting from the query source. Using our method, we obtain new state-of-the-art accuracy, outperforming all previous models, on NELL-995 and FB-122. We also demonstrate that our model is robust in low data settings, outperforming recently proposed meta-learning approaches.

   @inproceedings{das2020a,
     title = {A Simple Approach to Case-Based Reasoning in Knowledge Bases},
     author = {Das, Rajarshi and Godbole, Ameya and Dhuliawala, Shehzaad and Zaheer, Manzil and McCallum, Andrew},
     booktitle = {Automated Knowledge Base Construction},
     year = {2020},
   }

2019

1. Chains-of-Reasoning at TextGraphs 2019 Shared Task: Reasoning over Chains of Facts for Explainable Multi-hop Inference

   Rajarshi Das, Ameya Godbole, Manzil Zaheer, Shehzaad Dhuliawala, and Andrew McCallum

   In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), Nov 2019

   Abs DOI Bib HTML

   This paper describes our submission to the shared task on “Multi-hop Inference Explanation Regeneration” in TextGraphs workshop at EMNLP 2019 (Jansen and Ustalov, 2019). Our system identifies chains of facts relevant to explain an answer to an elementary science examination question. To counter the problem of ‘spurious chains’ leading to ‘semantic drifts’, we train a ranker that uses contextualized representation of facts to score its relevance for explaining an answer to a question. Our system was ranked first w.r.t the mean average precision (MAP) metric outperforming the second best system by 14.95 points.

   @inproceedings{das-etal-2019-chains,
     title = {Chains-of-Reasoning at {T}ext{G}raphs 2019 Shared Task: Reasoning over Chains of Facts for Explainable Multi-hop Inference},
     author = {Das, Rajarshi and Godbole, Ameya and Zaheer, Manzil and Dhuliawala, Shehzaad and McCallum, Andrew},
     editor = {Ustalov, Dmitry and Somasundaran, Swapna and Jansen, Peter and Glava{\v{s}}, Goran and Riedl, Martin and Surdeanu, Mihai and Vazirgiannis, Michalis},
     booktitle = {Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13)},
     month = nov,
     year = {2019},
     address = {Hong Kong},
     publisher = {Association for Computational Linguistics},
     doi = {10.18653/v1/D19-5313},
     pages = {101--117},
   }

2. Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering

   Rajarshi Das, Ameya Godbole, Dilip Kavarthapu, Zhiyu Gong, Abhishek Singhal, Mo Yu, Xiaoxiao Guo, Tian Gao, Hamed Zamani, Manzil Zaheer, and 1 more author

   In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, Nov 2019

   Abs DOI Bib HTML

   Multi-hop question answering (QA) requires an information retrieval (IR) system that can find \textitmultiple supporting evidence needed to answer the question, making the retrieval process very challenging. This paper introduces an IR technique that uses information of entities present in the initially retrieved evidence to learn to ‘\textithop’ to other relevant evidence. In a setting, with more than \textbf5 million Wikipedia paragraphs, our approach leads to significant boost in retrieval performance. The retrieved evidence also increased the performance of an existing QA model (without any training) on the benchmark by \textbf10.59 F1.

   @inproceedings{das-etal-2019-multi,
     title = {Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering},
     author = {Das, Rajarshi and Godbole, Ameya and Kavarthapu, Dilip and Gong, Zhiyu and Singhal, Abhishek and Yu, Mo and Guo, Xiaoxiao and Gao, Tian and Zamani, Hamed and Zaheer, Manzil and McCallum, Andrew},
     editor = {Fisch, Adam and Talmor, Alon and Jia, Robin and Seo, Minjoon and Choi, Eunsol and Chen, Danqi},
     booktitle = {Proceedings of the 2nd Workshop on Machine Reading for Question Answering},
     month = nov,
     year = {2019},
     address = {Hong Kong, China},
     publisher = {Association for Computational Linguistics},
     doi = {10.18653/v1/D19-5816},
     pages = {113--118},
   }

2018

1. Progressively Balanced Multi-class Neural Trees

   Ameya Godbole, Spoorthy Bhat, and Prithwijit Guha

   In 2018 Twenty Fourth National Conference on Communications (NCC), Nov 2018

   Abs DOI Bib HTML

   Decision trees are discriminative classifiers that hierarchically partition the input space to achieve regions containing instances having uniform class label. Existing works in this area have mostly focused on C4.S trees that learn axis aligned partitions. On the other hand, neural trees learn oblique partitions from data and use lesser number of decision nodes hosting perceptrons. However, these perceptrons are susceptible to data imbalances. This motivated us to propose a progressively balanced neural tree where training dataset are balanced prior to perceptron learning. The second contribution is the optimization of the decision function with respect to entropy impurity based objective functions. This formulation also allows a parent node to have more than two child nodes. The proposed algorithm is benchmarked on ten standard datasets against three baseline multi-class classification algorithms.

   @inproceedings{8599945,
     author = {Godbole, Ameya and Bhat, Spoorthy and Guha, Prithwijit},
     booktitle = {2018 Twenty Fourth National Conference on Communications (NCC)},
     title = {Progressively Balanced Multi-class Neural Trees},
     year = {2018},
     volume = {},
     number = {},
     pages = {1-6},
     keywords = {Impurities;Vegetation;Entropy;Training;Optimization;Proposals;Decision trees},
     doi = {10.1109/NCC.2018.8599945},
   }