publications

2024

1. Analysis of Plan-based Retrieval for Grounded Text Generation

   Ameya Godbole, Nicholas Monath, Seungyeon Kim, and 3 more authors

   2024

2023

1. SCENE: Self-Labeled Counterfactuals for Extrapolating to Negative Examples

   Deqing Fu, Ameya Godbole, and Robin Jia

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Dec 2023

   Abs DOI

   Detecting negatives (such as non-entailment relationships, unanswerable questions, and false claims) is an important and challenging aspect of many natural language understanding tasks. Though manually collecting challenging negative examples can help models detect them, it is both costly and domain-specific. In this work, we propose Self-labeled Counterfactuals for Extrapolating to Negative Examples (SCENE), an automatic method for synthesizing training data that greatly improves models’ ability to detect challenging negative examples. In contrast with standard data augmentation, which synthesizes new examples for existing labels, SCENE can synthesize negative examples zero-shot from only positive ones. Given a positive example, SCENE perturbs it with a mask infilling model, then determines whether the resulting example is negative based on a self-training heuristic. With access to only answerable training examples, SCENE can close 69.6% of the performance gap on SQuAD 2.0, a dataset where half of the evaluation examples are unanswerable, compared to a model trained on SQuAD 2.0. Our method also extends to boolean question answering and recognizing textual entailment, and improves generalization from SQuAD to ACE-whQA, an out-of-domain extractive QA benchmark.

2. Benchmarking Long-tail Generalization with Likelihood Splits

   Ameya Godbole, and Robin Jia

   In Findings of the Association for Computational Linguistics: EACL 2023, May 2023

   Abs DOI

   In order to reliably process natural language, NLP systems must generalize to the long tail of rare utterances. We propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets. We create ‘Likelihood Splits’ where examples that are assigned lower likelihood by a pre-trained language model (LM) are placed in the test set, and more likely examples are in the training set. This simple approach can be customized to construct meaningful train-test splits for a wide range of tasks. Likelihood Splits surface more challenges than random splits: relative error rates of state-of-the-art models increase by 59% for semantic parsing on Spider, 93% for natural language inference on SNLI, and 33% for yes/no question answering on BoolQ, on our splits compared with the corresponding random splits. Moreover, Likelihood Splits create fairer benchmarks than adversarial filtering; when the LM used to create the splits is also employed as the task model, our splits do not unfairly penalize the LM.

2022

1. Knowledge Base Question Answering by Case-based Reasoning over Subgraphs

   Rajarshi Das, Ameya Godbole, Ankita Naik, and 5 more authors

   In Proceedings of the 39th International Conference on Machine Learning, 17–23 jul 2022

   Abs PDF

   Question answering (QA) over knowledge bases (KBs) is challenging because of the diverse, essentially unbounded, types of reasoning patterns needed. However, we hypothesize in a large KB, reasoning patterns required to answer a query type reoccur for various entities in their respective subgraph neighborhoods. Leveraging this structural similarity between local neighborhoods of different subgraphs, we introduce a semiparametric model (CBR-SUBG) with (i) a nonparametric component that for each query, dynamically retrieves other similar k-nearest neighbor (KNN) training queries along with query-specific subgraphs and (ii) a parametric component that is trained to identify the (latent) reasoning patterns from the subgraphs of KNN queries and then apply them to the subgraph of the target query. We also propose an adaptive subgraph collection strategy to select a query-specific compact subgraph, allowing us to scale to full Freebase KB containing billions of facts. We show that CBR-SUBG can answer queries requiring subgraph reasoning patterns and performs competitively with the best models on several KBQA benchmarks. Our subgraph collection strategy also produces more compact subgraphs (e.g. 55% reduction in size for WebQSP while increasing answer recall by 4.85%)\footnoteCode, model, and subgraphs are available at \urlhttps://github.com/rajarshd/CBR-SUBG.

2021

1. Case-based Reasoning for Natural Language Queries over Knowledge Bases

   Rajarshi Das, Manzil Zaheer, Dung Thai, and 6 more authors

   In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Nov 2021

   Abs DOI

   It is often challenging to solve a complex problem from scratch, but much easier if we can access other similar problems with their solutions — a paradigm known as case-based reasoning (CBR). We propose a neuro-symbolic CBR approach (CBR-KBQA) for question answering over large knowledge bases. CBR-KBQA consists of a nonparametric memory that stores cases (question and logical forms) and a parametric model that can generate a logical form for a new question by retrieving cases that are relevant to it. On several KBQA datasets that contain complex questions, CBR-KBQA achieves competitive performance. For example, on the CWQ dataset, CBR-KBQA outperforms the current state of the art by 11% on accuracy. Furthermore, we show that CBR-KBQA is capable of using new cases \textitwithout any further training: by incorporating a few human-labeled examples in the case memory, CBR-KBQA is able to successfully generate logical forms containing unseen KB entities as well as relations.

2020

1. Probabilistic Case-based Reasoning for Open-World Knowledge Graph Completion

   Rajarshi Das, Ameya Godbole, Nicholas Monath, and 2 more authors

   In Findings of the Association for Computational Linguistics: EMNLP 2020, Nov 2020

   Abs DOI

   A case-based reasoning (CBR) system solves a new problem by retrieving ‘cases’ that are similar to the given problem. If such a system can achieve high accuracy, it is appealing owing to its simplicity, interpretability, and scalability. In this paper, we demonstrate that such a system is achievable for reasoning in knowledge-bases (KBs). Our approach predicts attributes for an entity by gathering reasoning paths from similar entities in the KB. Our probabilistic model estimates the likelihood that a path is effective at answering a query about the given entity. The parameters of our model can be efficiently computed using simple path statistics and require no iterative optimization. Our model is non-parametric, growing dynamically as new entities and relations are added to the KB. On several benchmark datasets our approach significantly outperforms other rule learning approaches and performs comparably to state-of-the-art embedding-based approaches. Furthermore, we demonstrate the effectiveness of our model in an “open-world” setting where new entities arrive in an online fashion, significantly outperforming state-of-the-art approaches and nearly matching the best offline method.

2. A Simple Approach to Case-Based Reasoning in Knowledge Bases

   Rajarshi Das, Ameya Godbole, Shehzaad Dhuliawala, and 2 more authors

   In Automated Knowledge Base Construction, Nov 2020

2019

1. Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering

   Rajarshi Das, Ameya Godbole, Dilip Kavarthapu, and 8 more authors

   In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, Nov 2019

   Abs DOI

   Multi-hop question answering (QA) requires an information retrieval (IR) system that can find \textitmultiple supporting evidence needed to answer the question, making the retrieval process very challenging. This paper introduces an IR technique that uses information of entities present in the initially retrieved evidence to learn to ‘\textithop’ to other relevant evidence. In a setting, with more than \textbf5 million Wikipedia paragraphs, our approach leads to significant boost in retrieval performance. The retrieved evidence also increased the performance of an existing QA model (without any training) on the benchmark by \textbf10.59 F1.

2. Chains-of-Reasoning at TextGraphs 2019 Shared Task: Reasoning over Chains of Facts for Explainable Multi-hop Inference

   Rajarshi Das, Ameya Godbole, Manzil Zaheer, and 2 more authors

   In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-13), Nov 2019

   Abs DOI

   This paper describes our submission to the shared task on “Multi-hop Inference Explanation Regeneration” in TextGraphs workshop at EMNLP 2019 (Jansen and Ustalov, 2019). Our system identifies chains of facts relevant to explain an answer to an elementary science examination question. To counter the problem of ‘spurious chains’ leading to ‘semantic drifts’, we train a ranker that uses contextualized representation of facts to score its relevance for explaining an answer to a question. Our system was ranked first w.r.t the mean average precision (MAP) metric outperforming the second best system by 14.95 points.

2018

1. Progressively Balanced Multi-class Neural Trees

   Ameya Godbole, Spoorthy Bhat, and Prithwijit Guha

   In 2018 Twenty Fourth National Conference on Communications (NCC), Nov 2018

   DOI