Ameya Godbole

I am a fourth year PhD student in Computer Science at the University of Southern California advised by Prof. Robin Jia. I work on hallucination-free and faithful generation with retrieval augmentation. I worked on leveraging the planning capabilities of LLMs to guide retrieval. I also study the behavior of factuality detection systems and their reliability in model selection. In the past, I have worked on retrieval for open-domain question answering (ODQA), knowledge base completion (KBC), and semantic parsing for knowledge base QA (KBQA).

Previously, I was a Research Fellow with the wonderful Information Extraction and Synthesis Lab (IESL) at the University of Massachusetts, Amherst. In addition to research, I worked on developing and testing features for the OpenReview platform.

news

Oct 22, 2024	I will be presenting Analysis of Plan-based Retrieval for Grounded Text Generation at EMNLP 2024! See you in Miami!

selected publications

Analysis of Plan-based Retrieval for Grounded Text Generation

Ameya Godbole, Nicholas Monath, Seungyeon Kim, and 3 more authors

2024
Benchmarking Long-tail Generalization with Likelihood Splits

Ameya Godbole, and Robin Jia

In Findings of the Association for Computational Linguistics: EACL 2023, May 2023

Abs DOI

In order to reliably process natural language, NLP systems must generalize to the long tail of rare utterances. We propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets. We create ‘Likelihood Splits’ where examples that are assigned lower likelihood by a pre-trained language model (LM) are placed in the test set, and more likely examples are in the training set. This simple approach can be customized to construct meaningful train-test splits for a wide range of tasks. Likelihood Splits surface more challenges than random splits: relative error rates of state-of-the-art models increase by 59% for semantic parsing on Spider, 93% for natural language inference on SNLI, and 33% for yes/no question answering on BoolQ, on our splits compared with the corresponding random splits. Moreover, Likelihood Splits create fairer benchmarks than adversarial filtering; when the LM used to create the splits is also employed as the task model, our splits do not unfairly penalize the LM.
Knowledge Base Question Answering by Case-based Reasoning over Subgraphs

Rajarshi Das, Ameya Godbole, Ankita Naik, and 5 more authors

In Proceedings of the 39th International Conference on Machine Learning, 17–23 jul 2022

Abs PDF

Question answering (QA) over knowledge bases (KBs) is challenging because of the diverse, essentially unbounded, types of reasoning patterns needed. However, we hypothesize in a large KB, reasoning patterns required to answer a query type reoccur for various entities in their respective subgraph neighborhoods. Leveraging this structural similarity between local neighborhoods of different subgraphs, we introduce a semiparametric model (CBR-SUBG) with (i) a nonparametric component that for each query, dynamically retrieves other similar k-nearest neighbor (KNN) training queries along with query-specific subgraphs and (ii) a parametric component that is trained to identify the (latent) reasoning patterns from the subgraphs of KNN queries and then apply them to the subgraph of the target query. We also propose an adaptive subgraph collection strategy to select a query-specific compact subgraph, allowing us to scale to full Freebase KB containing billions of facts. We show that CBR-SUBG can answer queries requiring subgraph reasoning patterns and performs competitively with the best models on several KBQA benchmarks. Our subgraph collection strategy also produces more compact subgraphs (e.g. 55% reduction in size for WebQSP while increasing answer recall by 4.85%)\footnoteCode, model, and subgraphs are available at \urlhttps://github.com/rajarshd/CBR-SUBG.
Multi-step Entity-centric Information Retrieval for Multi-Hop Question Answering

Rajarshi Das, Ameya Godbole, Dilip Kavarthapu, and 8 more authors

In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, Nov 2019

Abs DOI

Multi-hop question answering (QA) requires an information retrieval (IR) system that can find \textitmultiple supporting evidence needed to answer the question, making the retrieval process very challenging. This paper introduces an IR technique that uses information of entities present in the initially retrieved evidence to learn to ‘\textithop’ to other relevant evidence. In a setting, with more than \textbf5 million Wikipedia paragraphs, our approach leads to significant boost in retrieval performance. The retrieved evidence also increased the performance of an existing QA model (without any training) on the benchmark by \textbf10.59 F1.