Enhancing knowledge retrieval with in-context learning and semantic search through generative AI

Retrieving and extracting knowledge from sets of many complex research documents and large databases presents significant challenges in today's information-rich era. Existing retrieval systems, which rely on general-purpose Large Language Models (LLMs), often fail to provide accurate responses to domain-specific inquiries. Additionally, the high cost of pretraining or finetuning LLMs for specific domains limits their adoption. To address those limitations, a novel methodology is proposed that combines the generative capabilities of LLMs with the fast and accurate retrieval capabilities of vector databases. This retrieval system can handle tabular and non-tabular data, understand natural language queries, and retrieve relevant information without finetuning. The developed model, Generative Text Retrieval (GTR), is adaptable to unstructured and structured data with minor refinement. GTR was evaluated on manually annotated and public datasets, achieving more than 90% accuracy and delivering truthful outputs in 87% of cases. The proposed model achieved state-of-the-art performance with a Rouge-L F1 score of 0.98 on the MSMARCO dataset. A refined model, Generative Tabular Text Retrieval (GTR-T), demonstrated its efficiency in large database querying, achieving an Execution Accuracy (EX) of 0.82 and an Exact-Set-Match (EM) accuracy of 0.60 on the Spider dataset, using open-source LLM. Those efforts leverage generative AI and in-context learning to enhance human-text interaction.