Raqag: a framework for automatically generating Q&A datasets with retrieval-augmented generation

Authors: Bin Gao, Chen Yu, Qi Zhang, Chunwei Lou, Luhan Wang, Jin Xu, Dajiang Chen

Year: 2026

Journal / Conference: Knowledge-Based Systems

Paper Link: https://www.sciencedirect.com/science/article/abs/pii/S095070512600568X

GitHub: https://github.com/Fakerfigure/QAG_System.git

Abstract:
Objectives: Large Language Models (LLMs) necessitate high-quality Question-Answer (QA) datasets for fine-tuning and evaluation, but current resources are constrained by limited scale, diversity, and structured annotations. This scarcity is especially pronounced in business and management research (e.g., UTD24 journal domains like accounting, finance, marketing, strategy, and operations), where domain-specific QA datasets are rare beyond finance-focused ones. Methods: To bridge this gap, we present RAQAG, a Retrieval-Augmented Question Answer Generation framework that constructs QA datasets through a closed-loop process involving data synthesis, quality verification, and feedback optimization. RAQAG consists of three core modules: data preprocessing, QA generation, and QA evaluation. The QA generation module utilizes a dynamic retrieval-augmented generation approach, incorporating a three-tier retrieval strategy (semantic vector search, BM25 retrieval, and BGE Reranker) to support dynamic paragraph segmentation and enhance document grounding. The QA evaluation module employs RACAR, a multi-dimensional evaluation framework that rigorously defines five core dimensions: Relevance, Agnosticism, Completeness, Accuracy, and Reasonableness, along with dynamic indicators like evidence coverage. Novelty: RAQAG presents a novel framework that integrates LLMs with a closed-loop process for generating high-quality QA datasets. RAQAG enables the creation of large-scale, domain-specific QA corpora, e.g., UTD24 _ QA, which includes 170,226 instruction-style QA pairs derived from 17,542 articles. A standardized multiple-choice benchmark, UTD24 _ QA Benchmark, is developed for systematic evaluation of LLMs in business and management knowledge. Findings: Experiments demonstrate that RAQAG produces higher-quality QA datasets, achieving a 9.45% improvement in RACAR scores compared to the best existing baseline. The framework exhibits robust performance across varying input granularities and hyperparameter settings. The engineering-oriented QAG system, featuring metadata-driven orchestration and visualized pipelines, further validates the effectiveness of RAQAG.