XRAG 中文解读
XRAG logoXRAG

ICDE 2026 · Experiment, Analysis, and Benchmark

eXamining the Core.

Benchmarking Foundational Components
in Advanced Retrieval-Augmented Generation

Qili Zhang1Qianren Mao2Yangyifei Luo1Yashuo Luo1Hanwen Hao1Zhilong Cao1Weifeng Jiang3Zhijun Chen1Junnan Liu1Feng Yan1Xiaolong Wang1Jinlong Zhang1Zhenting Huang1Zhixing Tan2Jie Sun2Bo Li1Jianxin Li1Philip S. Yu4
1Beihang University2Zhongguancun Laboratory3Nanyang Technological University4University of Illinois at Chicago
Read the paper Code arXiv 中文解读

Understand the components behind the answer.
Compare modules. Compose workflows. Evaluate retrieval and generation together.

01 / The idea

Make RAG performance explainable.

XRAG opens up the pipeline so researchers can examine which components, workflows, and data characteristics shape the final answer.

Abstract

Retrieval-augmented generation (RAG) synergizes the retrieval of pertinent database with the generative capabilities of Large Language Models (LLMs), ensuring that the generated output is contextually relevant but also accurate and current. We introduce XRAG, an open-source, modular codebase that facilitates exhaustive evaluation of the performance of foundational components of advanced RAG modules.

These components are categorized into four core phases: pre-retrieval, retrieval, post-retrieval, and generation. We systematically analyze them across reconfigured datasets, providing a comprehensive benchmark for their effectiveness. These components can be combined in different ways through an orchestrator, of which there are five types: sequential, conditional, iterative, parallel, and hybrid. Our work thoroughly evaluates the performance of advanced core components in RAG systems, offering actionable insights for optimizing RAG architectures in dataset-driven AI applications.

4

Core stages

From query preparation
to answer generation

5

Orchestrators

Sequential, conditional, iterative,
parallel, and hybrid

40

Evaluation metrics

7 retrieval · 10 generation
23 LLM-based metrics

Four unified benchmarks: HotpotQA, DropQA, NaturalQA, and FinanceBench. Coverage and metric counts follow Tables I–II in the paper.

02 / The framework

Open up the RAG pipeline.

A final answer score cannot tell you whether the bottleneck lies in retrieval, context processing, or generation. XRAG makes each stage available for controlled comparison.

XRAG framework: unified datasets support four component stages, five orchestrators, and three families of evaluators.
01

Pre-retrieval

Prepare the query

Examine query transformations such as Step-back Prompting and HyDE before searching the corpus.

02

Retrieval

Find the evidence

Compare BGE and JINA retrievers with methods such as reciprocal rank fusion and sentence-window retrieval.

03

Post-retrieval

Refine the context

Assess BGE and JINA rerankers to prioritize useful evidence before it reaches the generator.

04

Generation

Produce the answer

Evaluate open and closed LLMs through local deployment and API-based generation interfaces.

From modules to agentic workflows

Five ways to connect the components.

Orchestrators control the flow of retrieval and generation: run a sequence, select a branch, iterate, combine parallel results, or mix these strategies.

Five XRAG orchestrators: sequential, conditional, iterative, parallel, and hybrid retrieval-augmented generation workflows.
Figure 3: orchestration logic for agentic RAG.

03 / The evidence

Compare the workflow, then examine the failure.

A shared benchmark reveals how orchestration changes answer quality, and why retrieving relevant evidence is only part of the story.

Orchestrator comparison

Table XI · DeepSeek-R1-Distill-Llama-70B generator

HotpotQA · Mean scores (%) from Table XI; higher is better.
Method Orchestrator ChrF ↑ METEOR ↑ ROUGE-L ↑
BGE-Large Sequential 26.06 31.08 17.17
Self-RAG Conditional 17.26 13.00 11.72
SIM-RAG Iterative 62.30 52.39 65.48
RRFusion Parallel 23.14 25.47 24.59
Adapt-RAG Hybrid 31.20 33.56 23.17
FinanceBench · Mean scores (%) from Table XI; higher is better.
Method Orchestrator ChrF ↑ METEOR ↑ ROUGE-L ↑
BGE-Large Sequential 19.82 19.14 11.20
Self-RAG Conditional 11.98 7.87 6.09
SIM-RAG Iterative 21.12 15.81 21.92
RRFusion Parallel 18.83 15.67 10.91
Adapt-RAG Hybrid 19.97 18.07 12.16

Selected generation metrics from the paper’s benchmark. Bold marks the best score in each column. Switch datasets to compare the same five orchestration strategies.

On HotpotQA, the iterative SIM-RAG configuration reaches 52.39 METEOR, compared with 31.08 for the sequential BGE-Large configuration. On FinanceBench, the best method depends on the metric: a single orchestration choice does not lead every column.

Explore the complete results
HotpotQA hard questions with GPT-4o-mini: answer ChrF++ distributions for Hit@1 retrieval hits and non-hits, with means of 0.28 and 0.14.
Retrieval and generation must be evaluated together. With GPT-4o-mini on the hard HotpotQA subset, mean ChrF++ is 0.28 for Hit@1 hits and 0.14 for misses. Some hits still produce weak answers. Figure 7(c).
SIM-RAG on hard HotpotQA questions: generation quality across retrieval iterations, including questions that reach the five-iteration cap.
More iterations do not guarantee a better answer. The hard-subset analysis shows that reaching the five-iteration limit can still leave low-quality outputs. Figure 8(c).

04 / The open-source toolkit

From a benchmark to your own experiments.

Configure the data, choose the components, and inspect the results through XRAG’s command-line tools and Web UI.

  1. Prepare a unified dataset.Connect queries, retrieved contexts, reference contexts, and expected answers.
  2. Compose a pipeline.Select the retriever, post-processor, generator, and workflow.
  3. Inspect both ends.Evaluate retrieval quality alongside answer quality and semantic consistency.
Explore the code ↗Documentation ↗
XRAG Web UI from the paper: dataset and model configuration, retrieval setup, interactive generation, and evaluation.
Figure 2: the XRAG interface presented in the paper.

05 / Cite this work

Build on XRAG.

If you find XRAG useful, please consider citing our paper.

@misc{zhang2026xragexaminingcore,
  title={{XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation}},
  author={Qili Zhang and Qianren Mao and Yangyifei Luo and Yashuo Luo and Hanwen Hao and Zhilong Cao and Weifeng Jiang and Zhijun Chen and Junnan Liu and Feng Yan and Xiaolong Wang and Jinlong Zhang and Zhenting Huang and Zhixing Tan and Jie Sun and Bo Li and Jianxin Li and Philip S. Yu},
  year={2026},
  eprint={2412.15529},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2412.15529}
}
Download .bib