GraphRAG found the evidence naive RAG missed. The metric that scored them couldn't see it.

AeroOps is a diagnostic reasoning system for turbofan engine maintenance — it traces fault chains, cross-references procedures, and synthesizes answers across 260 pages of failure-mode, deterioration, and maintenance documentation, the same way an experienced engineer would work a diagnosis across multiple manuals. GraphRAG was built to test whether representing that documentation as a knowledge graph produces more reliable multi-hop and cross-document reasoning than standard vector-similarity retrieval. Maintenance decisions in this domain carry real safety and cost consequences, which is why this evaluation holds both systems to a hard standard: not just whether an answer sounds right, but whether it's actually traceable to the evidence.

Every item in this set is either machine-verified by independent consensus or personally checked by hand. Nothing in the gold set is unverified.^{[1](opens the full frozen gold set on GitHub, in a new tab)}

Is the judge itself reliable?

The same 45 GraphRAG answers were re-scored 3 times by the RAGAS judge, with nothing regenerated between runs. The aggregate mean held tight. Individual items didn't.

Dense vector search (MiniLM embeddings, ChromaDB) merged with BM25 keyword search^{[3](opens naive_rag.py on GitHub, in a new tab)}, deduplicated and truncated to the top 5 chunks per query. A keyword metadata filter narrows retrieval to the relevant document before search runs.^{[2](opens the naive RAG RAGAS evaluation results on GitHub, in a new tab)}

Retrieval dispatches into one of two main paths depending on the query^{[5](opens graphretriever_v5.py on GitHub, in a new tab)}, both restricted to real semantic edge types — never structural co-occurrence. Generation runs under a 15-rule grounding prompt^{[7](opens answer_generator_groq.py on GitHub, in a new tab)}, including a rule that preserves the source's own cause→effect direction rather than inverting it.

RAGAS Faithfulness checks whether each claim in an answer can be traced to a single literal passage of retrieved text. It gives no credit for combining several separately-true passages into one synthesized, correct conclusion — which is exactly what GraphRAG is built to do.

Three representative examples follow. Every answer below links directly to its full raw record — GraphRAG's to the exact graph edges backing its claim^{[9](opens the full GraphRAG raw evaluation outputs on GitHub, in a new tab)}, naive RAG's to its retrieved context and generated answer^{[10](opens the full naive RAG raw evaluation outputs on GitHub, in a new tab)}.

Seven metrics computed directly against the graph and the gold set^{[11](opens the GraphRAG mechanical scores on GitHub, in a new tab)} — no judge, no subjectivity. For the first six columns, darker amber = weaker score. Overfetch uses a separate blue-gray scale (darker = more over-retrieval) since higher is worse there, the opposite direction of every other column — values are printed in every cell either way, so the color is a supplement, not the only signal.

Known limitations — stated plainly

Every numbered citation in this report links to a real file in the public repository — full raw output, not a summary of it.

1. [1] Frozen gold set — 45 items, 7 categories, jury-verified (Gemini-generated, cross-checked by Qwen3.8/GPT-5.6/Opus 5).
   evaluation/gold_set/aeroops_goldset_v1_frozen.json(opens on GitHub, in a new tab)
2. [2] Naive RAG — RAGAS evaluation results — Faithfulness and Context Recall, scored per item and by category.
   evaluation/naive_rag/aeroops_naive_rag_ragas_results.json(opens on GitHub, in a new tab)
3. [3] naive_rag.py — Hybrid BM25 + dense-vector retrieval and generation implementation.
   naive_rag.py(opens on GitHub, in a new tab)
4. [4] query_understanding_v3.py — QueryRouter: rule-based entity extraction, hop-count and intent classification.
   query_understanding_v3.py(opens on GitHub, in a new tab)
5. [5] graphretriever_v5.py — GraphRetriever: path-traversal, star retrieval, and no-path fallback logic.
   graphretriever_v5.py(opens on GitHub, in a new tab)
6. [6] context_builder.py — ContextBuilder: formats retrieved graph facts with inline chunk-level provenance.
   context_builder.py(opens on GitHub, in a new tab)
7. [7] answer_generator_groq.py — AnswerGenerator: the 15-rule grounding prompt, including the directional-inversion fix.
   answer_generator_groq.py(opens on GitHub, in a new tab)
8. [8] GraphRAG — RAGAS evaluation results — Faithfulness and Context Recall, scored per item and by category.
   evaluation/graphrag/aeroops_graphrag_ragas_results.json(opens on GitHub, in a new tab)
9. [9] GraphRAG — raw retrieval/generation outputs — All 45 items: model answers, retrieved contexts, matched entities, reasoning chains with edge claim IDs and confidence. Source for every worked-example citation in Section 4.
   evaluation/graphrag/graphrag_ragas_inputs.json(opens on GitHub, in a new tab)
10. [10] Naive RAG — raw retrieval/generation outputs — All 45 items: model answers and retrieved contexts.
    evaluation/naive_rag/naive_rag_ragas_inputs.json(opens on GitHub, in a new tab)
11. [11] GraphRAG — mechanical scores — Deterministic, no-LLM-judge metrics: node/citation recall & precision, chain coverage, edge authenticity, overfetch.
    evaluation/graphrag/aeroops_graphrag_mechanical_scores.json(opens on GitHub, in a new tab)