Naive RAG vs GraphRAG Evaluation / Final Architecture Decision
FINAL ARCHITECTURE DECISION — HOW THREE ARCHITECTURE DECISIONS GOT MADE, IN ORDER

Naive RAG, a router, three dead-end router designs — cut, one at a time, as better evidence arrived.

The Naive RAG vs GraphRAG evaluation exposed RAGAS's blind spots. This is what happened after: the router built on top of it, the jury harness that replaced it, and the GraphRAG-only system that came out the other side.

87.5%
final router accuracy — on a target later shown unreliable
0.867
GraphRAG = Agentic, jury-scored
21/45
naive refusals, independently re-verified
$0.000106
mean cost / query, live 44-item batch

↑ Builds on Naive RAG vs GraphRAG Evaluation → Methodology: Is the judge itself reliable?

Final Decision: GraphRAG Only

CUT, NOT HIDDEN · REAL NUMBERS · SELF-HOSTED

Try the final AeroOps system →(opens the live AeroOps app in a new tab)

BEFORE
Query
→
structural_pattern_router.py
→
naive_rag.py
FAISS · BM25 · embeddings
or
GraphRAG
AFTER
Query
→
GraphRAG
→
Answer + citations
Router and naive's entire retrieval stack cut, not hidden — shown crossed out so the removal itself is visible, not just absent.
Cost of complexity, surfaced by the harness, not by router accuracy: two real bugs in naive_rag.py — a DOC_CONFIG filename mismatch that left 2 of 7 documents with no title metadata in production[T17](opens pre-fix naive_rag.py at the last commit before archiving, on GitHub), and a page-number off-by-one that made every citation naive_rag.py ever showed a real user one page wrong.[T18](opens pre-fix naive_rag.py at the last commit before archiving, on GitHub)

One more bug worth naming, found separately: a retrieval bug in graphretriever_v5.py's path-traversal logic could occasionally state two unrelated facts as if one caused the other — a structural co-occurrence edge narrated as a real causal link. Found and fixed before this simplification.[T19](opens the fix on GitHub)

GraphRAG alone, self-hosted, in production

Self-hosted Neo4j Community Edition, same EC2 instance as the app (replaces AuraDB free tier). Through the live public /query endpoint, cross-checked against LangSmith with zero mismatches. 1 of 45 (MH01) excluded — a pre-existing API-layer bug unrelated to retrieval quality. n=44.[T21](opens the raw batch run on GitHub)

COST / QUERY (USD)
$0.000106
Mean
$0.000103
Median
$0.000056
Min
$0.000180
Max
GENERATION TIME (s)
3.400
Mean
3.120
Median
1.290
Min
7.810
Max
RETRIEVAL TIME (s)
0.039
Mean
0.030
Median
0.010
Min
0.290
Max
Retrieval under 0.1s for 42/44 items — app and Neo4j share one EC2 instance over a Docker bridge network, not the public internet.

Why RAGAS Got Demoted, Not Rejected

JUDGE INSTABILITY · STRUCTURAL BLINDNESS · REFUSAL ARTIFACT

RAGAS wasn't thrown out — it was demoted from sole authority to one signal among several, once three separate, independently-diagnosed failure modes were understood.

The largest single-item swing in the dataset

Same identical answer, scored three times, nothing regenerated between runs.[T1](opens run 1 record on GitHub)[T2][T3]

AF03
Faithfulness
0.667 SWING · LARGEST IN DATASET
Run 1: 0.0  ·  Run 2: 0.667  ·  Run 3: 0.0
Run 1 Run 2 Run 3
"Fuel starvation or interruption, compressor surge or stall, and fuel-filter clogging are the distinct root causes that can lead to an engine flameout" — cites 3 evidence chunks, matches gold ground truth. Byte-identical across all 3 runs. The score is not the answer's fault.

Structural blindness to graph-verified multi-hop claims

RAGAS Faithfulness checks whether a claim traces to one literal chunk of retrieved text. It has no visibility into the graph traversal that actually backs a multi-hop claim.

MH03
How could the failure of a single fuel tank boost pump potentially cause a multiple-engine flameout?
GOLD: The boost pump failure can generate debris that clogs the fuel filter. Once clogged, disrupted fuel flow or contamination bypassing the filter introduces the potential for a multiple-engine flameout.
GraphRAG
"A failure of one fuel-tank boost pump can generate debris that clogs the fuel filter, which in turn can cause fuel starvation and lead to flameout. Because the filter feeds multiple engines, a single pump failure can precipitate a flameout in more than one engine." CAUSES · claim CL051 · conf 0.8 · D3_c29 → LEADS_TO · claim CL052 · conf 0.75 · D3_c30 (opens raw evaluation record for MH03 on GitHub)
Faithful.0.0
Recall0.5
Two real, individually-sourced graph edges, chained correctly, cited inline. Scored 0.0 because no single passage states the full chain in one sentence — RAGAS can't credit a synthesis it never sees the structure of.
OS08
In flight, the pilot notices brief fluctuations in EPR and EGT, but the engine continues running normally because the continuous ignition system activated automatically. What event did the ignition system prevent?
GOLD: It prevented a total flameout by automatically restarting combustion after unstable engine operation or a temporary fuel interruption.
GraphRAG
"The continuous ignition prevented a flameout." MITIGATES · claim CL033 · conf 0.85 · D3_c12 · LEADS_TO · claim CL030 · conf 0.8 · D3_c09 (opens raw evaluation record for OS08 on GitHub)
Faithful.0.0
Recall1.0
100% of the needed evidence retrieved (Recall = 1.0). Scored 0.0 anyway — the judge wouldn't accept "prevented a flameout" as entailed by a source stating combustion "will restart automatically," a legitimate paraphrase rejected on strict literalism.

The refusal-artifact bug

RAGAS scores a refusal as trivially faithful — no claims made, none can be unfaithful. On items where naive RAG refused, that artificial ~1.0 beat GraphRAG's real, cited answer. build_router_ground_truth.py's own header names AF03 as one of the two cases that surfaced this.[T10](opens sourcing note on GitHub)

NAIVE RAG 1.0FAITHFULNESS

"The provided context does not contain information on the root causes that can result in an engine flameout."[T6][T7]

GRAPHRAG 0.0FAITHFULNESS

"The graph identifies three distinct root causes that can lead to an engine flameout: fuel starvation or interruption [D3_c08], compressor surge or stall [D3_c09], fuel filter clogging [D3_c30]."[T4][T5]

A refusal scored higher than a substantively correct, cited answer. Fixed in router_ground_truth.json's labeling logic — if exactly one backend refuses, the other wins outright, RAGAS isn't consulted for that item. 9 of 45 labels flipped.[T10]

The Agentic Detour

5 ROUNDS · A REASONABLE DECISION AT THE TIME

The router's original target said naive won specific categories outright. Building a router in response was the right call given that evidence — the mistake would have been never revisiting it once better evidence arrived.

100% 50% 0% 60% 51.3% 56.4% 85% 87.5% Round 1 embedding-centroid Round 2 entity-count Round 3 keyword-pattern Round 4 3-layer + refusal fix Round 5 pattern + default
Built on a target later shown to be unreliable — see §T2.
Backend accuracy against router_ground_truth.json, by router design round.[T11](opens sourcing note on GitHub)

Round 2's entity count was diagnosed as an alias-inflation artifact of ENTITY_MAP, not a real property of the question. Round 3's comparison-language rule was 4/4 wrong as a standalone signal. Round 4 gained 10.6 points purely from fixing the refusal-artifact bug in the ground truth itself (§T2) — not from a better router. Round 5 dropped the semantic layer entirely after it returned a confident-looking match at similarity 0.07–0.19 on out-of-domain junk queries.

The 87.5% figure was scored against a target itself built from RAGAS Faithfulness/Context Recall comparisons — the same metric shown in §T2 to swing 0.667 points on identical input. A real number describing real router behavior against that specific target, measured before the target's own reliability had been independently stress-tested.[T12](opens sourcing note on GitHub)

The Jury Harness

BLINDED · 3-MODEL MAJORITY VOTE · ANSWER QUALITY ONLY

A panel of independent judges scoring by majority vote — the same principle behind "Panel of LLM evaluators" (PoLL) in evaluation literature† — reused this project's own established method: the gold set's jury_agreement field was itself built by 3-model majority vote (Opus 5, GPT-5.6, Qwen), reprised here to score naive/GraphRAG/agentic's answers, blinded, answer-quality only.[T16](opens the jury prompt on GitHub)

0.511
naive — correct rate, 23/45
0.867
graphrag — correct rate, 39/45
0.867
agentic — correct rate, 39/45[T13](opens jury results on GitHub)

Every category, naive vs. GraphRAG

Naive RAGGraphRAG = Agentic
GraphRAG and agentic are identical on all 7 categories — rendered as one bar, not two, since there's no real difference to show.[T14](opens per-category results on GitHub)
n=2
DA04 · DA05
The only 2 of 45 items the router ever actually diverted to naive. On exactly those 2 — the only items where the router's decision was ever load-bearing — naive's correct rate = 0.5, GraphRAG's correct rate = 0.5. An exact tie. This is the number that carries the "not worth it" conclusion, not the aggregate.[T15](opens the routed-items comparison on GitHub)

n=2 is small enough that a coin flip wouldn't be surprising as noise on its own — but it isn't contradicted by the larger sample, it's corroborated: naive's overall 0.511 vs. GraphRAG's 0.867 holds across all 45 items, not just the 2 the router sent its way.

Net Result

DECISION RECORD

Naive RAG and the agentic router are gone from the live system. Three router designs were built, measured, and discarded in sequence as better evidence arrived — not reversed on a hunch. What's left is a single GraphRAG pipeline, verified end-to-end on infrastructure this project fully owns, with real per-query cost and latency pulled from an actual production run, not estimated.[T23](opens Net Result sourcing on GitHub)