GraphRAG against vector retrieval
A worked comparison of GraphRAG against traditional vector retrieval, written as a tutorial for the students taking the course next.
What it is
A two-notebook tutorial for the Social Networks and Media course, written with one colleague as extra material for the students taking it next. Both notebooks answer WebQSP questions over Freebase with the same retrieval: SentenceBERT embeds every node and edge, and a prize-collecting Steiner tree pulls out a small connected subgraph for each question.
They differ only at generation. The first flattens the subgraph into text triples and asks Llama 3.1 8B zero-shot. The second uses PyTorch Geometric's G-Retriever, where a graph attention network turns the subgraph into soft prompt tokens and LoRA teaches the model to read them. On the same 100 test questions, exact match rose from 41% to 85% and F1 from 69.3% to 88.2%, after one epoch on 300 samples.
Exact match
F1
The numbers
| Group | Method | Score |
|---|---|---|
| Exact match | Textualised triples, plain prompt | 41.0% |
| Exact match | G-Retriever, GAT and LoRA | 85.0% |
| F1 | Textualised triples, plain prompt | 69.3% |
| F1 | G-Retriever, GAT and LoRA | 88.2% |
Evaluation cells of both notebooks, SQuAD exact match and F1.
- A two-notebook series over WebQSP and Freebase, framed for a reader meeting the technique for the first time.
- SentenceBERT indexing and PCST subgraph extraction feeding a 4-bit Llama 3.1 8B Instruct.
- Compares a PyTorch Geometric G-Retriever, GAT with LoRA, against a textualised-triples baseline.
What I learnt
- Retrieval quality sets the ceiling. Even a strong model cannot answer well from an incomplete or irrelevant subgraph.
- Text triples are readable but flatten the graph into a sequence. The graph encoder keeps multi-hop structure that text loses.
- The two approaches are complementary. G-Retriever feeds the model both the graph tokens and the text description.
- A little LoRA training goes a long way. One epoch on 300 samples was enough for the jump in exact match.
What I would do next
Train on more of WebQSP and for more epochs, and try larger language models.
Explore hybrid retrieval, graphs built on the fly, and multimodal GraphRAG.