Local Llama 3 8B Q&A System
Offline Q&A and summarisation powered by Meta Llama-3-8B-Instruct, 4-bit NF4 quantisation, retrieval-augmented generation, and decoding tuned specifically to suppress hallucination.
The project, quickly
Solo developer - inference engine, retrieval pipeline, evaluation suite, and CLI.
Meta-Llama-3-8B-Instruct, 4-bit NF4 quantised with double quantisation via BitsAndBytes.
FAISS cosine search over all-MiniLM-L6-v2 embeddings, sliding-window
chunking.
Local / offline tool - interactive CLI and a Python API, no external API calls.
The problem
Private, document-grounded Q&A shouldn't require sending your documents to an external API or paying per query. Running a capable model entirely offline means no data leaves your machine and no per-token cost - but it also means you can't lean on a hosted provider's infrastructure to keep answers fast, grounded, and non-repetitive.
This project is my attempt at that trade-off: quantize a real 8B model down to something that runs on a single consumer GPU, pair it with retrieval so answers stay grounded in actual source documents, and tune the decoding strategy specifically to suppress the hallucination and looping that smaller local setups are prone to.
Architecture
Prompt templates
Four prompt modes - QA, Summarize, Chain-of-Thought, and Extraction - each with its own template, feeding into the retrieval pipeline.
FAISS + MiniLM RAG
Documents are chunked with a 400-word sliding window and 80-word overlap, embedded with MiniLM, and retrieved by FAISS cosine search.
Quantized Llama 3
Meta-Llama-3-8B-Instruct runs in 4-bit NF4 with double quantisation and bfloat16 compute, for a real speed-up on consumer hardware.
Output cleaning loop
Generated answers pass back through a response cleaner before returning to the user, closing the loop shown in the system's architecture diagram.
Decoding strategy
The default generation settings are deliberately tuned against hallucination rather than left at framework defaults: temperature at 0.3 for grounded, low-creativity output, nucleus sampling at top-p 0.85 with top-k 40, and a repetition penalty of 1.15 to suppress the looping local models are prone to.
Evaluation results
0.84 ± 0.08 F1, measured on an 8-question benchmark set with structured answer validation.
0.81 ± 0.06 coherence score on the same benchmark set.
3.1% hallucination rate, with keyword grounding checks against source documents.
~38.2 tok/s at ~1,240ms mean latency, tested on an RTX 3090.
Evaluated on an 8-question benchmark set, separate from the 25+ unit tests that verify the pipeline itself runs without a GPU in under 5 seconds.
Tech stack
What I'd improve next
The evaluation set is honestly small - 8 questions is enough to catch obvious regressions, not enough to make strong claims about accuracy at scale. Growing that benchmark set is the single highest-value next step, before any further architecture work.
Beyond that, I'd like to add re-ranking after the initial FAISS retrieval, since cosine similarity alone sometimes surfaces chunks that are topically related but not actually the most useful for answering the specific question asked.