Still think fine-tuning is better than RAG for chatbots? I tested both for a month

I run a small customer support bot for a local HVAC company in Columbus and everybody kept telling me to fine-tune an LLM on their repair manuals. So I spent 30 hours cleaning data and running fine-tuning jobs. Then for comparison I just dumped the same manuals into a vector database with RAG and tied it to GPT-4. The RAG setup caught 92% of questions correctly on the first try while fine-tuning only hit 78%. Fine-tuning also hallucinated model numbers that didn't exist three times in one afternoon. Has anyone else found RAG way easier to maintain when your source data keeps changing?

3 comments

3 Comments

gavin4692h ago

cole_flores44 has a point about them solving different things but for my setup the RAG route just worked better. I run a similar support bot for a plumbing company in Cincinnati and when I tried fine-tuning it kept making up part numbers for water heaters that don't exist. I had a customer ask about a 10 year old model and the fine-tuned model confidently told them a part number that was for a completely different unit. Switched to RAG with GPT-4 and it pulls the exact page from the manual every time. When the distributor updates a part number I just re-upload the PDF and it's done. No more spending weekends cleaning datasets.

faith_shah881h ago

gavin469 nailed it with the "re-upload the PDF and it's done" part. That hits on something bigger I've noticed everywhere these days, like how my dad still prints out maps for road trips even though his phone has GPS. People want a system that just works with what you give it instead of having to teach it every little thing from scratch. Fine-tuning feels like trying to learn a new accent to order coffee, when RAG is just handing the barista a note with your order written down. Plus when the menu changes you just rewrite the note instead of retraining your whole voice.

cole_flores446h ago

Fine-tuning and RAG solve different problems though. RAG is better for quickly finding specific facts from your docs while fine-tuning is better for teaching the model to follow a certain tone or format.