Businesses considering a language-model assistant often encounter two options: retrieval-augmented generation, usually called RAG, and fine-tuning. They address different problems and can sometimes be combined. The right choice depends on whether the application needs access to specific information, more consistent task behaviour, or both. Before selecting either, test whether a clear instruction and a small amount of suitable context already meet the requirement.
Use retrieval for information the answer needs
RAG retrieves relevant material from an approved collection and supplies it as context when generating a response. In a hypothetical internal policy assistant, this could mean finding the current travel policy before answering an expense question. Updating the collection can make new information available without retraining the model, although indexing and retrieval must also be refreshed. Retrieval is useful when answers should refer to changing business documents and users need to inspect the supporting sources.
Use fine-tuning for a demonstrated behaviour gap
Fine-tuning adjusts a model using training examples to improve behaviour for a defined task. It may be worth evaluating when repeated prompting does not produce sufficiently consistent classifications, terminology, or response structure. The examples must be representative, correctly labelled, and permitted for that use. Fine-tuning is not a dependable substitute for looking up current account balances or frequently changing policies. It also does not guarantee factual accuracy or eliminate the need to validate outputs.
Account for permissions and untrusted content
A retrieval system must enforce the user’s access rights before information reaches the model. Filtering only the final answer is insufficient if restricted documents have already entered its context. Treat retrieved text as information, not as authority to change instructions or perform actions. A document can contain misleading commands as well as ordinary mistakes. For fine-tuning, review sensitive information and deletion obligations before preparing examples; do not assume removing a source file reverses information learned during training.
Evaluate the entire workflow
Build a held-out evaluation set containing ordinary questions, ambiguous requests, missing information, and attempts to access restricted material. For RAG, measure whether the correct evidence is retrieved separately from whether the answer faithfully uses it. A citation is useful only when the cited passage supports the claim. For fine-tuning, compare results against the unchanged model using the same tasks. Assess latency, review effort, operational cost, and appropriate refusal, not just whether an answer sounds polished.
Use a decision checklist
- Separate the need for current knowledge from the need for consistent behaviour.
- Test a prompting baseline before adding architectural complexity.
- Check source ownership, document freshness, and user-level permissions.
- Assess training-example quality and permitted use before fine-tuning.
- Reserve unseen cases for comparison and include failure scenarios.
- Assign owners for content updates, evaluation, monitoring, and rollback.
Choose the smallest approach that meets the need
Retrieval and fine-tuning are design choices, not indicators of project maturity. A search interface may be sufficient when users mainly need to locate a document. A prompted model may be sufficient for a narrow drafting task. Adopt a more complex approach only when evaluation shows a specific gap it can address and the organisation can maintain the additional components. Revisit that decision as the task and source information change.