Quick Recap : Last Sunday, we learnt about importance of effective monitoring and governance lifecycle in Financial Institutions. In case if you had missed, read it here : SS01: governance lifecycle

Picture this: You're six months into building an AI assistant for your bank's customer service team. You decided to fine-tune a large language model on your internal documentation: product guides, policy manuals, customer interaction transcripts.

The model works beautifully in testing. Accuracy is great. Your team is excited.

Then you present it to the Model Risk Committee.

The Chief Risk Officer asks: "Where exactly is our proprietary data now?"

You explain it's embedded in the model's parameters.

"So if this model leaks, our entire knowledge base—including customer patterns, internal policies, competitive strategies—goes with it?"

You hadn't thought of it that way.

"And when we update our lending policies next quarter, we need to retrain the entire model?"

Well, yes.

"And can you explain why the model gave this specific answer to this customer question?"

That's... harder when knowledge is baked into billions of parameters.

The meeting ends with: "Come back when you have a solution that doesn't create a permanent record of our IP inside a black box."

This scenario plays out constantly. According to recent industry data, over 60% of financial institutions hit governance roadblocks with fine-tuned LLMs during deployment review.

Here's what I've learned: In regulated environments, the choice between Retrieval-Augmented Generation (RAG) and fine-tuning isn't primarily technical—it's a risk management decision. And increasingly, RAG is winning because it's governable.

Fine-Tuning = Teaching the model new knowledge by updating its internal "brain"

You take a base model (GPT-4, Claude, Llama) and continue training it on your company's data. The knowledge gets permanently embedded in the model's parameters.

Example: Fine-tune on 10,000 policy documents. Now when asked "What's our mortgage approval process?", the model answers from memory—knowledge literally stored in its parameters.

Retrieval-Augmented Generation (RAG) = Giving the model access to a search system

The model itself doesn't change. When someone asks a question:

Example: When asked about mortgage approval, the system retrieves the current policy document, and the model answers based on that specific document—like a smart librarian.

With Fine-Tuning: Your proprietary data is embedded in model parameters. If someone gains access to those files (breach, insider threat, cloud misconfiguration), they have everything.

Real incident: In 2023, Samsung employees uploaded confidential code to ChatGPT. OpenAI was using inputs for fine-tuning. Result: Samsung's code potentially learned by the model. Samsung banned ChatGPT immediately.

With RAG: Your data lives in a separate database. The model never "learns" it—only reads it temporarily. If the model gets compromised, attackers get a base model with no proprietary knowledge. Your data remains in your secured database.

Think of it like:

Why this matters for BFSI: Banks handle extremely sensitive data—customer PII, transaction patterns, credit decisions. Fine-tuned models create concentrated risk: all that data in one artifact that, if compromised, represents catastrophic IP loss.

Banking regulations change constantly. Product policies evolve. Lending criteria adjust.

With Fine-Tuning: Every update requires:

Timeline: 2-6 weeks per update

With RAG:

Timeline: Hours to 1 day. No retraining. No re-approval.

Real example: A regional bank needed quarterly lending criteria updates. With fine-tuning, they were always behind—deploying last quarter's rules. After switching to RAG, policy updates went live same-day.

When a model denies a loan, you need to explain why. Regulators demand it. Customers have a right to understand.

With Fine-Tuning: Knowledge is embedded in billions of parameters. Pinpointing which training document influenced a decision is nearly impossible.

Regulator question: "Why did your model deny this loan?" Fine-tuning answer: "The model learned patterns suggesting high risk. We can show attention weights, but can't point to a specific policy." Regulator reaction: "Not acceptable."

With RAG: Every answer comes with citations.

Regulator question: "Why did your model deny this loan?" RAG answer: "The model retrieved Section 3.2 of our Credit Policy (v2024-Q3), which states DTI ratios above 45% require additional documentation. The applicant's DTI was 48%. Here's the exact text." Regulator reaction: "Perfect. I can verify that policy is compliant."

Why this matters: In high-stakes financial decisions, provenance beats sophistication. A slightly less accurate model that can prove its reasoning beats a more accurate black box.

Fine-Tuning Costs:

RAG Costs:

Real comparison from a fintech: RAG was 6x cheaper, not counting the organizational overhead of repeated governance reviews for each model retrain.

I'm not saying fine-tuning is always wrong. There are specific scenarios:

Use Case 1: Style and Tone Consistency When you need the model to consistently use specific language or formatting deeply embedded in how your company communicates.

Use Case 2: Highly Specialized Domain Language When your domain uses language so specialized that base models struggle even with retrieval. Example: Legal contract analysis requiring deep understanding of clause interdependencies.

Use Case 3: Hybrid Approaches Fine-tune for domain expertise and style, use RAG for current facts and policies. About 15% of production systems use this pattern.

The reality: Looking at production BFSI deployments in 2024-2025:

Pure fine-tuning is becoming rarer as governance requirements tighten.

Document Store

Semantic Search

Citation & Logging

What makes this audit-ready:

Three characteristics pass compliance review:

Q1: How often does your knowledge change?

Q2: How sensitive is your data?

Q3: How important is explainability?

Q4: What's your compliance burden?

Decision rule: If 3-4 questions point to RAG → Use RAG. In my experience, 80% of BFSI use cases do.

2026-2027: RAG Becomes Default As European AI Act requirements kick in (August 2026), RAG's transparent architecture becomes increasingly attractive. Financial institutions will standardize on RAG for most use cases.

2027-2028: Knowledge Graph Integration Next-generation RAG systems combine document retrieval with knowledge graphs—understanding how policies and concepts relate across documents, not just retrieving isolated chunks.

2028-2030: Continuous Learning RAG Systems that improve from user interactions—not by changing model parameters, but by improving retrieval, updating rankings, and flagging documentation gaps. Learning at the knowledge base level, not model level.

The principle that remains: Keep sensitive data out of model parameters, keep it in controlled, auditable stores.

Key takeaways:

Start here:

Looking ahead (2026-2030):

Open questions:

Fine-Tuning: Continuing to train a pre-trained model on your specific data, permanently embedding your knowledge into the model's parameters. Like teaching someone skills through practice until it becomes second nature.

RAG (Retrieval-Augmented Generation): Architecture where the model searches a database when answering questions, reads relevant documents, and generates responses based on what it just read. Like a smart librarian who looks up answers.

Model Parameters: The billions of numbers inside a model that determine its behavior. Training adjusts these numbers. Once information is encoded, it's difficult to remove without retraining.

Embeddings: Numerical representations of text that capture semantic meaning. Enables finding documents based on concepts, not just keywords.

Data Leakage: When sensitive information from training data can be extracted from a model through clever prompting. Major risk with fine-tuning.

Hybrid Architecture: Combining fine-tuning (for domain expertise and style) with RAG (for current facts and sensitive info). Attempts to get benefits while mitigating risks.

Semantic Search: Finding documents based on meaning and context rather than exact keyword matches. Core capability of RAG systems.

Model Exfiltration: Security risk where an attacker gains access to model parameters. With fine-tuned models, this can expose all proprietary training data.

On The Samsung Incident Impact: When Samsung banned ChatGPT after employees uploaded confidential code, it wasn't just about the immediate leak—OpenAI's terms allowed using inputs for model improvement, meaning Samsung's code could surface in responses to other users. Within six months, three major US banks cancelled planned fine-tuning projects and pivoted to RAG. The incident became a case study in why enterprises need full control over their data.

On Real Cost Differences: A large European bank calculated €180,000 annually for maintaining fine-tuned compliance Q&A (€45K compute, €85K data prep, €50K additional governance reviews). Their RAG alternative: €22,000 annually. The 8x difference wasn't technical—it was organizational overhead of repeatedly validating a changing model versus updating a static document repository. Cost aside, the team velocity improvement was more significant: policy updates went from 6-week cycle to same-day deployment.

Next Sunday : We will be exploring How Embeddings Represent Meaning and Similarity—the mathematical foundation that makes semantic search and RAG systems work. You'll understand why similar documents cluster together in vector space and how to leverage this for better retrieval.

This is part of our ongoing work understanding AI deployment in financial systems. If you've made the RAG vs fine-tuning decision in your organization, I'd love to hear what factors drove your choice.