Chapter 17 · Local Models Foundations Track CLI Project

Teach an open model your own knowledge, without data ever leaving the laptop.

Take a free open-source LLM (Qwen2.5-Coder, Llama 3, Mistral, Phi) and adapt it to your code repository, Jira project, PDFs or wikis. RAG first, LoRA second, and a clear rule for when each is worth it. No keys, no SaaS, 100% offline.

01 What students build

A private Q&A bot over their own data. First the RAG route: walk a repo (or a folder of PDFs, or Jira exports), chunk it, embed with nomic-embed-text, index into LanceDB, and answer with qwen2.5-coder:14b via Ollama. Then, once retrieval is solid, an optional LoRA adapter via MLX-LM (Apple Silicon) or Unsloth (CUDA) bakes tone and domain jargon into the model itself.

RAG

No weights changed, CPU is fine. Best for facts that keep moving: repos with new commits, PDFs you keep adding. Your data is injected at query time.

LoRA / QLoRA

A small adapter is trained on top of frozen weights. Best for style, tone and a fixed body of knowledge. Slower to update, but answers feel native.

Full Fine-Tune

All weights retrained on a multi-GPU cluster. Almost never the right choice for a single developer. The chapter teaches you to recognize this and skip it.

02 Architecture

Your datarepo · Jira · PDFs
──▶
Chunker512-1024 tokens
──▶
Embednomic-embed-text
──▶
LanceDBlocal vector index
──▶
qwen2.5-coderOllama · 14b
──▶
+ LoRAMLX-LM / Unsloth
──▶
Answercited · offline
rag first · lora second · everything runs offline once models are pulled

03 Live demo (simulated with sample data)

This is a UI walkthrough with pre-baked sample data so you can compare the two adaptation strategies without a GPU. In the course you run the real thing locally: Ollama + index_repo.py + ask.py, with the Advance-Playwright-Framework repo as the first corpus.

fine-tune-lab · ollama · qwen2.5-coder:14b demo · sample data

04 Inputs & outputs

What goes in

  • An open model pulled via Ollama (qwen2.5-coder:14b or :7b)
  • Any text-shaped source: code repo, Jira issues, PDFs, Confluence, chat logs, test-case CSVs
  • Chunking config (~512-1024 tokens per chunk, metadata tags)
  • Optional: instruction pairs for LoRA training (MLX-LM / Unsloth)

What comes out

  • Grounded answers with file-level citations from your own data
  • A local vector index that re-builds on every commit
  • An optional LoRA adapter file that makes answers sound in-domain
  • A decision framework: RAG vs LoRA vs full fine-tune

05 How it's built

Pull qwen2.5-coder:14b and nomic-embed-text with Ollama, so the whole stack runs offline.
Write index_repo.py: walk the tree, chunk by file and function, tag each chunk with path and metadata.
Embed every chunk with nomic-embed-text into a file-backed LanceDB (or Chroma) index.
Write ask.py: retrieve top-k chunks for a question, stuff them into the prompt, answer with citations.
Swap the data source: point the indexer at PDFs, Jira REST exports or Confluence pages, same pipeline.
Once retrieval is solid, train a LoRA adapter (MLX-LM on Apple Silicon, Unsloth on CUDA) for tone and domain style.

06 Tech stack

Ollamaqwen2.5-coder:14bnomic-embed-textLanceDB / ChromaPythonMLX-LM (Apple Silicon)Unsloth (CUDA)100% offline