Building My Own Local LLM — A Realistic Way to Run Sonnet-Level Performance on My Mac

"I want to build a model like Claude too, don't I?"

Using Claude Code, a thought crossed my mind: "Couldn't I build an LLM like this myself and run it locally? Nothing grand needed — Sonnet-level would be plenty." I bet quite a few people have had a similar thought. The API cost adds up, sometimes you don't want your data leaving your machine, and above all, there's the pull of wanting something that's "mine."

So I looked into it. Short answer:

Building a Sonnet-level model from scratch is impossible for an individual. But "using something Sonnet-level locally" is entirely achievable.

These are completely different problems. The former is a domain requiring thousands of GPUs, trillions of tokens, and costs in the tens of millions of dollars. The latter, though, can start tonight with a few lines in a terminal. This post covers 4 realistic options, organized by goal.

First, decide on your goal

Before picking a method, it helps to get clear on what I actually want. At first I lumped it all together as "I want to build a model," but it turned out to be 4 different goals mixed together.

What I want	Realistic approach
Sonnet-level performance, locally	run an open-weight model (Ollama)
Have it work with my documents/knowledge	RAG
Match my tone/domain style	LoRA fine-tuning
Learn how LLMs actually work	train a small model from scratch with nanoGPT

Let's go through them one by one.

Method 1: run an open-weight model — the most realistic starting point

Download an already-published, high-performance model and use it as-is. Open models have come a long way lately, and the larger ones perform close to a previous-generation Sonnet. Models like Qwen, Llama, DeepSeek-R1 distills, Gemma, and Mistral are the go-to options here.

Prerequisite: what size model can your Mac handle

An Apple Silicon Mac has an advantage here. Thanks to its unified memory architecture, RAM size directly determines the model size you can run.

RAM	Model you can run	Feel
16GB	7–14B	fine for everyday tasks
32–64GB	30–70B	a noticeable jump in coding/reasoning

Check your memory like this.

sysctl hw.memsize | awk '{print $2/1024/1024/1024 " GB"}'

Step 1: Install Ollama

There are a few tools for running local LLMs, but for a first try, Ollama is the simplest.

brew install ollama

If a GUI is more comfortable, there's LM Studio; if you need low-level control, there's llama.cpp — but starting with Ollama is what I'd recommend.

Step 2: Pull the model and run it

ollama run qwen3:14b

This one line covers everything from downloading the model to starting a conversation. The first run takes a while to download the model file (several GB), but once downloaded it launches instantly afterward.

Step 3: Use it via API

Ollama also spins up a local OpenAI-compatible API server. In other words, code you've already written against an API can be pointed at the local model instead.

curl http://localhost:11434/v1/chat/completions \
  -d '{
    "model": "qwen3:14b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Frequently used Ollama commands

ollama list          # list downloaded models
ollama pull <model>  # only download the model
ollama run <model>   # start a conversation
ollama rm <model>    # delete a model (free up disk)
ollama ps            # check what's currently loaded in memory

Method 2: RAG — consider this before fine-tuning

If you want "a model that knows my documents," fine-tuning is the first thing that comes to mind. Same for me. But looking into it, fine-tuning turns out to be inefficient for injecting knowledge. Fine-tuning is great for imprinting a tone or format, but it's not well suited for making the model remember new facts.

If you want it to work with your documents/knowledge, RAG (Retrieval-Augmented Generation) is far cheaper and more effective. The mechanism is simple.

Chop your documents into small pieces and store them in a vector DB
When a question comes in, search for relevant document chunks
Insert the retrieved content into the prompt and hand it to the model

The model itself stays untouched — it's a matter of "handing it reference material." No training needed, so if documents change, just update the DB. A well-written system prompt combined with RAG is often more than enough for personal use.

Method 3: LoRA fine-tuning — when you want to imprint your own style

There are things RAG can't solve — a model's tone, response format, phrasing specific to a domain. This is where fine-tuning comes in.

Rather than retraining all the parameters, though, you use the LoRA (Low-Rank Adaptation) approach. It leaves the model body untouched and trains only a small adapter, which makes it feasible even on a personal GPU or Mac. If memory is even tighter, there's a quantized variant called QLoRA too.

Standard tools per environment:

Environment	Tool
Apple Silicon Mac	MLX (`mlx-lm`)
Linux + NVIDIA GPU	unsloth, axolotl

On a Mac, Apple's own MLX framework handles local fine-tuning well.

pip install mlx-lm

# LoRA training with your prepared training data (JSONL)
mlx_lm.lora --model Qwen/Qwen3-14B --train --data ./my_data

One thing to watch out for. Fine-tuning is 80% preparing training data. Running the tool itself isn't hard — the real work is producing hundreds to thousands of good example entries that demonstrate "the response style I want."

Method 4: nanoGPT — if you want to learn the actual "building" process

Finally, separate from practicality, if you want to learn "how an LLM is actually built," training a small model from scratch yourself is the best way.

Andrej Karpathy's nanoGPT — a project that implements and trains GPT in a few hundred lines of code
Best paired with the same author's "Let's build GPT" YouTube lecture

A model built this way has no practical use. But you come away understanding, down to the bone, how tokenizers, attention, and the training loop mesh together. If the root of your "I want to build one" urge was actually a desire to learn, this is the right answer.

Troubleshooting

The model is way too slow — running a model too large for your RAM causes swapping, and things slow to a crawl. Check memory usage with ollama ps and try switching to a smaller model or a more aggressively quantized version (e.g. the q4 family).

Response quality falls short of expectations — comparing a 7B-class model to Sonnet is bound to disappoint. Use as large a model as your RAM allows (30B+), and it also helps to split models by use case (one for coding, one for summarizing, etc.).

Running out of disk space — a single model is several GB to tens of GB. Clean up unused models with ollama rm.

Summary

Here's the whole flow at a glance.

Start by just running an open model with Ollama — doable today
Attach RAG if you need your own knowledge — comes before fine-tuning
LoRA fine-tuning if you need to adjust style — MLX on a Mac
nanoGPT if you're curious about the mechanics — for learning

Breaking down the vague urge of "I want to build a model" showed that most of it was actually solved by options 1–2. Building from scratch isn't within an individual's reach, but assembling something to your own taste while standing on the shoulders of giants turned out to have a lower barrier than expected. Start with one line — ollama run — and go from there.