The Engine Room: Running AI on Your Own Machine

On Day 3 of building my own AI agent, I decided to understand the brain behind AI, and what it takes to run it locally on my own hardware.

A model is a file full of numbers. Several AI labs (Meta, Mistral, Google, Alibaba, even OpenAI) give some of these files away for free, so anyone can download one and run it on their own machine. No subscription, and no data leaving your house.

If you're a security person, you're already thinking it: where did that file come from? You're right to ask. Some older model formats can run code the moment you load them, and a model can also be quietly trained to misbehave. It's the same supply-chain problem open source has always had. Stick to trusted publishers and safer formats like GGUF or safetensors.

You'll see names like 8B or 70B. The B stands for billions of parameters, the numbers inside the file. Bigger generally means smarter, but local hardware has limits, and chasing bigger models gets expensive fast. That's where quantization comes in.

Quantization shrinks a model by storing each number with less precision, say 4 bits instead of 16. The file gets about 4x smaller and you lose a little quality, but not much. It is comparable to photos. You can compress an image and still see and comprehend it. Quantized models usually come as .gguf files, the format used by tools like Ollama and llama.cpp.

The whole model has to fit in memory, with room to spare. At 4-bit, a rough rule is about 0.6 GB per billion parameters:

The model isn't the only thing in memory. Your OS needs its share, and so does the context window. On a 16 GB machine, an 8B model at higher quality (8-bit, ~8.5 GB) plus the OS can leave you only a couple of GB for context. Drop to 4-bit and you get breathing room.

The context window is the total amount of text the AI can process at once. Think of:

It's measured in tokens. In English, 1 token is roughly ¾ of a word, so 1,000 tokens ≈ 750 words. A bigger context window takes more memory.

Having enough memory doesn't mean fast answers. How fast the model responds, called inference speed, is measured in tokens per second, and it comes down mostly to how fast your memory is. For every token it writes, the model has to read through nearly the whole file.

Rule of thumb: tokens/sec ≈ memory speed ÷ model size in GB

Estimates at 4-bit (8B / 32B / 70B):

These are theoretical. Real-world speeds usually come in lower.

So, in terms of speed:

For scale, people read at roughly 5–8 tokens per second. Around 10 tok/s feels like a live conversation. Only the M4 Pro came close with a 70B model in the above comparison.

That's why the valuations of memory makers like Micron, SK Hynix, Samsung, and SanDisk are sky-high right now. Frontier companies need more and more tokens.

AI is billed per token, in and out. And agents burn far more tokens than a chat session, because they think, run a command, read the output, think again.

I originally bought an OptiPlex with 16 GB of DDR4, but Day 3 taught me that this was too little. So I upgraded the RAM to 64 GB. Will this be enough to take this to the finish line? Let me know your guesses, but we shall find out.

Meta Muse did tell me that a 7090 i7 had room for a GPU, so we are not entirely cooked.

I would love to start building, but the right call is always to assess security before deploying anything on your server. Day 4 is AI security fundamentals.

If you enjoyed the read, subscribe to stay in the loop.