Technorat Tech news and explainers, minus the noise · No ads · No tracking · Sources on every post
Technorat
Search

How-to/AI

How to Run AI Models Locally on Your Own Computer

What hardware you need to run an open-weight AI model at home, how parameter count and quantization set RAM and VRAM needs, and how tools like Ollama work.

Short answer

To run an AI model locally, install a runner such as Ollama or LM Studio, download an open-weight model, and pick a size and quantization that fits your memory. A rough rule: a 4-bit model needs a little over half a gigabyte of RAM or VRAM per billion parameters, plus room for context.

Key takeaways

  • Model memory is roughly parameters times bits per weight divided by 8, so a 7B model at 4-bit is about 4 GB before context.
  • Quantization (for example 4-bit GGUF files) cuts memory use to about a quarter of 16-bit weights, with a small loss in quality.
  • A GPU with enough VRAM is fastest; Apple Silicon Macs use unified memory, and CPU-only works for small models but is slower.

Running a language model on your own computer means no subscription, no usage cap, and prompts that stay on your machine. The catch is hardware: a model that does not fit in memory either refuses to load or crawls. This guide explains how to size a model to your RAM or graphics card, what open-weight models and quantization are, and how the common desktop tools fit together.

What does "running a model locally" actually involve?

You need three pieces: an open-weight model file, a runner program that loads it, and enough memory to hold the model while it generates text.

An open-weight model is one whose trained parameters are published for download, usually on Hugging Face, under a license that allows local use. The runner (Ollama, LM Studio, llama.cpp, and others) reads the file, places it in GPU memory or system RAM, and handles the token-by-token generation described in what a large language model is.

Licenses vary. Some open-weight models allow commercial use, while others restrict it, so read the model card before building anything on top of one.

How much memory does a model need?

Memory use is roughly the parameter count times the bits stored per weight, divided by eight, plus extra for the conversation context.

At 16-bit precision each parameter takes 2 bytes, so a 7-billion-parameter model needs about 14 GB just for its weights. Quantized to around 4.5 bits per weight (the figure Hugging Face lists for the common Q4_K type in GGUF), the same model drops to about 4 GB. The table below uses that formula; real files vary slightly, and longer context windows add more on top.

Model size 16-bit weights ~4.5-bit quantized Typical fit
3B ~6 GB ~1.7 GB Most laptops, even CPU-only
7B to 8B ~14 to 16 GB ~4 to 4.5 GB 8 GB GPU or 16 GB RAM
13B to 14B ~26 to 28 GB ~7.3 to 8 GB 12 GB GPU or 32 GB RAM
30B to 32B ~60 to 64 GB ~17 to 18 GB 24 GB GPU or 32 GB+ unified memory
70B ~140 GB ~39 GB 48 GB+ VRAM or 64 GB+ unified memory

The "typical fit" column leaves headroom for the operating system and context. If you are unsure what your machine has, how to check your PC specs shows where to find RAM and GPU details on Windows and macOS.

What is quantization, and does it hurt quality?

Quantization stores each weight with fewer bits, which shrinks memory use and speeds up generation at the cost of a small, usually acceptable drop in accuracy.

Hugging Face's GGUF documentation lists types from 8-bit down to roughly 1.5 bits per weight. In practice:

  • 8-bit (Q8_0) is close to the original quality and about half the size of 16-bit.
  • 5-bit and 6-bit types (Q5_K, Q6_K) are a good middle ground if you have spare memory.
  • 4-bit types (Q4_K_M and similar) are the common default for home use.
  • 3-bit and 2-bit types fit very large models into small memory, but answers degrade noticeably.

A larger model at 4-bit often beats a smaller model at 8-bit for the same memory budget, so try that comparison before settling.

Which hardware runs local models best?

A dedicated GPU with plenty of VRAM is fastest, Apple Silicon Macs are strong because of unified memory, and CPU-only setups work for small models.

On a Windows or Linux PC, the model's speed depends heavily on whether it fits entirely in VRAM. If it spills into system RAM, generation slows sharply. NVIDIA cards have the widest software support; AMD and Intel GPUs are supported by many runners too, with more variation between tools.

On Macs with Apple Silicon, the GPU shares the same memory pool as the CPU, so a Mac with 32 GB of unified memory can load models that would need a 24 GB graphics card on a PC. The NPUs found in newer laptops are used by some apps, though most general-purpose local runners still rely on the GPU or CPU as of November 2026.

How do you set up a local model, step by step?

Install a runner, choose a model small enough for your memory, download a quantized version, and start chatting from the app or command line.

  1. Check your free RAM and VRAM, then pick a target size from the table above.
  2. Install a runner. Ollama offers downloads for macOS, Linux, and Windows and is driven mostly from the command line. LM Studio is a desktop app with a graphical model browser and chat window; its site says it uses MLX and llama.cpp under the hood.
  3. Download a model. Both tools can fetch popular open-weight models for you; on Hugging Face, filter by the GGUF tag to find files for llama.cpp-based runners.
  4. Start with a 4-bit version. If replies are fast and memory is left over, step up to a larger model or a higher-bit quantization.
  5. Connect other apps if you want. Local runners commonly expose an API on your own machine so editors, note apps, or scripts can send prompts to the model.

One caution: some runners now also offer cloud-hosted models alongside local ones. Ollama's download page, for example, describes running models either locally or in its cloud. If privacy is your reason for going local, confirm the model you picked is actually running on your hardware.

What should you expect from a local model?

Small local models handle summarizing, drafting, and coding help well, but they make more factual mistakes than the largest hosted models.

Treat answers the same way you would treat any chatbot output, and verify facts that matter, as described in AI hallucinations explained. Keep a 4-bit 7B or 8B model as your everyday default, and load a bigger one only when a task needs it, since a model that barely fits leaves no memory for anything else you have open.

Frequently asked questions

Can I run a local AI model without a graphics card?

Yes. Runners based on llama.cpp can run models on the CPU using system RAM. Small models in the 1B to 8B range are usable this way, though replies arrive more slowly than on a GPU.

How much RAM do I need for a 7B or 8B model?

At 4-bit quantization the model file is roughly 4 to 5 GB, so 8 GB of free RAM or VRAM is a practical minimum and 16 GB of system memory gives comfortable headroom for the operating system and longer context.

Is running AI locally private?

When a model runs fully on your machine, prompts are processed locally and do not need to leave your computer. Check each tool's settings, since some runners also offer optional cloud models that do send prompts to a server.

What does GGUF mean?

GGUF is a binary file format for model weights designed for llama.cpp and related engines. It stores the tensors plus metadata, and it is the format most local runners download by default.

Sources

  1. GGUF (Hugging Face Hub documentation)
  2. Download Ollama
  3. LM Studio

Research-based: Technorat does not claim hands-on testing. See how we research. Spotted an error? contact@technorat.com

#Local AI#LLMs#GPUs#RAM