We Put AI on Our Own Computer With Ollama – Here’s What Running Private, Offline AI Is Actually Like
Here’s a question worth sitting with: every time you paste a document into a chatbot, where does it go? For most cloud AI tools, the honest answer is “onto someone else’s computers.” That’s fine for a lot of things. It’s not fine for your medical notes, your client contracts, your unpublished manuscript, or the pile of business documents you’d never email to a stranger. So we tried the other way. We put AI directly onto an ordinary Windows PC, pointed it at our own files, and never sent a byte to the cloud. Here’s what that’s actually like.
What Ollama actually is
Ollama is a free, open-source tool that downloads and runs AI models on your own machine. That’s the whole pitch, and it’s a big one. Instead of renting intelligence from a company by the token, you download a model once and run it locally – offline, private, with no API key and no monthly bill. You install one app, type one command like ollama run llama3.2, and a few minutes later you’re chatting with an AI that lives entirely on your computer. Pull the network cable out and it keeps working.
The models on offer are the open-weight ones you’ve heard about – Llama, Mistral, Gemma, Qwen, Phi and dozens more – in a range of sizes. Size is the whole game here, and we’ll come back to it, because it’s also where an ordinary computer starts to sweat.
The part that surprised us: embeddings, not just chat
Everyone thinks of local AI as “a chatbot on your PC.” The more useful thing we built had nothing to do with chatting. We used a small, specialized model called nomic-embed-text to make an assistant’s memory searchable – all on-device.
Here’s the idea in plain terms. A chat model talks. An embedding model reads a piece of text and turns its meaning into a list of numbers, so a computer can measure how similar two pieces of text are by meaning rather than by matching exact words. Do that to a folder of your own documents and something quietly powerful happens: you can ask a question in your own words and get back the passages that actually mean the same thing, even when they don’t share a single keyword. That’s semantic search, and running it locally means the folder never leaves your machine. In our build it made an assistant able to recall the right notes on demand, fully private, no API key, nothing phoning home.
Why you’d actually want this
Strip away the jargon and local AI shines in a specific situation: you have your own accumulated stuff, and you want AI’s help with it without handing it over. A few concrete cases where it earned its keep:
Your own documents, searchable. Years of notes, contracts, research, receipts, meeting minutes – embed them once and you can ask “what did we agree about payment terms?” and get the relevant lines, not a keyword soup. Private drafting and cleanup. Summarizing, rewriting, and reformatting sensitive text stays on your desk. Offline reliability. On a plane, in a dead-zone office, or during an outage, a local model just works. No metering. Once it’s downloaded, you can run it a thousand times without watching a usage counter tick.
Setting it up on Windows – genuinely simple
We were braced for a weekend of configuration pain. It took minutes.
The short version: download the Windows installer from ollama.com, run it, and Ollama sits quietly in your system tray. Open a terminal (PowerShell is fine) and type ollama run llama3.2 for a small chat model, or ollama pull nomic-embed-text to grab the embedding model. The first run downloads the model – a few hundred megabytes to a few gigabytes depending on which one – and after that it’s on your disk for good. That’s the entire ceremony. No account, no key, no credit card.
The honest limits (this is the important part)
We’re not going to pretend a laptop is a data center. Here’s where reality bites.
Without a strong graphics card, big chat models are slow. On a CPU-only machine, a small model (1-3 billion parameters) feels snappy enough for chat and summarizing. Push up to a 7-8 billion parameter model on the same CPU and you’ll watch words appear at reading pace or slower – usable for a considered answer, frustrating for rapid back-and-forth. A good GPU changes everything; without one, stick to small models. RAM and disk are real constraints. Bigger models want more memory to load and gigabytes of disk to store. Have a couple and you’ll juggle. Small local models are not frontier-smart. A 3-billion-parameter model running privately on your desk is genuinely useful, but it will not out-reason the giant cloud models on hard, nuanced tasks. For heavy lifting, the cloud still wins on both speed and raw capability.
The good news: embedding models are cheap to run. nomic-embed-text is small and fast even on a CPU, which is exactly why the “search your own files” use case works so well on ordinary hardware – the expensive, slow part (a big chat brain) is optional.
So who should bother?
If your work touches anything you wouldn’t want on someone else’s servers – health records, legal files, financials, client data, private writing – local AI isn’t a novelty, it’s the responsible default for those tasks. If you just want the smartest possible answer to a general question, the cloud is still the easier call.
The way we landed on it: run small local models for the private, everyday, offline stuff and for searching your own documents, and reach for a cloud model when you need raw horsepower. You don’t have to choose a side. The real unlock is simply knowing that AI on your own computer, on your own files, is now a free download and a one-line command away – and that some of your data never had to leave home in the first place. That’s the kind of tool that actually expands what a person can do without asking you to give anything up.
Sources & further reading:
Related Reading
- Run Your Own Private AI: A Beginner’s Guide to Offline LLMs with Ollama and LM Studio
- AI PCs Explained: The Everyday Tasks That Run Faster, and Private, on Your Own Device
- AI Is Training on Your Data by Default — Here’s How to Turn It Off, Platform by Platform