How to Run a Qwen Model Locally with Ollama

The short answer:
Install Ollama, pull a Qwen model that fits your hardware, and start chatting or calling a local OpenAI-compatible endpoint. For most developers this is the lowest-friction way to get a working local Qwen setup in under an hour. You do not need a Python environment, CUDA toolkit, or manual GGUF conversion to begin.

Who This Is For

This guide is for developers who have already decided they want to try a local Qwen model and prefer the simplest reliable path. You may have followed the earlier hardware checklist and size explanations. You are ready to move from theory to a running endpoint on your own machine.

If you later need finer control over quantization, multi-GPU serving, or maximum concurrent throughput, other tools (llama.cpp, vLLM) become relevant. Start here first.

What You Will Have When Finished

• Ollama installed and running as a background service

• At least one Qwen model downloaded and loadable

• The ability to chat interactively in the terminal

• An OpenAI-compatible API endpoint at http://localhost:11434/v1 that your own code or tools can call

Prerequisites

• A machine that can run at least a small quantized model (see the earlier hardware checklist).

• Enough free disk space for the model you choose (a few GB for small models, 15–20+ GB for mid-size ones).

• Basic comfort with a terminal.

No prior experience with local language models is required.

Step 1: Install Ollama

Ollama provides simple installers for the three major desktop platforms.

macOS (Homebrew)

Bash

brew install ollama

Linux

Bash

curl -fsSL https://ollama.com/install.sh | sh

Windows
Use the official installer from ollama.com/download, or:

Bash

winget install Ollama.Ollama

After installation, confirm the binary is available:

Bash

ollama --version

On Linux the installer typically sets up a systemd service so the server starts automatically. On macOS and Windows the app or service usually handles this for you. You can also start the server manually with ollama serve if needed.

Step 2: Choose a Model Size That Fits

Match the model to the hardware you already checked. A practical starting map for Qwen models available through Ollama looks like this:

Exact tag names change as new Qwen generations are added to the Ollama library. Check the current library at ollama.com/library and look for tags under the Qwen family (qwen2.5, qwen3, qwen3.5, qwen3.6, qwen3.8, etc.). Always prefer a size you are confident will fit before downloading.

Step 3: Pull and Run a Model

Once you have chosen a tag, the basic commands are:

Bash

# Download the model (only needed once)
ollama pull qwen2.5:7b

# Or whatever current tag fits your hardware, for example:
# ollama pull qwen3:8b
# ollama pull qwen3.5:9b

# Start an interactive chat
ollama run qwen2.5:7b

The first run downloads the quantized weights. Subsequent runs load from local storage. Type a prompt after the >>> prompt and press Enter. Type /bye to exit the chat session.

You can list what you have downloaded at any time:

Bash

ollama list

Remove a model you no longer need with:

Bash

ollama rm <model-tag>

Step 4: Use the OpenAI-Compatible API

Ollama exposes an OpenAI-compatible endpoint by default at:

text

http://localhost:11434/v1

This is the most useful part for building applications. Any code or tool that already talks to the OpenAI Chat Completions API can usually be pointed at this local address with only a base-URL change.

Quick test with curl

Bash

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:7b",
    "messages": [
      {"role": "user", "content": "Reply with a short greeting."}
    ]
  }'

Python example (OpenAI library)

Python

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama",  # required by the client library, ignored by Ollama
)

response = client.chat.completions.create(
    model="qwen2.5:7b",
    messages=[
        {"role": "user", "content": "Explain quantization in one sentence."}
    ],
)

print(response.choices[0].message.content)

Replace the model name with whatever tag you actually pulled. The same pattern works for many existing tools, agents, and frameworks that expect an OpenAI-compatible endpoint.

Step 5: Basic Operational Notes

• Keep-alive — By default Ollama may unload a model after a period of inactivity. You can adjust this with environment variables or by keeping a process that periodically touches the model if you need it always resident.

• Context length — Default context is often modest. Longer contexts consume more memory. Start with moderate values and increase only when your task requires it.

• Multiple models — You can have several models downloaded. Only the ones you explicitly run or that are kept loaded consume significant RAM/VRAM.

• GPU detection — Ollama generally auto-detects NVIDIA, AMD, and Apple Silicon. If a model appears to run only on CPU when a GPU is present, check the official troubleshooting notes for your platform.

• Custom GGUF files — If you already have a GGUF file (for example a community quantization), you can create a Modelfile that points at it and then ollama create a local tag. This is useful once you move beyond the pre-packaged library tags.

What Can Go Wrong (and What to Check)

Document the exact tag, quantization, and hardware you used. That information is more valuable later than generic “it didn’t work” reports.

What This Setup Is Good For

• Personal coding assistance and local chat

• Prototyping small applications against a local endpoint

• Privacy-sensitive or offline work

• Learning how local inference actually behaves before investing in more complex stacks

It is not automatically the best choice for high-concurrency multi-user production serving. When that becomes the real requirement, vLLM or similar tools are usually the next step.

What Comes Next

You now have a working local Qwen endpoint. Later articles in the Local Model Lab series will cover:

• Understanding quantization choices (Q4, Q5, Q6, Q8) in more detail

• Measuring memory use and practical limits

• Running on CPU when a GPU is unavailable

• Exposing and hardening the local API for small internal tools

The cleanest path is the one that gets a useful model running on the hardware you already have. Measure, then decide whether to go larger, more quantized, or more sophisticated in the serving layer.

Make it run. Make it useful.