Small language models: 2026 examples and how to run one

A small language model (SLM) is a language model small enough to run on a normal laptop or phone, often with no internet. Most SLMs have fewer than 10 billion parameters. Popular examples in 2026 are Google's Gemma 4 E2B, Meta's Llama 3.2 1B and Qwen3.5-4B from Alibaba's Qwen team.

Below, you will learn what SLMs are, how they differ from large language models (LLMs) and why they matter in India. It also shows how to run one on your laptop today, and what you can research with it.

What is a small language model?

A small language model is a language model that runs on your own device instead of a big data centre. There is no official size limit. Different companies and papers draw the line in different places.

A clear definition comes from a 2025 NVIDIA paper, "Small Language Models are the Future of Agentic AI". It calls a model small if it "can fit onto a common consumer electronic device" and answers fast enough to be practical for one user. The authors add that, as of 2025, they would count most models below 10 billion parameters as SLMs.

A parameter is one number inside the model that is learned during training. More parameters usually means more knowledge and skill. It also means more memory and more computing power.

The NVIDIA authors tie the definition to a device, not a number, on purpose. Parameter counts go out of date: in their words, what is "small" today may be "large" tomorrow.

How are small language models made small?

Small language models are made small with a few standard methods. You will meet these words in every SLM paper:

• Pruning: remove the parts of a big model that matter least.
• Distillation: a big "teacher" model trains a small "student" model. Meta used pruning and then distillation for Llama 3.2 1B and 3B. Outputs from Llama 3.1 8B and 70B became training targets for the small models.
• Quantization: store each number with fewer bits, so the file gets smaller. On Ollama, Gemma 3 1B is an 806 MB download because it is stored in a compressed 4-bit format called Q4_K_M.
• Smart design: Gemma 4 E2B has 5.1 billion parameters in total, but only about 2.3 billion do the main work. The rest are lookup tables that are cheap to use. The "E" stands for "effective".

SLM vs LLM: what is the difference?

The main difference between an SLM and an LLM is where it runs. An SLM runs on your laptop or phone. An LLM usually needs many powerful GPUs in a data centre, so you reach it over the internet.

• Size: the NVIDIA paper compares a 7 billion parameter SLM with LLMs of 70 to 175 billion parameters.
• Cost: the same paper says serving that 7 billion parameter SLM is 10 to 30 times cheaper. This counts delay, energy and computing.
• Privacy: a local SLM keeps your text on your computer. An online LLM sends it to another company's server.
• Internet: a downloaded SLM works offline. An online LLM needs a connection.
• Skill: a big LLM usually knows more and handles hard, open-ended tasks better. An SLM does best on clear, narrow tasks.
• Training: the NVIDIA paper says fine-tuning an SLM for your own task takes only a few GPU-hours.

Many real systems use both. The NVIDIA paper argues that AI agents should send many simple, repeated steps to small models. Big models are kept for the steps that really need them. NeurIPS 2026, one of the biggest AI conferences, even has a December workshop on this idea: "SLM-Agents: 1st Workshop on SLMs for Agentic Systems".

Small language model examples in 2026

Here are well-known small language models you can download for free, as of November 2026. All of them are on Ollama, the tool we use in the next section.

• Gemma 4 E2B and E4B: Google released these on 2 April 2026 under the Apache 2.0 license. They cover over 140 languages and read text, images and audio. Google says they can run fully offline on phones.
• Gemma 4 12B: released by Google on 3 June 2026. Google says it runs on laptops with 16 GB of GPU memory or unified memory. At 12 billion parameters, it sits just above the usual 10 billion line, which shows how fuzzy the line is.
• Gemma 3: comes in five sizes, from 270 million to 27 billion parameters, and covers over 140 languages. The 270M version is only a 292 MB download on Ollama.
• Llama 3.2 1B and 3B: released by Meta on 25 September 2024 and built for phones and other small devices. Eight languages are officially supported, including Hindi.
• Qwen3.5-4B: a 4 billion parameter model from Alibaba's Qwen team, released in 2026 under the Apache 2.0 license. It reads text and images and supports 201 languages and dialects.
• Phi-4-mini: a 3.8 billion parameter model from Microsoft (February 2025), under the MIT license, made for devices with little memory.

A newer Microsoft model, Phi-4-reasoning-vision-15B (March 2026), has 15 billion parameters and reads text and images. Its model card says it is trained mainly on English and is not meant for other languages.

Why do small language models matter?

Small language models matter because they make AI private, cheap and available offline. For a student in India, each of these is practical:

• Privacy: Ollama's FAQ says, "We don't see your prompts or data when you run locally." That matters for medical notes, legal papers or your own unpublished research.
• Cost: after the download, a local model costs nothing per question. There is no subscription and no API bill.
• Offline use: once the model is on your laptop, it works without the internet. This helps on weak or costly mobile data.
• Learning: you can open, test and retrain a small model yourself. You cannot do that with a closed chatbot.

Are there Indian small language models?

Yes. Two Indian small language models are Sarvam-1 from Sarvam AI and PARAM-1 from BharatGen. Many global SLMs also list Hindi and other Indian languages, but listed support does not always mean good quality.

• Sarvam-1: a 2 billion parameter model released on 24 October 2024. It covers English and 10 Indian languages: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil and Telugu. It is a base model, so you must fine-tune it before you can chat with it. Its license allows only non-commercial use.
• PARAM-1: a 2.9 billion parameter model from BharatGen, described in a July 2025 paper. It was trained on Hindi and English only, with 25% of its training data set aside for Indian languages.

Among global models, Llama 3.2 officially supports Hindi, Gemma 3 and Gemma 4 cover over 140 languages, and Qwen3.5 covers 201 languages and dialects. Phi-4-mini's list of 23 languages has none from India. Our advice: test any model in your own language before you trust it. That test can become your first research project.

How to run a small language model on your laptop with Ollama

Ollama is a free app that downloads open models and runs them on your own computer. It works on Windows, macOS and Linux. Get it only from the official site, ollama.com.

1. 1Download and install Ollama from ollama.com/download.
2. 2Open a terminal: PowerShell on Windows, or Terminal on macOS and Linux.
3. 3Type "ollama run gemma3:1b" and press Enter. The first time, Ollama downloads the model (806 MB). Then a chat starts on your computer.
4. 4Ask a question. Type "/bye" to leave the chat.
5. 5Try other models the same way: "ollama run llama3.2:1b" (1.3 GB), "ollama run qwen3.5:4b" (3.3 GB) or "ollama run gemma4:e2b" (4.6 GB).
6. 6Type "ollama ls" to see your models. Type "ollama rm gemma3:1b" to delete one and free up space.

Two tips. First, start with the smallest download and move up. Ollama's own download page warns that large models are slow on a computer without a strong GPU. Second, avoid model names that end in ":cloud", such as "gemma4:cloud". Those run on Ollama's servers, not on your laptop, so your text leaves your computer.

To use a model from Python, install the library with "pip install ollama". The Ollama GitHub page has a short chat example you can copy.

Which is the best small language model?

There is no single best small language model. The best one is the one that does your task well on your hardware. Here is how we would start in November 2026:

• Older laptop: gemma3:1b (806 MB) or llama3.2:1b (1.3 GB).
• Newer laptop: qwen3.5:4b or gemma4:e2b. Both can also read images.
• 16 GB GPU memory: gemma4:12b, which Google says fits laptops with 16 GB of GPU memory or unified memory.
• Coding: qwen2.5-coder, which Ollama offers in 0.5B, 1.5B, 3B and 7B sizes.
• Indian languages: compare Gemma 4 and Qwen3.5 on your own language first.

Rankings change often. Write 20 questions from your own work, run them on two or three models with the same settings, and compare the answers. That small test tells you more than any ranking.

What are the limits of small language models?

The biggest limit of a small language model is wrong facts. A small model has less room to store knowledge, so expect more mistakes on general questions. Google's Gemma 3 model card says it plainly: the models "are not knowledge bases" and "may generate incorrect or outdated factual statements."

Never ask an SLM to find references for your paper. It can invent papers that do not exist, as we explain in our post on fake citations in AI research papers.

Small models work best on clear, narrow jobs: summarise this note, sort these emails, fix this function. The same Gemma card says models "are better at tasks that can be framed with clear prompts and instructions."

Research questions students can work on

Small language models are a good research area for students because you can run real experiments on a laptop. Here are five questions to start with:

• Language test: write 50 questions in Hindi, Bengali or Tamil. Run them on three SLMs. Which models fail, and how do they fail?
• Compression cost: compare a 4-bit version and a full-size version of the same model on one task. How much quality do you lose?
• Small vs big: the NVIDIA paper says small models can replace big ones for many agent steps, and the authors invite critique. Pick one narrow task and test whether a 1B to 4B model matches a big one.
• Fine-tuning: fine-tune a 1B model on one narrow task, such as sorting college notices, and compare it with a bigger general model.
• Made-up facts: ask SLMs for paper references on one topic, then check how many of those papers are real.

Before you start, read the NVIDIA paper with the three-pass method. Then plan a fair test: the same questions, the same settings and a clear score for every model.

At APJ-RAISE, students work on questions like these in small groups, with just a laptop and an idea. Joining is free.