Open Models, SLMs, Retrieval, and Cost

SLM
I read the blog post “Build and Buy: Why the Smartest AI Teams Do Both” by Lambda AI. It set off a flurry of ideas that I’ve been accumulating for the past month.
Author

Vishal Bakshi

Published

August 31, 2026

I read the blog post “Build and Buy: Why the Smartest AI Teams Do Both” by Lambda AI. It set off a flurry of ideas that I’ve been accumulating for the past month..

First, a quick summary of the article (it’s a five-minute read, so I recommend pausing and reading it):

They talk about how open-weight models (now in the trillions of total, not necessarily active, parameters) can handle 90% of tasks for 90% of the people in your company, and the frontier-class models are required for only that final 10%. Owning the model on your own hardware gives you more control, reliable availability, data security and flexibility in how you want to use and fine-tune the model. You can also rent GPUs from cloud compute providers like Lambda AI for tasks that don’t fit in the open-weight/frontier split.

I’ll explore four questions in this article.

What are grand total self-hosting costs by open model size?

Suppose your company has 100 people using your self-hosted open weight model. How much would it cost (GPUs, cooling, electricity, etc.) based on model size?

I asked ChatGPT this question and gave it a range of model sizes. Here are its back-of-the-envelope estimates. Talk to your local inference expert for accurate costs ;) (for reference, the Lambda AI article quoted $350k to self-host Kimi K3, excluding cooling and electricity costs).

  • Kimi K3 2.8T / 104B active (1× 8×B300 server): $600–700k deployed CAPEX
  • Qwen3.8-2.4T-A95B (2.4T total, 95B active; 1× 8×B300 server): $600–700k
  • Qwen3.8-Flash-Next (~180B total / 6B active; 1× 4×B200 server) $250–350k
  • Qwen3.8-27B (1-2× 80-96GB GPUs): $30–60k
  • Qwen3-4B (1-2 × 24GB-class GPU): $10–20k
  • Qwen3-1.7B (1-2 × modest 12–24GB GPU): $5–15k

As the model size decreases, the GPU share of the cost decreases as well.

Note: GPU count doesn’t scale with the number of users, necessarily, as long as your peak number of users’ requests fit in a batch and you’re okay with any request queues. (See the HuggingFace articles on continuous batching and asynchronous batching for nitty gritty details).

What level of fully open machine intelligence is accessible to orgs?

What do the costs above tell us about what kinds of machine intelligence are truly accessible and available to most people in a truly “you own your own tech” way? What does that mean about how the rest of us (aka the GPU poor) think about using local LLMs?

This is far too broad a topic for me to fully grasp, let alone cover in this article, as it depends on your particular use case. What I will say is that there are a lot of clear signals in practice about the power of small models. Simon Willison writes about local LLMs extensively in his blog. There are also plenty of anecdotes, like the following, that make me pause and think about my underestimation of small models:

I have personally done a number of experiments on tiny and small models. Now, these experiments are very, very small in scope, so I’ve only scratched a small region of the surface. But what I can say is that every time I start a task with a small model, expecting it to fail, it succeeds. Some examples:

I’ve always been fascinated by small models. When it comes to local models, this is probably where I’m going to spend most of my time experimenting. The challenge of being extremely compute-constrained is exciting to me. It also helps me boil down my use of LLMs to the absolute necessary steps in a workflow.

What role does efficient retrieval play in unlocking SLM utility?

Given the recent Tom Arsen/Omar Khattab experiments that showed us that fine-tuned multi-vector models can outperform general retrievers (an experiment that Youngjoon Jang extended with asymmetric quantization) what do their results mean for pairing multi-vector retrievers with SLMs?

Multi-vector embedding models, aka information retrieval using late interaction, is one of the most interesting areas of ML research. Case study after case study shows that when you embed documents at the token level, index them efficiently, and use MaxSim for scoring similarity between query and document tokens, you get much more accurate results. With smaller models, your context window is smaller as well, so you can’t just stuff a codebase’s worth of tokens in there. One experiment I want to try sooner than later is asking the question: How well can a small model search a codebase if you give it a ColBERT-style index for retrieval?

Note: read the original ColBERT papers (1, 2, 3), they are canon. I also have a blog post (as part of an unfinished series on ColBERT fundamentals) walking through Omar’s fantastic tweet thread in 2023.

What if I’m wrong?

A final humbling thought that’s absoluely necessary to have when working in ML: what if I’m wrong? I hope we can get away with smaller models, but after seeing the following posts I’m not so sure!

Blog post: Machine Studying by Jacob Xiaochen Li, Rick Battle and Omar Khattab.

Some excerpts after scanning the X post and the blog post:

In the same way, language models knowing a fact internally, without tool calls, is meaningful. The first reason is that we obviously care about speed; you’d much rather get an answer immediately than have the model think a long time to be sure of its answer or browse the web. A second reason is that there are some things that are simply best learned via backpropagation over lots of data. If you ask about how people generally think of the Shambhala music festival, you’d rather a large language model give you an aggregate opinion based on all the data on the internet, than get a regurgitation of the first three reviews that show up in a web search. A third reason is that having to do a lot of work to find an answer is not as reliable as already knowing the answer. While this does not have to be true in theory, it is probably true in practice, at least for now. If you have to re-look up facts or redo a mathematical derivation all the time there is a higher chance of mistakes, which can compound in a long-horizon task.

Agents can grep, read files, and run code at test time, so why not just spend more inference tokens per question? But this conflates having access to the corpus with developing deep expertise: you wouldn’t hire any of us as a lawyer just because we can Google the legal literature very intelligently. At minimum, what makes a lawyer a good lawyer is knowing what to look for, where to look, and what to do with a passage after they find it.

Taking this into account, my hypothesis would be something like: In order to make small models + retrieval + system prompt work well, the human SME has to curate the “what to look for, where to look and what to do with a passage after they find it” portion of the task. Ultimately, even if successful, this endeavor might just be an interesting proof of concept and not something that can be scaled to an organization level. But I think if we have reasonable expectations for which tasks we can apply SLMs to, the future is exceptionally bright and relatively affordable.