Article · · 17 min read
A 125B model on an RTX 3060, provided you have plenty of RAM. Testing Strata
Strata runs Qwen3.8-Flash-Next, a 125-billion-parameter model, on a computer with a single graphics card. It is an open engine under the MIT licence, published by an author who goes by Niko1221 and unaffiliated with the Qwen team. A 12 GB card is enough because most of the model sits in RAM: with such a card the tested variant needs about 60 GB of RAM according to the documentation, and 64 GB in practice. I tested it on a computer with a GeForce RTX 3060 (12 GB), an Intel Xeon E5-2699 v4 processor from 2016 and 121 GB of RAM.
There, Strata wrote answers at 43–54 tokens per second, slowest in Polish, and read long prompts at about 800 tokens per second. The same model file in llama.cpp, with the experts in RAM, wrote 16 tokens per second and read at most 226. On 17 checkable tasks in Polish, Strata answered correctly in 88% of attempts without thinking mode and in 97% with it. Two smaller models run on the same computer for comparison, gemma-4-E4B and Qwen3.6-27B, answered correctly in 41–71% of attempts and made up facts.
How 125 billion parameters fit into 12 GB
Qwen3.8-Flash-Next is a mixture-of-experts (MoE) model. Each of its 48 layers has 512 "experts", separate blocks of the network, and the router, a small network at the start of the layer, picks 10 of them for every token plus one shared expert. According to the model card, about 6 billion parameters work on any one token, less than 5% of the total. The remaining weights do not have to sit on the graphics card; they only have to be reachable quickly enough.
The model card counts 125 billion parameters for the network itself and adds two components, 180 billion in the files altogether. The first is a table of n-gram embeddings (51 billion parameters): ready-made vectors of numbers for 20 million frequent pairs and triples of tokens, which the model looks up at the second layer instead of computing them. The second is the MTP layer (multi-token prediction, 4 billion parameters), described below.
Strata spreads all of this over three levels of memory:
- The graphics card holds the attention layers (the parts of the network that compare the current token with the earlier text), the routers, the shared experts, the context cache (KV cache) and a stock of the most frequently used experts. My card fitted 1,891 of them (3.9 GB), 8% of all of them. Even so, in the requests from my tests 55–69% of the experts chosen by the router were already on the card; the engine swaps them as it works, and in the first request after start-up only 29% were on the card.
- RAM holds all 24,576 experts, 50 GB in the tested variant. An expert that is not on the card is computed by the processor, in parallel with the card. On my machine this was done by 20 of the 22 cores (Strata leaves one to the system, and the controlling thread runs on another), using AVX2 instructions because this processor has no AVX-512.
- The SSD holds the n-gram table (29 GB). The model reads a few rows of it per token. When RAM is very large (for this variant from 107 GB, according to the documentation), Strata on Linux loads it into memory after all, because by its own measurements this shortens prompt reading by 18% at 4K tokens and by 2% at 98K. That was the case on my machine, which is why the engine process took 83 GB of RAM.
The weights are compressed by quantisation, storing numbers with fewer bits. The variant name IQ3_S denotes a compression scheme in which an expert takes about 3.3 bits per weight on average instead of 16. Strata's authors write that this variant "matches the full model on the published tests", but they do not say which tests. Numbers come from their own measurement on 5,000 tokens of text (code, documentation, prose in several languages): IQ3_S picks the same most likely next token as the 8-bit variant at 87.7% of positions, and perplexity, a measure of how surprised the model is by a text, rises from 4.052 to 4.097, by about 1%. A different token need not change the meaning of an answer, but the documentation does not show how often it does.
The MTP layer proposes up to three next tokens, and the full model checks them in one pass and keeps the correct ones. This is a form of speculative decoding: the answer is the same, just faster. On 12–16 GB cards Strata also skips, by default, rarely chosen experts that happen not to be on the card. According to the documentation, on an RTX 3060 the model then picks the same most likely token as with the exact computation at 97.7% of positions. All my quality tests ran with this setting; I did not measure its effect on the task results, only how much the two speed-ups add to speed (below).
Dan Woods's project, which ran Qwen 397B on a MacBook [Polski] with 48 GB of memory, read the experts from disk on the fly. Strata keeps them in RAM: it needs far more of it, but it does not wait for the disk.
How much memory you need
Strata runs the same model at several levels of compression. The installer adds RAM to the card's memory, subtracts 5 GB for the system and proposes a variant based on the result:
| Installer threshold (RAM + VRAM − 5 GB) | Variant | Occupies (RAM + VRAM) | Notes |
|---|---|---|---|
| 32 GB | Coder IQ1_M | approx. 32 GB | half of the experts, chosen for programming; weaker outside code |
| 40 GB | IQ2_XS | 39.2 GB | |
| 48 GB | IQ3_XXS | 47.0 GB | |
| 64 GB | IQ3_S | 54.8 GB | the slowest and most accurate of the 2–3-bit variants; tested here |
The sizes in the table do not include the n-gram table, which stays on disk by default. The threshold is higher than the memory occupied because the experts have to fit in RAM alone: the documentation recommends RAM equal to the size of the experts plus about 10 GB, about 60 GB for IQ3_S. With a 12 GB card the 64 GB threshold means 57 GB of RAM, and the documentation says IQ3_S runs with 64 GB of RAM when few other programs are open. On computers whose memory is shared by the processor and the graphics, such as a Mac with an Apple processor, what counts is its total size.
At a threshold of 96 GB and above there are also 4–8-bit variants prepared by Unsloth, a company that publishes compressed versions of open models: from UD-IQ4_XS (94 GB to download, the experts take about 72 GB in RAM) to Q8_0 (188 GB to download, about 136 GB of RAM for the experts). The installer still proposes IQ3_S then, and a larger variant has to be chosen by hand; I stayed with IQ3_S.
The card needs at least 12 GB of memory (NVIDIA from the RTX 20 series, newer Radeons, Intel Arc), and the processor must support AVX2. A bigger card speeds the model up because it holds more experts, but it does not reduce the need for RAM.
The model natively supports 262,144 tokens of context. The installer sets 32K for cards below 14 GB, but the limit can be raised at start-up. From 64K, Strata keeps most of the context memory in RAM and leaves only a 32K window on the card. With the 128K I set, this took 1.7 GB of RAM, and the card's memory use did not change.
Installation on Ubuntu
I chose installation through the strata command in the terminal, which needs no administrator rights:
curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh | sh
strata launch --plan --model iq3_s # shows the plan, changes nothing
strata launch --model iq3_s --context 128k --yes --no-tui
The install script checks the archive's SHA-256 sum and the release signature, and writes everything to the home directory. Downloading the engine and the model with its add-ons (90 GB altogether) took about 3 hours 20 minutes at 4–9 MB/s. Loading the model takes about a minute. The server provides APIs compatible with OpenAI (http://127.0.0.1:8080/v1) and Anthropic (http://127.0.0.1:8080), plus a chat in the browser.
At start-up Strata warned that only 200–450 MB of memory was left free on the card, because the desktop uses part of it, and suggested --vram-reserve-mib 1024. With the card that full, the driver may move GPU memory to RAM, and writing then drops to a few tokens per second. That did not happen in my tests.
Measurement conditions
- Ubuntu 26.04.1, kernel 7.0, NVIDIA driver 595.99, Strata 0.2.3 with engine 0.1.48 for CUDA 13, IQ3_S variant, 128K context, graphical desktop running.
- DDR4 RAM (the Xeon E5 v4 has four memory channels), card on PCIe 3.0 x16; Strata measured transfers to the card at 10.5 GB/s.
- Speeds come from the
timingsfield in the server's responses; Strata and llama.cpp report them the same way. Writing speed includes tokens accepted from MTP. The share of experts on the card, the size of their memory and the number of cores come from the engine log. - Strata and llama.cpp count tokens the same way: the same text gave 7,911 and 7,922 tokens in them (the difference is the chat template), so speeds in tokens per second can be compared.
Writing speed and what drives it
Four prompts: a Python function, an essay in Polish and one in English (about 400 words each) and a list of 30 names with reasons. I sent each twice, at temperature 0, after a fresh server start; the table shows the mean of both runs in tokens per second:
| Configuration | code | essay PL | essay EN | list |
|---|---|---|---|---|
| Strata, default settings | 53.8 | 42.9 | 50.3 | 49.2 |
| Strata without MTP | 40.3 | 42.4 | 42.1 | 42.2 |
| Strata without skipping rare experts | 45.5 | 37.8 | 43.2 | 43.1 |
llama.cpp, all experts in RAM (--cpu-moe) | 15.6 | 15.6 | 15.6 | 15.7 |
llama.cpp, experts of 2 layers on the card (--n-cpu-moe 46 -ub 2048) | 15.9 | 16.0 | 16.0 | 15.9 |
In ordinary use, at the default temperatures (0.7 without thinking, 1.0 with thinking), Strata wrote at a similar speed: in the quality tests the median for answers longer than 150 tokens was 52 tokens per second without thinking and 49.5 with thinking.
MTP speeds up writing code by a third, English prose and the list by 17–19%, and Polish prose hardly at all (1%). In Polish text the model accepted about 40% of the proposed tokens, in code over 70%, and checking the rejected proposals also takes time. Strata's documentation promises 1.6–1.8 times; on this hardware and these texts the gain was smaller. Skipping rare experts gave 13–18%, in line with the documentation.
Llama.cpp (version b10809) loaded the same file unchanged but wrote three times more slowly. In this configuration it did not use the MTP layer, and the processor computed the experts. Moving the experts of two layers to the card and processing the prompt in batches of 2,048 tokens (the -ub and -b parameters) barely changed writing speed; more layers did not fit on the card.
Long context
For the context measurements I took a random slice of Strata's English documentation, 1K to 115K tokens long and different every time, so that the engine could not reuse a stored prompt. In the middle I inserted one Polish sentence containing a password and asked for the password and for one sentence about what the documentation covers. I measured six lengths twice each, and four of them once more with a request for a longer summary, 16 attempts in all.
Reading the prompt, the prefill, took from 4 s for 1K tokens to 140 s for 115K. From about 17K tokens the speed was 790–850 tokens per second; with short prompts it is lower because a fixed overhead dominates. Llama.cpp read the same kind of text at 85–91 tokens per second, and with batches of 2,048 tokens at 226. When reading the prompt, Strata sends experts to the card over PCIe, so PCIe 3.0 is probably the slowest link here. Strata's documentation gives about 1,620 tokens per second at a 32K prompt for an RTX 5070 with a Ryzen 5 7600 processor, twice as much as on my machine. That computer, however, also has a different processor and memory, and the measurement was made with an older engine version, so the cause of the difference cannot be pinned down.
Writing hardly depended on context length: for a 500-token summary, 46 tokens per second with 1.3K tokens of context and 42 with 115K. Only 12 of the model's 48 layers are attention layers, and even they read a limited number of context blocks for each token (up to 2,048 tokens according to the model card). The other 36 layers (Gated DeltaNet) do not scan the context; they update a state of fixed size.
The model gave the password in 15 of 16 attempts, including at 115K tokens. In one attempt, at 66K tokens, it refused: it took the password sentence for a prompt injection, an instruction hidden in the data. A Polish sentence in English text is easy to spot, though, so I added a harder test, in five separate prompts from 9K to 122K tokens long. I inserted two English sentences in the style of the documentation at one quarter and three quarters of the text: "the nightly regression tests run on a machine called ORCA-7" and "the ORCA-7 machine is located in the Frankfurt office". Asked in which city the machine with the nightly tests is located, the model answered correctly in all five attempts.
Answer quality
I prepared 20 tasks in Polish because I wanted to test the model in a language other than English; results in English may differ. In 17 tasks the answer can be checked automatically: arithmetic (multiplication, a discount and VAT, counting days in a leap year), logic puzzles, factual questions, an answer in JSON format, an instruction with several constraints at once, a question about a non-existent novel by Bolesław Prus (a 19th-century Polish writer, the author of "Lalka", The Doll) and three Python functions, which I ran against tests. The hardest of them writes numbers from 0 to 999,999 in Polish words; Polish numerals change form depending on the number ("dwa tysiące", "pięć tysięcy"), and the prompt required "tysiąc" rather than "jeden tysiąc" (one thousand). I judged three tasks (a technical explanation, an email to a client, translating idioms) by reading the answers.
By default Strata uses the temperatures from the model card, so answers differ between runs. I ran the set three times without thinking and twice with thinking. I ran the reference models through llama.cpp in Q4_K_M variants (about 4.8 bits per weight), with the temperatures from their model cards: gemma-4-E4B three times without thinking and twice with thinking, Qwen3.6-27B twice and only without thinking, because at its speed a run with thinking would, by my estimate, take over an hour.
| Model and mode | Correct answers on 17 tasks | Time for the whole set (20 tasks) |
|---|---|---|
| Strata, Qwen3.8-Flash-Next, no thinking (3 runs) | 45 of 51 (88%) | 83–96 s |
| Strata, Qwen3.8-Flash-Next, thinking (2 runs) | 33 of 34 (97%) | 412–534 s |
| gemma-4-E4B, no thinking (3 runs) | 21 of 51 (41%) | 78–88 s |
| gemma-4-E4B, thinking (2 runs) | 21 of 34 (62%) | 215–216 s |
| Qwen3.6-27B, no thinking (2 runs) | 24 of 34 (71%) | 658–690 s |
Without thinking, Strata made mistakes on four tasks, most often on two. It solved the puzzle about three people, some of whom always lie, once in three attempts, each time answering in under a second. The function writing numbers in words failed the tests in two attempts out of three: once it wrote "jeden tysiąc" although the prompt explicitly required "tysiąc", once it dropped the word "tysięcy" (thousands). In one run it also got the multiplication wrong (40,701,304 instead of 40,701,104), and the function converting "1h 30m 15s" into seconds rejected upper-case letters although the prompt told it to accept them. With thinking, the only mistake was "jeden tysiąc" again. Asked about Prus's "Szklany ogród" (The Glass Garden), the model replied in all five attempts that no such novel exists.
Thinking costs a lot of time. The set took five times longer, and simple programming tasks took the most: the duration function mentioned above took 101 s and 4,712 tokens instead of 6 s, with the same test result.
Gemma-4-E4B (5 GB) fits entirely on the card and writes 75 tokens per second, one and a half times faster than Strata. Without thinking it answered correctly in 21 of 51 attempts, with thinking in 21 of 34. In all five runs it got the multiplication, the letter count and the question about the system Linux was created on wrong, and each time it invented a plot for the non-existent Prus novel, with a different hero every time. In the first run without thinking, two of its programming answers were cut off by the 1,200-token limit because the model kept adding its own tests to the code.
Qwen3.6-27B (17 GB) is a dense model, meaning it computes all its weights for every token. On a 12 GB card part of them stays in RAM, so it wrote 5 tokens per second and the set took 11 minutes. It answered correctly in 24 of 34 attempts. In both runs it got the publication year of "Lalka", the liars puzzle and the numbers-in-words function wrong. Asked about "Szklany ogród", both times it replied that Prus did not write it, but added invented facts, for example crediting Prus with "W pustyni i w puszczy" (In Desert and Wilderness), a novel by Henryk Sienkiewicz.
The maker reports, among others, 91.7 on GPQA Diamond and 62.5 on SWE-bench Pro for the full Qwen3.8-Flash-Next model; these are Qwen's results for the uncompressed model, which my set does not check.
Claude Code and other agents
The strata connect command prints and writes the configuration for OpenCode, Codex CLI, Claude Code, Hermes Agent (an open-source agent from Nous Research), OpenClaw and a few other tools. With the --check option it sends one question and one tool call (tool calling, the model asking for a function to be run) the way the given application does. For Claude Code and Hermes Agent both checks passed in 3–4 seconds.
I gave Claude Code a real task (version 2.1.310, in --bare mode, without plugins and MCP servers): a small Python project with a bug in word counting, an unimplemented top_words function and six tests, five of which failed. The agent ran the tests, fixed the bug, wrote the function and ran the tests again. This took 19 steps and 133 seconds, all tests passed, and the test file was left untouched. One inconsistency remained in the code: a helper function is annotated as returning a list but returns a generator. This is one attempt at one simple task, not an assessment of work on a larger repository.
The first step sent 4.6K tokens of system prompt and tool descriptions, which Strata read in 6.3 s. In the following steps 95–99% of the prompt was already stored, so the engine read only the appended part in 0.3–1.1 s, even though the conversation grew to 8.3K tokens. Strata keeps the prompts it has read for several conversations in RAM (up to 8 GB on my machine), but it handles them one after another: by default it answers one request at a time. According to Strata's documentation, the full Claude Code prompt without --bare mode has 15.6K tokens and Hermes Agent's 12.9K, so for work with agents it is worth raising the context above the default 32K.
When it pays off
The possibility of doing part of the work entirely locally, on a model smaller than the leading cloud models, is tempting, and this test shows what it takes. Gemma-4-E4B, which fits entirely on the card, answered fastest but was wrong in more than half of its attempts and made up facts. Qwen3.8-Flash-Next through Strata answered correctly in 88% of attempts without thinking at 43–54 tokens per second, and in 97% with thinking, although the set then took five times longer. In return it needs about 60 GB of RAM and almost the whole card. Qwen3.6-27B, a dense model that does not fit on the card, did worse than Strata and was ten times slower: in this comparison, speed was decided by how many weights have to be computed for one token, not by the size of the whole model.
On a computer with plenty of RAM, a local model therefore makes sense for data I do not want to send to the cloud, for finding information in long documents (it reads 115K tokens in about 2.5 minutes), for simpler programming tasks and for agents that add little to the conversation from step to step. It will do worse when several people use the server at once, when the graphics card is needed for something else, or when every task needs thinking. There are no per-token fees, but a computer under load uses electricity: the card alone drew about 140 W while writing; I did not measure the whole computer.
Sources
- Strata, repository and documentation, Niko1221, version 0.2.3, engine 0.1.48: README,
docs/MODELS.md,docs/DETAILS.md,docs/QUANTS.md,docs/HOW_IT_WORKS.md. - Qwen3.8-Flash-Next, model card, Qwen.
- llama.cpp, ggml-org, version b10809.
- Hermes Agent, Nous Research.