In my experiment, a local language model on a workstation completed a work package I would also hand to a junior developer. That took a worked-out brief and a process that finds errors. Requirements, tests, review and sign-off sat outside the model. Within that frame, its performance came close to a closely supervised junior role. It did not clear that bar consistently yet.
In the software development life cycle (SDLC), the model takes over implementation. An owner approves the work item, executable acceptance tests check the behavior, and an independent auditor examines the result. The merge decision rests with a human again. I have already evaluated this workflow in two larger projects.
The experiment on an M4 Pro
The test machine was a MacBook Pro with an M4 Pro, 64 GB of unified memory and macOS 26.6.2. On it ran Qwen3.8-27B in a 4-bit quantization and Gemma 4 12B-it in 8 bit. Both models were less than two weeks old on measurement day. The runtimes were llama.cpp, MLX, LM Studio and omlx.
The short performance test consisted of a prompt of around 700 tokens and 128 tokens of output. After a warm-up, three measurement runs followed at temperature 0. The tables give the median. All runtimes used GPU acceleration, Flash Attention and a context window of 200,000 tokens.
| Qwen3.8-27B, 4-bit | Generation | Cold prefill | Warm prefix |
|---|---|---|---|
| LM Studio, MLX | 15.0 tok/s | approx. 102 tok/s | approx. 1.9 s, partial |
| omlx | 14.3 tok/s | approx. 100 tok/s | 6.8 s, no reuse observed |
| mlx-lm | 14.2 tok/s | approx. 109 tok/s | n/a |
| llama.cpp | 10.7 tok/s | 94 to 108 tok/s | 0.3 s, full reuse |
| LM Studio, GGUF | 10.2 tok/s | approx. 100 tok/s | 0.3 s, full reuse |
For Qwen, the MLX stack led llama.cpp by around 40 percent in generation. The hybrid model uses Gated DeltaNet, a form of linear attention, in 48 of its 64 layers. At the time of measurement, the Metal implementation in the MLX stack was faster. The result does not transfer to every model architecture, as the second series of measurements shows.
| Gemma 4 12B, 8-bit | Generation | Cold prefill | Warm prefix |
|---|---|---|---|
| omlx | 16.9 tok/s | approx. 230 tok/s | 2.7 to 3.2 s, no reuse observed |
| llama.cpp | 15.6 tok/s | 191 tok/s | 0.39 s, full reuse |
| LM Studio, MLX | 15.4 tok/s | approx. 198 tok/s | 0.34 s, full reuse |
| LM Studio, GGUF | 15.3 tok/s | 194 tok/s | 0.39 s, full reuse |
| mlx-vlm | 15.25 tok/s | approx. 230 tok/s | n/a |
Gemma’s 40 sliding-window layers and eight full-attention layers ran at about the same speed in every runtime. The field stayed within around ten percent. Which runtime to pick therefore also depends on the architecture and the workload profile. With repeated long prefixes, a working prompt cache can outweigh several tokens per second of generation rate.
The M4 Pro provides 273 GB/s of memory bandwidth. Qwen’s weights occupy around 16 GB depending on the format, Gemma’s about 12.7 GB. Both models fit 200,000 tokens of context into 64 GB only because a small share of their layers keeps a KV cache that grows with context length. Measured RSS peaks lay between 13 and 29 GB. The entire series ran on the notebook.
The coding task
The coding side needed a work sample. Qwen and Gemma received the same brief: a complete German marketing landing page as a single HTML file. The brief prescribed the content and the sections. It required inline CSS, a responsive layout and a light and dark mode. External resources were ruled out.
Each run consisted of a single chat-completion request. Qwen worked without thinking mode on a budget of 24,000 tokens. Gemma got 16,000 tokens. Temperature, top-p and top-k followed the model vendors’ recommendations. All six combinations of two models and three server runtimes delivered a complete HTML document.
| Model and runtime | Duration | Output | QA points, max. 50 |
|---|---|---|---|
| Qwen, LM Studio MLX | 17.9 min | 14,578 tokens / 42.7 KB | 43 |
| Qwen, omlx | 18.2 min | 14,612 tokens / 41.7 KB | 38 |
| Qwen, llama.cpp | 25.3 min | 17,123 tokens / 46.7 KB | 37 |
| Gemma, omlx | 4.3 min | 3,989 tokens / 14.9 KB | 22 |
| Gemma, llama.cpp | 5.5 min | 5,414 tokens / 15.7 KB | 21 |
| Gemma, LM Studio MLX | 3.7 min | 3,215 tokens / 11.4 KB | 14 |
For the quality check, the pages were anonymized as A through F. Chromium rendered them at 1,280, 768 and 390 pixels wide in the light and the dark color scheme. A separate Opus model run also examined HTML and CSS. Ten points each were available for marketing effect, visual design, fidelity to the brief, code quality and overall impression. Nobody from the target audience took part in this scoring.
All three Qwen pages reached 37 to 43 points. Gemma came in at 14 to 22. Qwen wrote 745 to 881 words and laid out a recognizable design system. Its pages held 20 to 28 CSS variables and 14 to 37 inline SVGs. No required content was missing. Gemma produced 310 to 355 words, eight to nine CSS variables and at most one SVG. Every Gemma output lacked parts of the brief.
Qwen’s longer outputs contained errors too. One page hid its main heading behind a broken reveal animation. On another, the hero stayed two-column on the phone and got cut off. The Qwen page generated with LM Studio rendered without visible defects in every view checked. The Gemma field showed a transparent sticky header, overly wide tables and visible ASCII transliterations of German umlauts.
The best Qwen output covered all mandatory content. Its operating-system requirement was invented, and the dark mode deviated from the brief. In the SDLC described above, such findings go back to the implementer before sign-off.
There was one run per combination and no human control group. The 43 points therefore do not place Qwen in any ranking against real junior developers. What the experiment demonstrates is the takeover of a bounded work package with a checkable result. General parity across backend code, debugging and longer-running changes would take more tasks and direct comparison runs.
Hardware for around €10,000
My own measurement ran on a mobile M4 Pro with 64 GB. Two days after the benchmark, Apple presented a much larger local platform. The new Mac Studio with M5 Ultra starts at €6,599 in Germany. It can be configured with up to 512 GB of unified memory and reaches 1.2 TB/s of memory bandwidth according to Apple. In the largest configuration, the chip has 36 CPU and 80 GPU cores. The 512 GB model is slated for late October 2026. Apple quotes up to 4.3 times the AI performance. Independent inference measurements for the new machine do not exist yet.
The M5 Max in the same case already starts at €2,999, supports up to 128 GB and reaches up to 614 GB/s depending on configuration. The likewise new Mac mini with M5 Pro starts at $1,699 in the US, offers up to 64 GB and 307 GB/s. Apple claims up to four times faster prompt processing than the M4 Pro for one selected LM Studio test. This number too needs independent reproduction.
Nvidia’s DGX Spark has a different focus. For a $4,699 list price, the small machine offers 128 GB of coherent memory, a GB10 CPU-GPU combination and 273 GB/s of memory bandwidth. NVIDIA quotes up to one petaFLOP of FP4 compute with sparsity and inference for models up to 200 billion parameters. Two units can be coupled via ConnectX-7. At US list price, two Sparks stay just under $9,400 before tax.
Theoretical compute numbers are of limited use as a speed measure for language models. During token-by-token generation, moving the weights out of memory often caps the throughput. An independent comparison by Tom’s Hardware in July 2026 found higher decode rates on the M4 Max than on GB10 and AMD Strix Halo, while GB10 led in prompt processing. The DGX Spark opens up other models and modes of operation through its 128 GB and the CUDA stack. The Mac delivers plenty of memory bandwidth for dense models and a simple local desktop setup.
For around €10,000 you get a Mac Studio M5 Ultra in the base version or, at US list price, two DGX Sparks before tax. After the purchase come running costs for power, storage, replacement hardware and software-stack upkeep. The human work on specification and verification belongs in that calculation as well.
Documented model progress
Qwen’s model page allows a comparison of two generations in the same local size class. Qwen3.6-27B appeared on April 22, 2026, Qwen3.8-27B on August 14. In the results published by the vendor, the score on Terminal Bench 2.1 rose from 63.4 to 73.0, on SWE-bench Pro from 53.5 to 61.7. On DeepSWE 1.1 it grew from 13.3 to 42.2. Some tests ran in a Claude Code harness. The numbers document rapid improvement on these three tests, but they remain vendor figures.
An analysis of 20,574 real agent sessions from 1,639 repositories found seven recurring forms of misalignment. In 91.49 percent of the visible resolutions, the user had to step in with an explicit correction (arXiv 2605.29442, May 28, 2026). A supervised run has to budget for such corrections.
SpecBench shows a wide gap for specification work. The benchmark uses design discussions from Kubernetes, React, Rust, TVM and vLLM. The best agent tested caught 44.4 percent of the flaws the maintainers had found (arXiv 2605.30314, May 28, 2026).
A separate specification phase measurably improves implementation. SpecFirst splits the reconstruction of a program into behavior exploration and subsequent implementation. Across 200 tasks and four models, test pass rates rose by 6.9 to 21.3 percentage points over a combined pass (arXiv 2607.27167, July 29, 2026). In this SDLC, executable requirements give the implementer that groundwork.
The limits stay narrow on hard domain tasks. MDArena tested 50 real tasks from molecular-dynamics workflows. The best configuration examined solved 24 of them completely. On the hard tasks, every configuration stayed close to zero (arXiv 2608.02642, July 31, 2026). SWE-bench Science likewise comes in below 50 percent pass@1 for the best agent across 119 tasks from scientific repositories (arXiv 2608.19799, August 20, 2026).
These findings limit my claim to prepared implementation. System design and the clarification of incomplete requirements stay with experienced developers. For changes with domain risk, the implementation needs its supervision too.
My answer to the title question
For the measured part of the work, my answer today is “almost”. The local model turned a worked-out brief into a usable web page. At the same time, the spread between 37 and 43 points for Qwen shows how much reliability is still missing. The experiment demonstrates the performance in a closely supervised implementation role. It does not stand in for a whole position.
Factoring in the current pace, I expect local models on the hardware then available to clear this bar for prepared standard tasks within a year at the latest. The differences between the two Qwen generations and the new machine generation already point that way.