Two Bets on Standing Still, and a Dark Horse

I first read about Taalas in February, in a Hacker News thread I opened expecting another inference startup with a benchmark chart. Instead it was a Toronto company that had taken Llama 3.1 8B and etched it into silicon - not loaded it, etched it, the weights laid down as physical transistors at the fab.

My first thought wasn't that this was new, it was that we've done this before. Groq had been working the same problem for years and stopped one step short, keeping the weights on the chip but still rewritable. The same question sits under both: how much of a model are you willing to make permanent in exchange for speed? Inside eight months, Nvidia had taken one of them and AMD the other.

That's the part worth paying attention to. Something quite significant is happening behind the curtains right about now, and it's easy to miss because it reads as chip news. Every previous time a workload left general-purpose hardware, it permanently changed who could afford to run it, and AI inference is somewhere in the middle of the same move. How far it goes isn't a fixed quantity waiting to be measured, because every part of the problem is still being designed - the models, the silicon, and, as it turns out, the way a model produces output at all.

We've seen this movie

Bitcoin mining went from CPUs to GPUs to FPGAs to purpose-built ASICs - application-specific integrated circuits, chips that do one job and nothing else - in four years. Each step traded flexibility for efficiency, and each was irreversible. When the first Avalon units shipped in 2013, GPU mining didn't get slower, it got pointless.

The pattern is everywhere now, and invisible precisely because it worked. YouTube transcodes video on its own Argos chips, which Google says beat its previous optimized software setup by 20 to 33 times. Network switching is Broadcom silicon. Your phone has a fixed-function block for H.265 decode, an ISP (image signal processor) for the camera, a Secure Enclave that does nothing else. It's what happens when a workload settles down enough to be worth casting in metal - and that condition is where this whole story ends up.

The wall they're all attacking

The thing GPUs spend most of their energy on during inference isn't math, it's moving weights. Every layer, every token, the weights come out of HBM (high-bandwidth memory, the stacked DRAM beside the GPU die) and through the compute units, then the next layer's follow. A 32-layer model does that 32 times to produce a single token, then starts over for the next. That's the memory wall - inference is limited by bandwidth, not compute, and HBM is the most expensive component on an AI server.

A chip that holds its weights on-die needs no HBM at all. Its memory is SRAM, printed on the logic wafer at a fab that isn't one of the memory plants whose 2027 output is already sold - so pushing inference onto this kind of silicon spreads the manufacturing load onto capacity that isn't already spoken for, and takes some pressure off the RAM and VRAM the rest of us are trying to buy. How much pressure is the open part, since the GPUs these chips get paired with still carry HBM.

Two companies have built exactly that, and they part ways on how much of the model they were willing to freeze to do it. Everything else follows from that.

The same prompt-to-token job on both. On a GPU the weights make the trip, on the specialized chips the data does.

Bet one: the model is the silicon

Taalas puts the weights in a mask-ROM fabric - read-only memory, written once during manufacture - where a single transistor stores four bits and performs the multiply related to it. Data flows through the chip's physical layers in sequence and never goes back out to external memory.

The numbers, all vendor-run and none independently verified:

• 16,960 tokens/second per user on Llama 3.1 8B, against roughly 230 on an H200. People who hit the public demo reported 14,000-17,000.
• 0.015 J/token, against Groq's self-reported 1-3 and roughly 10-30 for H100-class systems, per EE Times.
• $0.0075 per million tokens, against a typical $0.20-0.50.
• 815 mm² on TSMC N6, 53 billion transistors, about 250 W per card. One chip holds the whole 8B model.

"Etched in silicon" sounds like the chip is fixed the moment it leaves the fab. That's how most of the coverage reads, and how I read it at first, but it isn't quite true. The die has two regions. The mask-ROM fabric holds the frozen weights. Beside it sits an SRAM recall fabric - rewritable memory on the same chip - holding the KV cache and LoRA adapters, small sets of extra weights that steer a model toward a domain without retraining it. Those swap while the chip runs: the legal adapter gives you legal language, the clinical one medical, same etched model underneath.

Nor is the frozen half a full redesign. A chip is built from a stack of patterned layers, each printed through its own photomask, and cutting that mask set is the expensive part - tooling paid for before a single wafer moves. Taalas reuses the same base stack every time, with only two of the roughly hundred layers carrying the weights. It quotes about two months from new weights to shipping cards, where a full custom design runs six or more. Cost is harder to pin down, since published mask prices vary by an order of magnitude. The most specific estimate anyone has put on Taalas itself is zach.be's: about 30 chip variants and $100M in masks for a DeepSeek-R1-class model, so roughly $3M per variant, needing a year of service life to pay back.

So it's a spectrum, not a binary, and $3M is a line item for anyone already serving inference at scale. But it is still silicon. Two months is design time, not delivery time - Taalas had already slipped its own schedule by a year, and fabrication, validation and swapping cards out of racks come after it. And the adapters steer the hardwired weights rather than replacing them, so the chip picks up your terminology but not a capability the etched model never had. A revised base model, a different tokenizer or a new architecture means new masks and new cards.

Bet two: the model is loadable

Groq attacked the same wall from the other side, and the more I look at both, the more I think the hedge is the interesting part. The LPU (language processing unit) also has no HBM. Weights live in on-chip SRAM, placed by a compiler that works out every instruction and data movement in advance, so nothing is decided at runtime and latency is known before a job starts. Any model can run on it, and switching takes a recompile and a reload - minutes, no new hardware.

What that costs is capacity. First-generation LPUs carry 230 MB of SRAM per chip, which won't hold even a 7B model at 8-bit, and a 70B model spans 576 chips - interconnect, power and floor space for every one. The LP30 die improves the ratio without changing the problem, at roughly 500 MB, 256 to a rack for about 128 GB of on-chip SRAM. Still hundreds of chips to hold what Taalas fits on one die.

What they're actually betting on

Taalas bets the weights freeze - that a model stays useful long enough to earn back its masks, which its own economics put at about a year. Groq bets only the architecture freezes - transformers stay transformers, while the weights inside them keep changing every few months. Taalas is expensive in silicon per model. Groq is expensive in silicon per change of model.

Both bets have been endorsed with money. Nvidia licensed Groq's architecture for $20 billion in December, and AMD agreed to acquire Taalas just over seven months later - hedges against their own product being the wrong shape for most of the work. More interesting than either deal is that both buyers described the same system design. Answering a prompt has two phases: reading the input, which is parallel math and exactly what a GPU is good at, and writing the reply one word at a time, each word needing the whole model again before the next can start. Both put the GPU on the first phase and the specialized chip on the second. Two opposite philosophies about what to freeze, one identical division of labour.

Where the analogy breaks

Bitcoin got its ASICs because SHA-256 is a specification, standardized in 2002, years before Bitcoin existed. It will never change - not "probably won't" but can't, because changing it would be a different currency. Video codecs went the same way: standardize the format, wait, then put it in silicon.

The precondition was never "this workload is popular." It was "this workload is finished."

LLMs are pretty fluid tools, and I don't think they're anywhere near finished. Llama 3.1 8B came out in July 2024, so HC1 shipped in February 2026 baking a nineteen-month-old model, with better ones already out in its size class. Silicon takes long enough to arrive holding something the world has moved past.

The usual counter is that the pace will settle. I half agree. The breakthroughs do look like they're slowing - the transformer has held for nine years, mixture-of-experts is standard, genuinely new architectural ideas arrive further apart. But releases won't slow much, and releases are what breaks the chip. Llama 3.1 to 3.2 to 3.3, Qwen 2.5 to 3, a post-training refresh, a tokenizer swap. Every one keeps the architecture identical and changes the numbers, and the numbers are what was cast in metal.

And most customers need flexibility first of all. Ask a phone assistant "what's this charge on my bill?" and it runs speech recognition, a vision model on the screenshot, a language model to reason about it, then speech synthesis - four models, three modalities, one question, and the mix shifts as better ones arrive. A chip that does one of those extremely fast and none of the others solves very little.

The dark horse nobody is buying

Every argument so far leaves one assumption standing: a model writes one word at a time. Both designs are built on it. But nothing forces a model to.

Diffusion language models don't. Google's Gemini Diffusion "generates entire blocks of tokens at once," refining a whole rough draft in place rather than extending a sentence, and reports 1,479 tokens per second of sampling. One sweep of the weights per block instead of one per word. Google publishes neither the model's size nor the hardware it ran on, so that figure won't line up against the ones above, and the demo is invite-only. The one you can put a prompt into today is Inception's Mercury, selling the same idea as a product - by its own claim, 5 to 7 times the throughput at 70% lower cost per task.

One token per sweep against a few sweeps per block. The pass count is illustrative, Google doesn't publish the real one.

That's the same trade the chips make, only in software. Diffusion spends extra arithmetic - several passes over a block rather than one pass per token - to avoid re-fetching weights. On a GPU that arithmetic is nearly free, because during single-stream decode the compute units sit idle waiting on memory. Filling idle silicon is precisely the inefficiency Taalas and Groq spent a chip design each to remove.

That extra arithmetic buys more than speed. An autoregressive model cannot unsay a word - once a token is out it conditions everything that follows, and the best the model can do is contradict itself further down the page. A diffusion model revisits the same positions across sweeps, which is why Google claims it "corrects errors during generation for more consistent outputs." Reasoning models chase the same thing from the opposite direction, improving an answer by generating more tokens rather than by going back over the ones they have. Every route to a better answer costs compute per delivered token.

So the two don't add up the way you'd hope. Groq can recompile for a diffusion model and collect little, its weights being local already. Taalas would have to etch one, and its compute is physically fixed at one multiplier per etched weight, so more passes per token runs the wrong way entirely. The hardware that gains most from diffusion is the GPU that was supposed to hand the job off. I'm reasoning from the architectures here - nobody has published diffusion numbers on either chip.

It's a dark horse in the market's eyes and a favourite in mine. The numbers don't back me up yet: Gemini Diffusion still trails Gemini 2.0 Flash-Lite on GPQA Diamond, 40.4% against 56.5%, and on Global MMLU, 69.1% against 79.0%. But it doesn't have to win. It only has to make the sequential bottleneck smaller than the thing built to remove it.

Unless everything moves toward everything else

That's the argument as it stands today, and today is not the interesting part. The line between what gets frozen and what stays loadable isn't handed down by nature. It's a design decision, and every side of it is still being designed.

Hardware is already bending. HC1 used a proprietary 3-bit format with some 6-bit weights mixed in - a bespoke chip for a bespoke encoding. HC2 moves to MXFP4, an industry-standard 4-bit format, so the silicon is adapting to what models already do rather than the other way round. Give the SRAM half a bigger share of the die and you buy more adapter capacity and longer context on the same frozen base.

Models can move too, and mixture-of-experts is the obvious opening. A model that routes each token through a few of many experts is already modular in the way silicon likes. Nothing stops someone etching the experts that get hit constantly and leaving the long tail loadable, or designing a model with a deliberately stable trunk and a replaceable top. Nobody has shipped this - I'm speculating - but it's the sort of thing that gets built when there's a chip waiting for it.

And either camp can build for the dark horse. A chip designed from the start for block-parallel generation is a different chip, not an impossible one. The question was never whether hardware or software wins. It's where all of this converges, and what the optimal path turns out to be.

Where hardwiring does earn its place

None of this makes the Taalas bet a bad one. It makes it a narrow one, and narrow isn't an insult.

The functions that are genuinely finished are the boring ones. Speech recognition. Translation. OCR. Text embeddings for retrieval. Content moderation. These already run as small dedicated models, at volumes that would pay off a mask set easily, and what we ask of them has barely moved in years. Nobody is going to demand a personality update for their OCR. Diffusion doesn't threaten them either. Text embeddings aren't produced a word at a time in the first place, and nothing about a moderation classifier is waiting on the sequential bottleneck.

Taalas got first silicon out with 24 people and $30 million, so this isn't structurally reserved for people who own datacenters. The physics rewards small directly too: an 8B model fits on one die, a trillion-parameter one needs about fifty of the next-generation chips wired in sequence. I still think small dedicated models are where most of this ends up, and the ones worth casting in metal are the narrowest, not the ones you talk to.

So

Casting a model into silicon is a bet on how long it can stand still. For SHA-256 the bet was free, because the answer is forever. For H.265 it cost a decade of standards work. For a general-purpose LLM in 2026, it's a bet that a nineteen-month-old set of weights is close enough for the year it takes to earn the masks back - hedged by a strip of SRAM that lets you change its accent but not its mind.

Taalas will probably win that bet on a narrow function. On the broad one, Groq is likely to win by refusing to make it - on today's models, with today's silicon. Or somebody shaped like either of them. But Nvidia's $20 billion on Groq and AMD's purchase of Taalas both went to the premise that the expensive part of inference is fetching weights one token at a time, and the cheapest way to be wrong about that is for the models to stop writing one token at a time. Neither buyer has bought a hedge against that one.