World Models at the Edge, in Real Time: V-JEPA 2.1 on the Quadric GPNPU

A world model running on the Quadric GPNPU: it watched one demonstration of a robot arm, and for each new run it finds the matching moment and says how far through the task the robot is. Nothing was trained.

Robots are about to get world models: networks that learn how the physical world behaves by watching video, the way language models learned language by reading. The catch is that a robot can't send its camera feed to a data center and wait for an answer. The model has to run on the robot, on a battery, as fast as the camera produces frames.

So can a world model run in real time at the edge? For Meta's V-JEPA 2.1 on the Quadric GPNPU, the answer is yes: 43.5 milliseconds per clip and 23 clips a second on a single core. The more surprising part is how we got there. On Thursday morning the model didn't run on our chip at all. By Saturday it did, and agents had done all of the work. The people involved, me included, mostly told them where to look next.

What a world model is

Large language models learned language by predicting the next word, over and over, across most of the text on the internet. It turns out you learn a great deal about the world that way. You do not, however, learn that a cup knocked off a table falls, or that a hand reaching for a plate is about to pick it up. That kind of knowledge lives in video, and the models that try to learn it are called world models.

Meta's V-JEPA is one of the more interesting attempts. The model watches video with chunks of it hidden and learns to predict what was in the hidden chunks. The key idea, which Yann LeCun has been arguing for years, is that it does not predict the pixels. Predicting every pixel wastes most of the model's effort on things that don't matter, like the exact flutter of leaves in the background. V-JEPA predicts a description of the hidden part instead: a list of numbers that captures what is there and how it is moving. Trained that way on over a million hours of video, it ends up understanding motion, objects and simple physics well enough that Meta has used it to plan robot arm movements it was never trained on.

The model we ported is V-JEPA 2.1 ViT-B, the 80-million-parameter version of the encoder, which is the part that does the watching. It takes a clip of 16 frames at 256 × 256 pixels, cuts it into 2,048 small patches (two frames by 16 by 16 pixels each), and produces 768 numbers for every patch. That's about 254 billion multiply-accumulates per clip, and a lot of it is attention across all 2,048 patches at once.

That last point is why this model is a good test for a chip. Attention over long sequences is a very different workload from the convolutions that most edge AI accelerators were designed around, and it's where many of them struggle.

Ninety minutes

On the morning of Thursday, October 22nd, I asked an agent to port V-JEPA to our GPNPU, and to use as many agents as it needed.

What made that a reasonable request is a set of engineering skills we've written down over the past year: how to read a profile, how the memory hierarchy behaves, how to write kernels against the Chimera Compute Library, how to prove a change is correct. They ship to customers through our Developer Studio MCP server, and they encode an order of operations that matters more than any single trick. You first write a PyTorch "twin" of the model in exactly the form the chip will compute it, and check it against the original. Then you pick a precision recipe by measuring, not guessing. Then you write the kernels, and every kernel carries a CPU reference that the simulator's output is checked against.

The agents split that work up and ran it in parallel. One built the twin and proved it matched Meta's original, including catching a deliberately broken copy, which is how you know the check is real. Another chose the precision: 8-bit weights, 16-bit floating-point activations, and our 32-bit fixed-point format for normalization and softmax. A classifier trained on the model's output gave the same answer for the quantized model as for the float one on every clip we tested. A third wrote the kernels and assembled them into one program.

Before lunch, a full transformer block was running on our cycle-accurate instruction-set simulator and matching its reference, and a draft pull request was open. A little after 1 pm the whole 12-block encoder had run. From the first prompt to a working model was about an hour and a half.

A few years ago, that sentence would have read very differently. Bringing up a new network on a new accelerator was weeks of work: someone studied the model, hand-wrote the operators the compiler didn't know, and chased numerical mismatches one layer at a time before anyone could even ask how fast it was. Getting it to a respectable fraction of the chip's peak was a separate project after that.

The first version was correct, but it wasn't fast. The agents spent the afternoon cutting a quarter of its cycles, with every step checked to produce identical output. Then we pointed the same code at QC-Ultra, our larger configuration (32 × 32 processing elements, 16 multiply-accumulate units in each, at 1.4 GHz), which took a one-line change. It also surfaced two real bugs in our own attention kernel, and the numerical checks caught both, which is the whole point of having them. That night, the full model ran on QC-Ultra in 98.9 milliseconds per clip.

A respectable result. Then we put it in the arena.

Into the arena

Earlier this month I wrote about the Kernel Arena, the internal game where engineers and their agents compete to make our kernels faster, with every result checked by an independent oracle. The V-JEPA task went up that night: the patch embedding plus one full transformer block on QC-Ultra, starting at 13,458,058 cycles.

Most arena tasks demand outputs that are byte-for-byte identical to the reference. This one didn't, deliberately. A world model's arithmetic has legitimate freedom in it (the order you sum things in, how you compute an exponential), so the task allowed a very strict error budget instead: on three different clips, every output token had to stay within a hair of a float64 reference, tighter than the starting kernel itself in places. Before publishing, we tried to cheat it: an older build that was faster but subtly wrong, a version that skipped some of the attention, and one that just wrote zeros. The checker rejected all three.

Then I went to bed. Here is what the board looked like over the next day and a half.

By 5:36 Friday morning, the first entry had cut 35% of the cycles. By 9 am the same engineer had pushed it to 40%. That afternoon a second engineer came in at 55%, and just after midnight on Saturday the first one answered at 57%. Both of them were working with agents.

The interesting thing is what they found. Most of it is the kind of work a compiler does generically and an expert does specifically. The patching of the clip was folded straight into the first matrix multiply, so video goes from memory into the multiply units without an intermediate copy. The positional encoding moved into the tail end of the matrix multiply that produces the queries and keys, and so did the residual additions. Data that used to be shuffled one 16-bit value at a time moved in 32-bit pairs. None of that changes the math, and the outputs stayed identical.

It's worth being clear about how little we put into this. The task was open for a day and a half, it drew a handful of submissions from two engineers, and nobody pointed a fleet of agents at it. The Qwen3 prefill task in the last post went through 67 accepted improvements; this one went through a few. So the result above isn't where V-JEPA's optimization ends, and with more people and more agents it would have come faster. It's a demonstration of what a small, unglamorous effort in the arena is worth, and the board is still open.

Bringing it home

A fast kernel on a leaderboard only helps if it makes it back into the product. An orchestrating agent split the leader's changes into independent pieces, handed each one to an implementing agent, and checked every piece on the simulator before merging it into our SDK example. The arena code was written to win a game, so part of the job was making it readable. Where hand-written assembly was worth real cycles, it stayed, in a few small named helpers.

Here is where the full model ended up, measured on the simulator for QC-Ultra at 1.4 GHz:

Is it real-time?

That's the first question anyone building a robot asks, so here is the direct answer. Real-time for video means keeping up with the camera, and V-JEPA looks at 16 frames at a time. On one QC-Ultra core it takes 43.5 milliseconds from the last frame of a clip arriving to its description being ready, about 1.3 frames at 30 frames per second, and it can process 23 clips a second.

A common way to run a video model is a sliding window that starts a new clip every 8 frames. At 30 frames per second that needs fewer than four clips a second, so one core has about six times the headroom, enough for roughly six cameras. Starting a fresh clip on every other frame needs 15 a second, which also fits. A fresh 16-frame clip on every single frame would need 30 a second, which one core can't do yet.

How close to the limit

Every chip has a theoretical maximum: the number of multiply-accumulates it can do per cycle if every unit is busy every cycle, the way a car has a redline. QC-Ultra at 16-bit precision can do 8,192 per cycle. V-JEPA now averages 4,170, which is 51% of the fp16 peak across the entire model, including all the parts that aren't matrix multiplies.

For a vision transformer at 2,048 tokens, half of peak is a good number. It's not a finished one. Attention is still about a third of the cycles, and the matrix multiplies are 768 columns wide on a 1,024-element array, so a quarter of the array waits on some of them. Given a day and a half and two players, the arena got us here; given more of either, I expect it to keep going.

How it stacks up

Qualcomm publishes V-JEPA latencies for its Snapdragon NPUs through AI Hub, so it's natural to compare. The comparison needs care. Qualcomm runs a different and larger model, V-JEPA 2 ViT-L, on fewer tokens (1,024 rather than 2,048), in 16-bit float. Latency alone would be apples to oranges, so the fair measure is utilization: how much of each chip's rated throughput the model actually uses.

The Snapdragon X2 Elite, rated at 80 int8 TOPS, runs its model in 167 ms, which works out to about 5% of its rated throughput. QC-Ultra, rated at 45.9 TOPS, uses about 25% of its own. Put differently, it gets through about 2.7 times as much work per second on a little over half the rated TOPS. Qualcomm only publishes an fp16 rating for one of these chips (the QCS8550, at a quarter of its int8 rating). Applying that same ratio to the others, which is the generous reading for them, the X2 Elite uses about 22% of its fp16 peak, against our 51%.

Seeing the world, for real

Simulators are deterministic, which is what makes the arena work, but at some point you want to see it on hardware. On Friday we ran the full model on an FPGA prototype of the chip, in its QC-Perf configuration. Every output word matched the simulator, all 1.57 million of them. (The prototype runs at 60 MHz, so it's a check of correctness, not of speed.)

Then we gave it something to look at. V-JEPA's output, with nothing added, is a description of the scene, and you can compare two descriptions and ask how similar they are. So we showed the chip one recorded run of a real robot arm picking up a plate and setting it down, and then nine runs it had never seen. For every second and a half of each new run, the chip found the most similar moment in the one demonstration. Nobody told it what the steps were, and nothing was trained. It placed the robot in the task with an average error of 6%, against 28% for guessing.

That's what the video at the top of this post shows, at real speed, for three of the nine runs.

A chip that can watch a robot and know where it is in a task, from a single example, is a chip that sees the world a little.

What's next

The obvious next step is Meta's most impressive V-JEPA result: planning robot arm movements toward a goal image. It pairs a 1-billion-parameter encoder with a second model that imagines what each possible movement would do, and runs it hundreds of times per step. That's the world model we want running on our GPNPU next.

Still, I think the more important thing here is the loop, not the model. A new model arrives. The skills port it to the chip in an afternoon, correct but slow. The arena makes it fast overnight, sometimes finding things none of us would have written. And the ideas behind those wins flow back into the compiler, so the next model starts from a faster floor. Not long ago, each of those steps was weeks of expert time.

World models are going to keep getting better at understanding the world. When the next one shows up, I expect we'll have it running on our GPNPU before the end of the day. What none of us can know yet is how fast the arena will make it.