A working guide for architects and the engineers who work with them: What's actually inside an inference SoC, why data movement (not math) dominates the design, and how teams really build AI chips.
AI chip architecture is the organization of compute, memory, and interconnects inside silicon built to run neural networks.
Six subsystems appear in nearly every AI SoC, but two of them are the most critical.
The tensor engine: parallel MAC arrays that execute the matrix math of neural networks.
Run the operating system, applications, and control logic; coordinate the rest of the chip.
Signal processing for audio, radar, and image pipelines in legacy heterogeneous designs.
Registers → local SRAM → L2 Memory → external memory.
AXI buses and Network-on-Chip (NoC) that move data between blocks.
The data sources inference exists to serve: camera pipelines, radar/LiDAR interfaces, and other connectivity.
A multiply-accumulate costs almost nothing, but fetching its operands from external memory costs ~200 times more power than from local registers. Multiply that asymmetry by billions of operations per frame, and it becomes clear the key to running AI isn't compute, it's memory architecture.
Strong designs are built on a hierarchy of ever-cheaper and more plentiful memory: local SRAM and registers are very small, but fast, in-silicon, and close to the compute. L2 memory is still in-silicon but a step removed from the compute cores and more plentiful. System memory is off-chip, plentiful, but comparatively much slower. The compiler choreographs transfers between memory stages deterministically, staging data next to compute before it's needed.
This is why bolting an accelerator to a memory bus seldom works well: every operator the accelerator can't run forces a round-trip across the interconnect and bleeds your power budget out through the fabric.
Approximate energy cost of reading data, by where it lives.
Deeper dive: SoC memory architecture for on-device AI
Teams creating modern SoCs assemble them from licensed, proven IP blueprints for the challenging blocks, adding their own differentiators as they go.
01
Which models and at what resolution, latency, and batch size? Today's networks aren’t the only target: what will the chip need to run three years after tape-out?
02
"PPA" stands for performance, power, and area, and refer to the performance of the chip, the power it consumes, and the silicon area it requires. Local/edge devices live on milliwatt budgets and dollar-scale die costs, and those factors dictate the manufacturing process and how many TOPS you can afford.
03
Size the MAC array and the SRAM that feeds it concurrently. This is where the decision to build or license the technology forks: most teams will license proven AI processor IP rather than design a compute core from scratch.
04
Compile models using the vendor’s compiler and measure performance before locking the architecture. Without a working toolchain, stated TOPS specifications are meaningless.
05
Assemble the IP blocks over standard interfaces (AXI), close timing, and verify: functional, performance, and in the case of automotive deployments, functional safety (ISO 26262/ASIL).
06
Release the silicon into the wild. It ships once but models change constantly and the architecture’s programmability dictates the useful life of the chip. If it’s not able to easily support new models, that life could be very short.
Related reading: where to place the NPU in your SoC and future-proofing AI silicon.
Every AI SoC team now faces a fork in the road: partition intelligence across specialized blocks, or run everything on a unified, programmable core like Quadric's Chimera.
Chimera GPNPU IP gives your SoC a unified, fully programmable AI core: 1 to 6,912 TOPS on a single toolchain with no fallback. Evaluate it against your own models.
