aiGalen Guan

Training Neural Networks on Apple's Neural Engine: How maderix Cracked Apple's Most Locked-Down Chip

Training Neural Networks on Apple's Neural Engine: How maderix Cracked Apple's Most Locked-Down Chip

Apple's Neural Engine (ANE) is the most opaque component in M-series chips. Since its debut in the A11 in 2017, the ANE has had exactly one official entry point: CoreML inference. Apple has never published the ANE's instruction set architecture (ISA), never provided a direct programming interface, and barely mentions it in developer documentation — it's just a number in the spec sheet: "M4: 38 TOPS."

On February 28, 2026, an engineer named Manjeet Singh (GitHub: maderix) published "Inside the M4 Apple Neural Engine, Part 1: Reverse Engineering" on Substack. This wasn't an ordinary blog post — he directly called _ANEClient and _ANECompiler, two private Apple APIs, and ran a complete Transformer training loop on the ANE.

As of late July 2026, the GitHub repository has 7,196 stars and 963 forks. The HN discussion on Part 1 accumulated 376 points and 117 comments. MIT license.

The most surprising detail is the author's opening note: Claude Opus 4.6 was his co-developer on this reverse-engineering effort — "we" refers to Manjeet Singh the human and Claude the AI working as a pair. One partner supplied intuition and direction; the other wrote code and experiments.

1. Why Apple Locks Down the ANE

The ANE is neither a GPU nor a CPU. It's a graph execution engine — a fixed-function accelerator that takes a compiled neural network graph and executes the entire thing as one atomic operation. You don't issue individual multiply-accumulate instructions. You submit a compiled program describing an entire computation graph, and the hardware runs it end-to-end.

Apple's design intent is clear: ANE = inference only. They provide CoreML as the sole entry point, abstracting away hardware details — and any possibility of training. The API-level restriction landed before the hardware capability ceiling.

Why not open up training? Several possible explanations:

  • Stability first. The ANE software stack isn't optimized for frequent weight updates — see the compilation cache dilemma below.
  • Fragmentation fear. Once custom compute graphs are allowed, the iOS/macOS API surface could fragment.
  • Competitive moat. The ANE is a differentiator for Apple Silicon. Open programmability means competitors can analyze the chip design more easily.

But maderix proved the third one is the most questionable: the barrier has always been software, not hardware.

2. Full-Stack Reverse Engineering: From CoreML to Silicon

Over several days, maderix mapped the entire ANE software stack from top to bottom:

┌─────────────────────────────────────────────────┐
│  CoreML (Public API)                             │
│  mlmodel → compile → infer                       │
├─────────────────────────────────────────────────┤
│  Espresso (Apple's internal inference engine)    │
│  Graph optimization, operator scheduling          │
├─────────────────────────────────────────────────┤
│  AppleNeuralEngine.framework                    │
│  _ANEClient, _ANEModel, _ANERequest,            │
│  _ANEIOSurfaceObject, _ANEInMemoryModel...      │
│  40+ private classes                             │
├─────────────────────────────────────────────────┤
│  ANECompiler.framework                          │
│  MIL text → E5 binary (FlatBuffer)               │
├─────────────────────────────────────────────────┤
│  IOKit (Kernel Driver)                           │
│  AppleANEDriver → ANE hardware registers         │
├─────────────────────────────────────────────────┤
│  M4 ANE (H16G)                                  │
│  16 cores, 127-deep queue, independent DVFS      │
└─────────────────────────────────────────────────┘

The key insight: CoreML isn't the only door. The _ANEClient class in AppleNeuralEngine.framework directly exposes the full compile → load → evaluate pipeline. CoreML is just a convenience layer on top.

The discovery process combined four techniques:

  1. Class-dump to extract private framework headers
  2. LLDB dynamic debugging to trace CoreML's internal call paths
  3. IOKit probing to identify hardware register mappings and DVFS (Dynamic Voltage/Frequency Scaling) channels
  4. Assembly-level analysis to reverse the E5 binary format

Over 40 private classes were discovered, including _ANEInMemoryModelDescriptor — a class that accepts MIL text directly in memory, completely bypassing the filesystem-based compilation path.

3. MIL: Apple's Intermediate Language for ANE

CoreML doesn't send neural networks to the ANE in ONNX or protobuf format. It uses MIL (Machine Learning Intermediate Language) — a typed SSA (Static Single Assignment) intermediate representation:

function main(float<fp32>[1, 1024, 1, 1024] A,
              float<fp32>[1, 1024, 1, 1024] B)
              -> (float<fp32>[1, 1024, 1, 1024]) {
  %1 = conv(input=A, weight=B, stride=[1, 1],
            pad_type=valid, groups=1,
            dilation=[1, 1])
      // shape: [1, 1024, 1, 1024]
  function_return %1
}

MIL is surprisingly readable. Every value carries type annotations (precision + shape). Operations are function calls with named keyword arguments. Tensor layout follows ANE's native NCDHW + Interleave format: [Batch, Channels, Depth, Height, Width]. A 1024×1024 matrix becomes [1, 1024, 1, 1024].

The most critical finding: the ANE's native compute primitive is convolution (conv), not matrix multiplication. Expressing matmul as a 1×1 convolution unlocks significantly higher throughput — validated in Part 2's benchmarks.

E5 Binary: Microcode, Not Machine Code

When ANECompiler processes a MIL program, it produces an E5 binary — a FlatBuffer-structured compilation artifact. A striking fact:

  • 1024×1024 matmul → 2,688 bytes
  • 128×128 matmul → 2,680 bytes

Nearly identical. The E5 binary isn't encoding the matrix multiplication algorithm — it's encoding a parameterized program whose behavior is controlled by runtime tensor descriptors. This "microcode" is more like a configuration than traditional machine code. The ANE hardware has a small set of fixed compute primitives (conv, matmul, element-wise), and the E5 binary describes how to chain them together.

4. Making Training Work: Dynamic-Weight Kernels and Six Engineering Decisions

With direct ANE access secured, maderix moved to the training implementation. The goal: run complete Transformer training on the ANE — forward + backward + optimizer updates. Two models successfully trained:

Model Params ms/step Layers Kernels/layer
Stories110M (12L, dim=768, MHA) 109M 91 ms 12 6 (MHA)
Qwen3-0.6B (28L, dim=1024, GQA) 596M 412 ms 28 10 (GQA)

The Core Trick: Weights Packed into Spatial Dimensions

Training's biggest obstacle: recompiling the MIL program after every weight update is too slow (first compile ~20-40ms). maderix's solution is elegant:

  • Pack activations and weights as contiguous slices of a single spatial input dimension
  • Separate them inside the MIL kernel via slice/reshape operations
  • When weights change, replace weight data in the IOSurface without recompiling

This means compile once, evaluate infinitely — each step only swaps weight I/O buffers.

Six Engineering Decisions

Several design choices in the training code deserve detailed attention:

1. Channel-First (NCDHW Format). ANE IOSurface's native layout is [N, C, D, H, W], while PyTorch and most deep learning frameworks use [N, H, W, C] or [B, T, H, D]. Working directly in ANE's native format eliminates all transpose overhead.

2. GPU↔ANE Zero-Copy Pipeline. IOSurface is the same shared-memory mechanism used for GPU textures. maderix runs prefill on the GPU (processing the full prompt in one shot), then passes intermediate state to the ANE for per-token decoding via a shared IOSurfaceRef:

GPU Prefill (Stories110M, seq=256): 6.7ms
         ↓ (zero-copy IOSurface)
ANE Decode: 1.9ms
         ↓
Total: 8.8ms

3. CPU-ANE Parallelism. Weight gradient sgemm operations run on the CPU in parallel with ANE forward/backward evaluations on a serial dispatch queue. ANE's wait_until_completed is pushed into the next step's forward pass for maximum overlap.

4. RMSNorm Folded into Kernels. RMSNorm forward and backward are folded into MIL kernels as reduce_sum + pow + mul operations rather than running as separate ANE evaluations.

5. Intermediate Exposure Bypasses CPU Recomputation. ANE forward kernels expose Q, K, V, attention scores, and hidden states via concat outputs that backward kernels read directly — eliminating all CPU-side recomputation.

6. Process Pooling Bypasses the ~119 Compile Limit. The ANE compiler limits each process to roughly 119 compilations. maderix works around this by wrapping the training loop in exec() restarts — each new process inherits IOSurface file descriptors and seamlessly resumes training state.

5. The Performance Truth: 5-9% Utilization — But That's Not Failure

This is where maderix is most honest. The README states directly:

Training works, but utilization is low (~5-9% of peak) with significant engineering challenges remaining. Many element-wise operations still fall back to CPU. This does not replace GPU training for anything beyond small research models today.

M4 ANE peak throughput:

Precision Peak TOPS Config
FP16 18.6 128x conv, 512ch, 64×64
INT8 W8A8 35.1 128x conv, 512ch, 64×64

INT8 quantization delivers 1.88× speedup (via MIL quantize/dequantize ops that halve L2 SRAM bandwidth), but actual training utilization remains at 5-9%. Why?

Three root causes:

  1. Causal attention is decomposed. ANE hardware ignores causal_mask in SDPA ops. Causal attention is forcibly decomposed into Q@K^T (ANE) → mask+softmax (CPU) → scores@V (ANE) — three stages with heavy CPU-ANE synchronization overhead.
  2. Compiler resource leaks. The ANE compiler leaks resources after repeated compilations — requiring exec() restart workarounds.
  3. fp16 backward underflow. Backward matmuls frequently underflow in fp16 — requiring global loss scaling (scale=65536) to maintain gradient precision.

But the existence of these limitations is precisely what makes this project most valuable. It answers, with experimental data, a question that was never answered before: if you gave the ANE training capability, where are the real bottlenecks?

6. Community Reaction and the Author's Stance

The 117 HN comments reveal several interesting perspectives:

  • A former Apple Xcode team member noted they know the lengths Apple goes to make reverse engineering difficult, calling this "excellent work."
  • A debate erupted over whether Apple actively obfuscates code — one commenter insisted the ANE compiler code "definitely doesn't employ obfuscation techniques," while another former Apple employee listed control-flow flattening, mixed boolean-arithmetic transformations, and other techniques.
  • The project also attracted legitimate skepticism about ANE training's practical value — a fair portion of those 7K stars are "this is cool but useless."

maderix himself is remarkably clear-headed. The README states:

I don't intend to grow this into a large community project. My focus is on original research (compiler infrastructure for edge AI optimization), and maintaining an open-source framework takes time away from that.

He isn't building a product. He's publishing a research paper — where the code is the paper. The MIT license and "Fork it, build on it" invitation are an extension of this research-first posture.

7. Scoring and Judgment

Dimension Score Notes
Technical novelty 10/10 First complete training on ANE, breaking Apple's long-standing restriction
Engineering quality 7/10 Clever designs (dynamic-weight kernels, zero-copy pipeline), but low utilization
Documentation completeness 9/10 Three Substack deep-dives + exhaustive README + commented source
Practical value 4/10 Doesn't replace any GPU training today, can't be used for inference deployment
Research value 10/10 Systematically answers questions about ANE programmability and real performance boundaries
Ecosystem impact 8/10 7.2K stars signals deep developer frustration with locked-down hardware

The scoring table itself tells the story: a project with 4/10 practical value earned 10/10 research value. maderix/ANE's worth isn't "you can now train models on ANE" — you absolutely cannot, and shouldn't. Its worth is proving "training is possible" and systematically documenting every current bottleneck.

Our Take

This project is worth reading carefully, but not worth using directly.

"Reading carefully" because maderix's three-part Substack series is currently the most complete, systematic public documentation of ANE reverse engineering. If you have any curiosity about Apple Silicon internals, this is required reading.

"Not using directly" because training a 110M model on an M4 MacBook at 5-9% utilization makes no sense versus just using a GPU. This project's actual users are future NPU toolchain developers — the pitfalls maderix documented today are the holes that MLX or CoreML teams will need to fill tomorrow.

A deeper question: how will Apple respond? Historically, Apple has two reactions to private API abuse — either crack down (iOS jailbreaking) or ignore (unofficial hardware access on Mac). The ANE sits between these extremes: it's hardware on a consumer device, and doesn't directly implicate the iOS security model. But Apple could make certain ANE compiler paths unavailable in a future macOS update — especially the in-memory compilation path.

As maderix himself wrote:

The goal was to demonstrate that training on the Apple Neural Engine — and potentially other NPUs — is possible, and that the barrier has always been software support, not hardware capability.

That sentence may carry more weight than all the code in the GitHub repository combined.

References

  1. maderix (Manjeet Singh) — ANE GitHub Repository (7.2K+ stars, MIT license)
  2. Manjeet Singh — Inside the M4 Apple Neural Engine, Part 1: Reverse Engineering (2026-02-28, Substack)
  3. Manjeet Singh — Inside the M4 Apple Neural Engine, Part 2: ANE Benchmarks (2026-03, Substack)
  4. Manjeet Singh — Inside the M4 Apple Neural Engine, Part 3: Training (2026-03, Substack)
  5. Hacker News — Inside the M4 Apple Neural Engine, Part 1: Reverse Engineering (376 points, 117 comments, 2026-03-01)
  6. hollance — neural-engine: Community ANE documentation
  7. mdaiter — ane: Early ANE reverse-engineering
  8. Asahi Linux Project — eiln/ane: ANE Linux driver