Rock Matrix open-sources EdgeRock-4B, a small reasoning model with a fixed-size KV cache

Rock Matrix released the weights and code of EdgeRock-4B, a 4B text-and-image reasoning model, on GitHub, Hugging Face and ModelScope, with NF4, INT8 and FP8 versions. A paper is on arXiv (2610.10118). The announcement was relayed on X by TestingCatalog on October 31.

The model is based on Qwen3.5-4B. Rock Matrix converted its eight full-attention layers to a fixed-capacity history design that combines recurrent state, local attention and "Rock Memory Anchors" (RMA). The last 1,024 tokens are kept in full, and important older details are saved to a separate bank. The attention KV cache stays capped at 72 MiB per conversation, where Qwen3.5-4B's grows to 8 GiB at 256K tokens.

Claims from Rock Matrix, not independently measured:

• 82.9% mean pass@1 on AIME 2024–2026, against 18.8% for the strongest bounded-state model it tested.
• 94.1% query accuracy on a long-context test with inputs from 16K to 130K tokens, against 0.4% with RMA turned off.
• On a MacBook Pro (M5 Pro, 28GB RAM) running Rock Matrix's PyTorch runtime on the Mac's GPU, it answered a simple multiplication step by step, started answering after 4.6 seconds and generated about 9 tokens per second.