Matricxon kernels and GPU

Packed weights

GGUF files store weights in compressed (quantized) blocks. By default Matricxon keeps them that way and multiplies the packed blocks directly from the memory-mapped file. There is no full uncompressed copy in RAM.

Mode	RAM use	Speed
Packed (default)	Close to the file size	About 2 to 4 times slower per step than uncompressed
Uncompressed	Several times the file size	Fastest, if it fits

For example, Ministral 3B in Q4_K_M needs about 2.6 GB packed and about 7.5 GB uncompressed. To use uncompressed weights, set MATRICXON_ENABLE_QUANTIZED_NATIVE_COMPUTE=false.

Native C kernels

The fast path is a small C library in app/native/src. It covers the quantized matrix multiply and, with fused ops, the hot steps of each token: RMSNorm, rotary embedding, attention over the cache and the sampler’s draw.

• Built on your machine. It compiles on first start with the system C compiler. There is no CMake and no extra package.
• Tuned for your CPU. It is built with -march=native, which is why it is never shipped prebuilt.
• Rebuilt when needed. Any source file newer than the library triggers a rebuild. The new file is renamed into place, so two instances starting together never load a half-written library.
• Multi-threaded. It uses OpenMP when available. Without OpenMP it still builds, on one thread, and stays far faster than Numba.

Platform	Compiler	Notes
Linux	gcc with -fopenmp	The main, tested platform
macOS	Apple clang, plus Homebrew libomp for threads	Not yet built on a real Mac. scripts/install_mac.sh sets it up

Tip

With the C kernels, Qwen 3.5 4B in Q4_K_M decodes about 10 times faster than with Numba, and fits on a 15 GB machine.

Numba fallback

When there is no C compiler, or the build fails, Matricxon uses its Numba kernels. Every C kernel has a Numba twin that is kept in sync and tested. They are much slower, but they run anywhere Python does. To pick them on purpose, set MATRICXON_GEMV_BACKEND=numba.

The C kernels also fall back to Numba for the rare cases they do not cover, such as an unsupported quantization type or an unusual tensor shape.

GPU mode (experimental)

Set MATRICXON_DEVICE=cuda (or cuda:1 for a second card) to run on an NVIDIA GPU.

• Weights. MATRICXON_GPU_WEIGHT_MODE=dequantized (the default) unpacks weights to bf16 or fp16 in VRAM. It is the fastest. packed keeps them compressed to save VRAM, at some speed cost.
• Coverage. Dense, mixture-of-experts and hybrid architectures.
• No silent fallback. If no GPU is found, Matricxon refuses to start instead of quietly running on the CPU.
• CPU only parts. The C kernels and fused ops are CPU code. On a GPU, PyTorch does the work.

Not tested on real hardware

GPU mode was written without a GPU on the development machine. It has been checked through the CPU path and shape tests only. Treat it as beta and report what you find.

Keeping the CPU cool

Long runs keep every core busy. MATRICXON_TORCH_THREADS sets how many threads are used. Fewer threads run cooler, at a small speed cost. On laptops, leaving one or two cores free helps.

Settings

Variable	Default	Meaning
MATRICXON_ENABLE_QUANTIZED_NATIVE_COMPUTE	true	Multiply packed weights directly
MATRICXON_GEMV_BACKEND	native	native (C) or numba
MATRICXON_ENABLE_FUSED_OPS	true	One native call for the hot per-token steps
MATRICXON_DEVICE	cpu	cpu, cuda or cuda:N
MATRICXON_GPU_WEIGHT_MODE	dequantized	dequantized or packed
MATRICXON_TORCH_THREADS	all cores	CPU threads