Matricxon overview

Matricxon is a small, readable LLM inference runtime written from scratch in Python and PyTorch. Optional native C kernels speed up dequantization and parts of the forward pass. It runs GGUF models locally for chat, text generation, embeddings and vision-language models, and serves them through an HTTP API that tools like pAIring can use.

It uses no llama.cpp, ggml or vLLM code or bindings. File parsing, dequantization, the transformer forward passes, caching, sampling and scheduling are all written here. It runs on CPUs, and on GPUs in BETA.

The project started inside the pAIring UI. Working with Ollama showed a need for more advanced options, such as memory management for concurrent workloads. Everything is managed from the pAIring UI, where Matricxon is the main inference engine.

Cluster mode (BETA)

Several Matricxon nodes can share models and chat caches and agree on membership through built-in Paxos consensus. The cluster is managed from pAIring, with live node status on an interactive map. It is designed for hundreds of nodes, but only small clusters have been tested so far.

One node or many

Matricxon can run on the same node as pAIring (a single node), or on one or more separate servers that do the heavy model work. It can also form a cluster. See Deployment layouts for the difference and the options.

What’s inside

Part	What it does
GGUF reader	Parses metadata and tensors, and memory-maps the file
Dequantization	25 quantization types, in Numba and native C
Architectures	Transformer, MoE, hybrid and encoder forward passes
Runtime	Tokenizers, KV caches, sampler, generation loop and scheduler
Vision	Image encoders that feed vision-language models
Server	The HTTP API and the Hugging Face model puller

License

Matricxon is released under GNU AGPL-3.0. You can run, modify and self-host it for personal or internal use. If you offer a modified version to others as a hosted service, you share those changes back.