Skip to content
UpupaBitDocs

Matricxon features

Compute

Section titled “Compute”
  • Quantized-native. Weights stay packed in the memory-mapped file and are multiplied directly, with no full dequantized copy in RAM. A native C kernel with OpenMP threads is the default. The Numba kernels are the fallback when there is no C compiler.
  • GPU (experimental). Set MATRICXON_DEVICE=cuda to run on an NVIDIA GPU with weights in bf16 or fp16. With MATRICXON_GPU_WEIGHT_MODE=packed, weights stay quantized in VRAM to save memory at some speed cost. It works for the dense, mixture-of-experts and hybrid architectures, and it refuses to start if no GPU is found. It never falls back to the CPU silently. Treat it as beta. The C kernels are CPU-only.
  • Lazy loading. Loading a model builds its structure only. Weight data is read on first use.
  • Hybrid models. Linear-attention and state-space layers keep a fixed-size recurrent state instead of a growing KV cache.

Caching

Section titled “Caching”
  • Prompt cache. Each model keeps several conversations’ caches, so users sharing a model do not make each other re-read their history. A new prompt reuses the cache that shares its longest prefix.
  • Persistent cache. Caches evicted from RAM, or still held when a model unloads or the server stops, are saved to disk with AES encryption. Returning to a chat does not re-read it. Limits are configurable from the pAIring UI (/api/cache/persistence) or with start variables.

Serving

Section titled “Serving”
  • Concurrent users. Replies on one model run side by side. A scheduler advances every active reply, and for most architectures their decode steps are batched into one forward pass. Batching helps most on memory-bandwidth-bound hardware. On a compute-bound CPU its main effect is that users stream at the same time instead of waiting in line. Mixture-of-experts models and Nemotron-H take one reply per step.
  • Tool calling. Send tools to /api/chat and a call the model writes comes back as a structured message.tool_calls entry with an id. Answer it with a role: "tool" message. Formats are read from the model’s chat template: Qwen/Hermes, Qwen3.5 XML, Mistral, Llama 3.x and Gemma 4.
  • Vision. CLIP and SigLIP-style projector files feed image embeddings into llama (LLaVA), phi2 (moondream2) and qwen35.
  • Safe memory use. A model that would not fit in free RAM, with a safety margin, is refused. Idle models unload after a keep-alive time.
  • Cancel one reply. A request_id on /api/chat stops just that reply. The model and other users’ replies are untouched.