UpupaBitUpupaBit
A small, readable LLM runtimeAn inference runtime written from scratch in Python and PyTorch, with optional native C kernels. It runs GGUF models locally and serves them over an HTTP API, with no llama.cpp, ggml or vLLM code.
Cluster map in pAIring, shown with demo data (BETA).
Weights stay packed and are multiplied directly, with 25 quantization types.
Run on an NVIDIA GPU, with weights kept packed in VRAM if needed (experimental).
Chats keep their own caches, so users do not re-read long histories.
Evicted caches are saved with AES encryption and restored on return.
Replies run side by side, with batched decoding where supported.
Structured tool calls for Qwen, Mistral, Llama 3.x and Gemma 4.
Image understanding for LLaVA, moondream2 and Qwen 3.5.
Nodes share models and caches and agree on membership with Paxos (BETA).
Models that do not fit in RAM are refused, and idle ones unload.
git clone https://github.com/upupabit/matricxon.git
cd matricxon
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8420More options and configuration are in the documentation.