Matricxon prompt cache

pAIring sends the whole conversation with every message. Without a cache, the model would re-read all of it each turn, and long chats would get slower and slower. Matricxon keeps each chat’s processed state (its cache) so the next turn only reads the new tokens.

There are two levels:

Level	Where	Speed	Survives a restart
RAM pool	Memory, per loaded model	Instant	No
Saved caches	Disk, encrypted	Fast, much faster than re-reading	Yes

The RAM pool

Each loaded model keeps a small pool of cached conversations, so several users can share one model without wiping each other’s cache.

• A new prompt is compared with every cached conversation, token by token. It reuses the one that shares the longest start.
• If two chats share only a start (for example, the same notes), the shared part is copied into a new slot. Copying is far cheaper than computing it again.
• The pool is limited by a number of conversations and a memory budget. When either is reached, the least recently used conversation leaves first. The one in use is never removed.
• A single very long chat that is bigger than the budget still keeps its cache, so one user is never worse off.

Messages with images

A prompt with images always starts a fresh cache. Image placeholders look the same for different pictures, so a match would not mean the same content.

Hybrid models

Models with recurrent layers, such as Qwen 3.5 and Nemotron-H, keep a fixed-size state instead of a growing cache. That state exists only at the latest position, so Matricxon saves a few snapshots while reading a prompt. The next turn restarts from the newest snapshot that fits and reads only the rest.

When pAIring trims old messages

When a chat grows past the context window, pAIring drops its oldest messages. With context shift on (the default), Matricxon removes those tokens from the cache and keeps the rest, instead of re-reading everything. It is very slightly approximate. Set MATRICXON_PROMPT_CACHE_CONTEXT_SHIFT=false to always re-read exactly.

Saved caches (disk)

When a conversation leaves the RAM pool, its cache is written to disk. This also happens when its model unloads and when the server stops. It is never written during a reply. Returning to the chat later loads it back instead of re-reading the history.

• Encrypted. Every file is sealed with AES-256-GCM. The conversation’s tokens, model and length cannot be read without the key, and a changed file is detected.
• Tied to the model. A cache only loads into the exact model file it came from. A replaced model never picks up an old cache.
• Limited. A disk budget (MB) and a keep time (hours). When over budget, the oldest files go first. Expired files are removed.
• Self-checking. At start-up, files that fail the check (wrong key or damaged) are deleted.

The key

The key is generated once into prompt_cache.key in the cache folder, readable only by its owner. To use your own, set MATRICXON_CACHE_KEY to any passphrase. In a cluster every node shares one key, which pAIring provides. A cluster node never creates its own.

Caution

If the key changes or is lost, every saved cache becomes unreadable and is deleted at the next start. Chats still work, they just re-read their history once.

Changing the limits

The limits in Configuration are only first-run defaults. Changes made at runtime are saved in prompt_cache_policy.json and win from then on. You can change them:

• In pAIring, under Admin settings → External servers → Matricxon → Saved chat caches.
• Through the API: GET, PUT or DELETE /api/cache/persistence. See API.

Turning saved caches off deletes every saved file.

Reading the logs

When a prompt reuses nothing, the log line “re-reading the entire chat” says why. Common reasons:

• RAM holds no conversations. The model was just loaded, or its pool was emptied.
• The disk cache is off. Saved caches are turned off.
• The disk cache is empty. Nothing was saved yet, or the files expired.

Settings

Variable	Default	Meaning
MATRICXON_PROMPT_CACHE_SLOTS	4	Conversations kept in RAM per model (1 to 64)
MATRICXON_PROMPT_CACHE_BUDGET_MB	2048	RAM those conversations may use together
MATRICXON_PROMPT_CACHE_CONTEXT_SHIFT	true	Keep the cache when old messages are trimmed
MATRICXON_PERSIST_PROMPT_CACHE	true	Save caches to disk
MATRICXON_PERSIST_CACHE_BUDGET_MB	4096	Disk space for saved caches
MATRICXON_PERSIST_CACHE_TTL_HOURS	168	Hours before a saved cache expires
MATRICXON_PROMPT_CACHE_DIR	./data/prompt_cache	Where saved caches and the key live
MATRICXON_CACHE_KEY	generated	Your own encryption passphrase