Matricxon cluster mode

In cluster mode, several Matricxon nodes work as one group. They share their models and their saved chat caches, and they agree on who is in the group.

Beta and under active development

Cluster mode is in BETA and changes often. It is designed for hundreds of nodes, but only small clusters have been tested so far.

You manage a cluster from pAIring’s Cluster page. This page explains how it works underneath. For the step-by-step process, see pAIring: Cluster. For the choice between layouts, see Deployment layouts.

What is shared and what is not

What	Shared?	Notes
Models	Shared	One models folder that every node reads. Pulls and deletes go there.
Saved chat caches	Shared	One folder of encrypted cache files that any node can pick up.
Cache key	Shared	One key for all nodes. It sits next to the shared cache folder and is never sent over the network.
Cache policy and cluster settings	Shared	One value for the whole cluster.
Who is in the cluster	Agreed by the nodes	Kept by the consensus protocol (see below).
Loaded models and the RAM prompt cache	Per node	Each node loads its own models into its own memory.
Native kernels, logs, process id	Per node	They depend on each node’s CPU/GPU.
Device and thread settings	Per node	Each node can differ.

Matricxon never moves gigabytes of models or caches by itself. Turning cluster mode on moves nothing. You move the data to shared storage yourself.

Shared storage

The shared folders must be a network filesystem that every node reaches, such as NFS, CIFS, Ceph or a FUSE mount, not the local disk. All nodes must see the same content.

• Models are loaded by memory-mapping the files. This works over a network filesystem, but each node reads the model over the network, so the first load is limited by network speed.
• Saved chat caches are portable between nodes. A cache file does not belong to a node, so a chat that moves can continue on another node without re-reading its history.
• A cache is encrypted with the shared key. A node joining the cluster reads the shared key file and never makes its own.

Roles

Role	Meaning
Standalone (off)	A normal Matricxon, not in a cluster. The default.
Seed	The first node. It keeps its data, and its shared paths move to shared storage. After the cluster is up and running, the seed is not participating in the cluster anymore and has no role.
Joiner	A fresh node that was added to the cluster.

A node can also be a compute node (it loads models and serves chats) or a voter node (it only takes part in decisions, and never loads or serves a model).

Among the nodes that decide:

• Voters vote on changes. The default is 3 voters.
• Learners follow the cluster and serve chats, but do not vote. A learner can be promoted.
• The leader keeps the voter count at 3 (if there are fewer than 3). It promotes a caught-up learner, and replaces a voter that is lost.

Growth rule: one seed, fresh joiners only

A cluster is one existing Matricxon (the seed) that grows with fresh installs.

• A joining node must be fresh: no models, no cache files, no saved cache policy and no cache key of its own.
• A joiner adopts the seed’s key, policy and settings. Nothing is merged or negotiated.
• Merging two nodes that both hold chats, users, models or caches is not supported.

This keeps history from ever diverging.

How nodes agree

The nodes keep a shared, ordered log of decisions using Multi-Paxos, built into Matricxon (no outside service). Every node applies the same decisions in the same order.

• What it holds: who is in the cluster, and the cluster settings.
• What it does not hold: model bytes, cache bytes, or the state of any request. Those never go through consensus.
• Majority needed: a decision needs a majority of the voters (a quorum). With 3 voters, the cluster keeps deciding if 1 voter is lost. If a majority is lost, changes stop until it returns. Chats keep working.
• Leader: one voter coordinates. If it fails, another takes over after a short silence (the election timeout).
• Safety first: the protocol never lets two different decisions exist for the same slot, even with lost or delayed messages.
• Messages between nodes are signed with a shared secret, by default the shared cache key.

Timing

Setting	Meaning	Default
Heartbeat	How often the leader says it is alive	0.2 seconds
Election timeout	How long a node waits in silence before it campaigns	1 second
Compute patience	Compute nodes wait this many times longer to campaign, so an idle voter-only node usually leads	1 (no preference)

Faster failover costs more network traffic. A slow or busy network or disk may need larger values. You change the timers from pAIring’s Cluster page.

The life of a node

1. Standalone. A node runs on its own, with all its paths local.
2. Seed. The admin turns cluster mode on in pAIring. The node reports a readiness checklist: for each shared folder, that it exists, is writable, is on a shared network filesystem with free space, has the cache key, and is the same folder the peers see. Until every row passes, it stays standalone.
3. Joiners. pAIring checks that a candidate is fresh, then sends it the cluster settings: the two shared folders and its role. The node writes them to its .env and restarts itself in place, so no supervisor is needed. It then follows the cluster.
4. Removal. Removing a node takes it out of pAIring’s pool. The node stays set up as a joiner.
5. Reset. A reset makes a node fresh again, so it can be added as a new node.

When the seed fails

“Seed” only marks where the cluster started. The other nodes keep serving from the shared models and caches, and pAIring fails over to them. Actions that need a quorum, such as changing members or pulling models, wait if a majority of the voters is lost.

Reset

A reset checks nothing and works in any state of the node. The only refusal is for an active voter, which must be taken out of the voters first. The whole-cluster reset forces it.

Both kinds remove, for that node:

• the cluster keys from its .env (folders go back to their defaults),
• its consensus journal,
• its cache policy,
• its own saved chat caches (only in its own default folder, a speed-up that is rebuilt),
• its cluster markers in the shared folder,
• its node id (it gets a new identity at the next start).

Models and anything in the shared folders are never deleted.

You reset from pAIring (the Cluster page), or by hand on the node:

scripts/reset_cluster_node.sh          # stop the server first, or add --stop

Settings

pAIring writes these for you when you add a node. They are listed here for reference. See the Configuration page for the other variables.

Variable	Meaning	Default
MATRICXON_CLUSTER_ROLE	off, seed or joiner	off
MATRICXON_NODE_KIND	compute or voter (consensus only)	compute
MATRICXON_MODELS_DIR	The shared models folder	./data/models
MATRICXON_PROMPT_CACHE_DIR	The shared saved-chat-cache folder	./data/prompt_cache
MATRICXON_CACHE_KEY	The shared encryption key	generated
MATRICXON_ADMIN_SECRET	Signs admin routes. Unset leaves them open	unset
MATRICXON_ADVERTISE_URL	How other nodes reach this one	unset
MATRICXON_CONSENSUS_TARGET_VOTERS	Voters the leader keeps (0 to 9, 0 means never changes them by itself)	3
MATRICXON_CONSENSUS_ELECTION_SECONDS	Election timeout	1
MATRICXON_CONSENSUS_HEARTBEAT_SECONDS	Heartbeat	0.2

API

Matricxon exposes cluster endpoints, which pAIring uses. You normally do not call them yourself.

Endpoint	Purpose
GET /api/cluster/readiness	The readiness checklist of this node
GET /api/cluster/fresh	Whether this node is fresh enough to join
POST /api/cluster/provision	Gives a fresh node the cluster settings
POST /api/cluster/activate, /configure	Announce a node and update its cluster settings
POST /api/cluster/reset	Reset this node to fresh
GET /api/consensus/status	Consensus state: leader, voters, quorum
GET /api/cache/holds, POST /api/cache/spill	Which node holds a chat’s cache, and saving it to the shared disk

Caution

Provisioning works only while a node is fresh, and the engine’s API has no login of its own unless you set the admin secret. Add nodes only on a trusted network.