pAIring cluster management

A cluster makes several Matricxon servers work as one group that shares its models and its chat memory. You manage it from pAIring’s Cluster page (admins only).

Beta and under active development

Cluster mode is in BETA and changes often. It is designed for hundreds of nodes, but only small clusters have been tested so far. Details here may change between versions.

Read Deployment layouts first if you are not sure that you need a cluster. For the engine side of the story, see Matricxon: Cluster.

Before you start

You need:

• Matricxon as the active engine. Cluster mode is a Matricxon feature. Set it under Settings → External servers → Active engine. Ollama has no cluster mode.
• A first node. This is an existing Matricxon that already has your models and chats. It is called the seed.
• Shared folders. One folder for models and one for saved chat caches. Every node must see the same content, usually through a network mount such as NFS. Models load over the network, so the first load of a model is limited by network speed.
• Fresh nodes to add. Each new node must be a fresh Matricxon install, with no models, saved chats or settings of its own.
• At least 3 Matricxon nodes if you want the cluster to survive a node failing. See Add nodes.
• A private network. See Security.

One seed, fresh nodes only

A cluster grows from one existing Matricxon (the seed) by adding fresh installs.

Words you will see

Word	Meaning
Seed	The first node, where the cluster started. After the cluster started there is no longer a need for the seed node, and the seed has no role.
Joiner	A fresh node that was added to the cluster.
Voter	A node that takes part in the decisions about who is in the cluster.
Learner	A node that serves chats and follows the cluster, but does not vote. It can become a voter.
Voter only	A node that only votes. It never loads models and never gets chats.
Leader	The voter that currently coordinates changes. Another voter takes over if it fails.
Quorum	A majority of the voters. Many actions need it.
Cluster id	A name that identifies the cluster, shown on the Cluster page.

Create a cluster

Open the Cluster page. It has four tabs: Cluster mode, Nodes, Add nodes and Cluster host names.

Step 1: Choose the first node and the shared folders

On the Cluster mode tab, under Create a new cluster, fill in First node and shared folders:

• First node: the address of the Matricxon that already has your models and chats. You can type it or pick a known host.
• Address the other nodes use to reach it: optional. The default is the node’s own address.
• Models folder and Saved chat caches folder: the shared folders. The cache key and policy files are kept next to the cache folder.

Press Save.

Saving moves nothing

pAIring never moves your models or caches by itself. If your models are on a local disk today, copy them into the shared folder yourself. The folders must already exist on each node.

Step 2 (optional): Set an admin secret

An admin secret is a shared password that protects the nodes’ admin actions: reset, adding or removing nodes and voters, pulling and deleting models. pAIring signs those calls with it, and a node that has the secret refuses unsigned calls. Chats and read-only calls need no secret.

• Set it once, here, after the first node works and before you add other nodes. It is sent to every node, and to each node you add later.
• To change or remove it, use the same form. Leave it empty to remove it.
• Without a secret, anyone who can reach a node can reset it or change the cluster. Keep the nodes on their own network either way.

Step 3 (optional): Tune the cluster

On the same tab you can also set the cluster timers and the saved chat caches. You can change these later too. See Settings of a running cluster.

Step 4: Turn cluster mode on

At the bottom, a checklist shows what must be true on the first node. It checks, for each shared folder, that:

• the path exists and can be written,
• it sits on a shared network filesystem, not the local disk, with enough free space,
• the cache key is in place,
• other nodes see the same folder.

Wait until every check is green. Then go to the top and turn Cluster mode on.

Matricxon restarts itself to apply the change, and this takes a few seconds. When it is back, the page shows the Cluster id. Use Copy to copy it. You need it to join another pAIring to this cluster.

Switch the first node to Remote

Once the cluster runs, you can switch the Matricxon mode under Settings → External servers from Local to Remote. Then pAIring only talks to the nodes and never starts or stops them, so the first node is treated like all the others. No node is lost by switching.

Add nodes

On the Add nodes tab, you bring a fresh Matricxon into the cluster.

1. Set up a new Matricxon node and start it. Leave it fresh.
2. Enter its address under Add a node. Fill in Address the OTHER nodes use to reach it only if it differs.
3. Choose the Node type:
   • Compute node: loads models and answers chats.
   • Voter only: takes part in decisions only. It never loads models or gets chats.
4. Press Check. pAIring confirms that the node is fresh and reachable.
5. Press Add node, and wait for it to restart.

pAIring gives the node the cluster’s settings, and the node restarts itself and joins. Watch it appear on the Nodes map. The node follows the cluster and uses the shared models and chat caches.

Joining as a voter. A new node becomes a voter by itself while the cluster has fewer than 3 voters. Later nodes stay learners. If a voter is lost, the cluster promotes a caught-up learner to replace it.

Join an existing cluster

If another pAIring already created the cluster, use Join an existing cluster on the Cluster mode tab instead. Enter the address of any node and the cluster id (shown on the Cluster page of the pAIring that set it up). The node must report that id. If the cluster has an admin secret, enter the same secret. Do not set a new one.

The Nodes map

The Nodes tab shows every server in the cluster live, with each node’s role and status. Click a node to open its details and actions.

• Status: Running (green), Needs attention (amber) or Not responding (red). You can filter by state and search for a node.
• Badges: show the roles, such as leader, voter, learner, seed, joiner, voter only, or “not in cluster”.
• Motion: turn it on for a moving map. It uses more CPU, so it is off by default.
• Big clusters: the map adapts, from a few nodes to hundreds.

Actions in a node’s panel. Buttons that do not apply are greyed out, and the reason is written below them.

Button	What it does
Activate in cluster / Add to cluster	Announces the node to the cluster
Make it a voter	Promotes a learner
Make it a learner	Demotes a voter. The last voter cannot be demoted
Update its address	Records a new address when a node has moved
Remove from cluster	Takes the node out of the pool. It stays set up as a joiner
Remove from voters	Drops a voter whose node is no longer in pAIring’s list

The Cluster host names tab lists every peer a node knows how to reach, with the address it uses. Each node has its own list. This helps when nodes cannot see each other.

Settings of a running cluster

These apply to the whole cluster at once and need no restart. They need a quorum.

Cluster timers. They control how the voters notice a dead leader.

Setting	Range	Default
Election timeout	0.5 to 60 seconds	1
Heartbeat	0.05 to 5 seconds, at most a third of the election timeout	0.2

A shorter election timeout replaces a failed leader sooner. A longer one avoids needless elections on a slow or busy network or disk. Two presets help: Fast (1 / 0.2) and Tolerant (5 / 1).

Saved chat caches. Each long chat’s processed state is kept on the shared disk, encrypted, so returning to it, even on another node, does not re-read the whole conversation.

• Save chat caches to disk: on or off. Turning it off deletes everything saved.
• Disk limit (MB): the oldest are deleted first. 0 saves nothing.
• Keep for (hours): 168 hours is 7 days.
• Delete all saved caches.

When there is no quorum

If a majority of the voters cannot be reached, the cluster has no quorum. pAIring shows a warning, and these are locked until enough voters are back:

• adding, removing and activating nodes,
• the shared folders, cluster timers and saved-chat-cache limits,
• changes to who pulls models, and model downloads.

Chats keep working. Everything resumes when enough voters return.

Leaving or resetting

• Reset to fresh (one node). On the Add nodes tab. It undoes only that node’s own cluster setup, then restarts it, so it can be added again. It never touches the shared folders. It is refused for a node that is still a voter. Take it out of the voters first.
• Reset cluster. Every node forgets the cluster (its setup, history and identity) and its own saved chat caches, and restarts as a fresh install. Cluster mode goes off, and pAIring forgets the shared folders and cluster settings. The hosts stay in its list as plain remote hosts. Models and the shared folders are not touched. It works without a quorum. A node that cannot be reset from here needs scripts/reset_cluster_node.sh run on that node.
• Leave cluster (this pAIring only). Turns cluster mode off for this pAIring only. The nodes are not reset, and they stay listed as plain remote hosts.

Security

• Add nodes only on a trusted network. A fresh node can be claimed by whoever reaches it first, and Matricxon’s API has no login of its own.
• Set the admin secret so nodes refuse unsigned admin actions.
• Keep the nodes on their own network, preferably with no internet access, unless you use external MCP servers and enable them for users’ chats.

Good to know

• Chat routing. pAIring prefers the node that already holds a chat in memory. When it sends the chat to another node, the old node saves the chat’s cache to the shared disk, and the new node continues from it, so it does not re-read the whole conversation.
• Models. Models live in the shared models folder, so every node sees the same ones.
• Ollama. Not supported in cluster mode.