Run a cluster
List your servers under one target, set them up, and deploy a Compose project across them.
A cluster is a target that lists several hosts under nodes:. mm sets every
host up, joins them over a private network, and deploys your Compose project
across them one node at a time. There are no managers, join tokens, or quorum
to look after: a cluster has nodes, services, networks, and volumes.
This guide goes from a node list to a running deployment, then through the
tasks that follow. Each section links to the full description in
Set up a remote host, which covers everything mm puts
on a node. If you run Docker Swarm today, Coming from Docker
Swarm maps its concepts and Compose keys to these.
What you need
- A project that
mmalready deploys:.config/mm.yamland its Compose files in a git repository. To create one, add mm to a project. - Two or more Debian or Ubuntu hosts with systemd and OpenSSH. Your machine must reach each one over SSH, and the hosts must reach each other on UDP port 51820.
- This device's mm key (
mm key create, thenmm key show) authorized forrooton every host. Setup logs in with that key, never with~/.sshor an SSH agent. - On your machine:
ansible-playbook,ssh-keyscan, and Linux builds ofmmandmmdfrom the same release. Installingmmon Linux with--with-daemonprovides both. - A git origin that every node can fetch the project from, or
delivery: pushon the target so thatmmsends each commit itself. Also a break-glass key pair whose private half you keep offline. See the requirements.
Declare the nodes
Give the target a name and a host for each node, and point an environment at it:
# .config/mm.yaml
targets:
production:
nodes:
- {name: app-1, host: 203.0.113.11}
- {name: app-2, host: 203.0.113.12}
- {name: db-1, host: 203.0.113.13}
environments:
production:
target: production
That is a complete cluster target. The mesh subnet, each node's mesh address,
the WireGuard port, the address range for networks, and the node that serves
shared volumes all have defaults; the optional cluster: block exists only to
change one.
Setup also needs to know where the nodes fetch the project from and which keys
can always get in. Add a setup: block to the target as described in
declare what setup installs,
then check the file:
mm -e production config validate
shop [production]
config: /srv/code/shop/.config/mm.yaml
config: valid
A node name that is not a hostname label, a duplicate host, a placement that
names an unknown node, and a Compose service with no entry under modules are
reported instead, each with the key to change.
Set up the nodes
Setup is a plan you review, then an apply that runs the same steps on every node:
mm -e production target setup deploy-key
mm config split targets
mm -e production target setup plan
mm -e production target setup apply --plan PLAN_HASH --yes
mm -e production target setup record
deploy-key prints a public key once; add it to the git origin as a read-only
deploy key and set it as setup.deploy-key.public-key. plan scans every
node's SSH host key and shows what each node gets, including its mesh peers.
apply installs Docker Engine with the Compose plugin (or Podman, for an
environment that uses podman-compose) and mmd, brings up the
WireGuard mesh, prepares shared storage, pushes the environment's secrets, and
applies the users file. record writes each node's address, host key, and
daemon identity to .config/mm/targets.yaml; commit and push that file.
Run setup explains each step and what to do
when one node cannot be reached.
Check the result:
mm -e production target nodes
Target production: 3 node(s)
NODE ADDRESS MESH STATE MMD DEPLOYED
app-1 203.0.113.11:22 10.90.0.1 ready 0.1.0 nothing yet
app-2 203.0.113.12:22 10.90.0.2 ready 0.1.0 nothing yet
db-1 203.0.113.13:22 10.90.0.3 ready 0.1.0 nothing yet
Networks (cluster.networks 10.64.0.0/12)
No network of this environment has addresses yet; the first deployment gives them out.
Every node is ready. Below this the command lists the addresses each node
already uses, which the cluster's ranges must not overlap.
mm -e production target mesh status shows the WireGuard link between each
pair of nodes and names the port to open when two of them have no handshake;
see the mesh between the nodes.
Say where each service runs
Give a service a placement: a node name, a group, the word all, or a list.
targets:
production:
nodes:
- {name: app-1, host: 203.0.113.11}
- {name: app-2, host: 203.0.113.12}
- {name: db-1, host: 203.0.113.13}
cluster:
groups: {web: [app-1, app-2]}
modules:
shop:
path: shop
services:
web: {stack: app, placement: web} # app-1 and app-2
db: {stack: app, placement: db-1} # db-1 only
worker: {stack: app} # no placement: the first node, app-1
A service runs one container on each node its placement names, unless it sets
replicas or autoscale (see scale a service). Every service
in a stack's Compose files needs an entry under modules, so that mm knows
where it runs; mm config validate names any that is missing. A named Compose
volume is local to its node, so keep a service that stores data pinned to one
node. Place services on nodes
covers moving a service and the Compose keys that tie two services to one
host.
Deploy
Push the commit, then plan and apply it. Every node fetches the commit from
the origin itself, or receives it from mm with
push delivery:
mm -e production deploy plan
mm -e production deploy apply --plan PLAN_HASH --yes
The plan opens with what the deployment does across the nodes, then has one section per node: what each node starts, changes, and stops, and the addresses each network gets there. One hash covers every section:
Cluster deployment plan 0a4ba0ae34677ecf6507139bc887347d251e8ca7ba19e9f006fc2f016be9f9ab
target: production (app-1, app-2, db-1)
commit: 2e445474cb8dc3608f384f908f60ab7cf47c89e9
expires: 2026-11-13T17:43:32Z
networks span the nodes: addresses from 10.64.0.0/12, between nodes through mm0 (MTU 1420)
service names: each node answers its own containers, on its address on each network (port 53), with a service's healthy containers on the same node, or else on every node
rollout: app on app-1, then app-2, then db-1
a stack may take 10m on a node to become healthy (deploy.convergence-timeout); the next node starts once the others resolve its containers
on failure every node this deployment changed returns to what it ran before
Apply replaces one node at a time. A node is done when its containers pass their health checks and every other node resolves them by name:
==> app-1: preparing
==> app-1: app
==> app-2: preparing
==> app-2: app
==> db-1: preparing
==> db-1: app
==> app-1: finishing
==> app-2: finishing
==> db-1: finishing
Deployment 0a4ba0ae34677ecf6507139bc887347d251e8ca7ba19e9f006fc2f016be9f9ab on production [production]
app-1 applied stacks: app; 0 route(s)
app-2 applied stacks: app; 0 route(s)
db-1 applied stacks: app; 0 route(s)
Give each service a health check in its Compose file: without one, a container
counts as healthy as soon as it runs. To replace a container without a gap,
set update: {order: start-first} on the service, and the apply says
app: starting the new web next to the old on each node. To order stacks, set
after: on a stack. Both are in rolling deploys and
rollback.
Modules that mm builds deploy from a build receipt, because a node never
builds. Follow deploy a build to a remote
host.
Roll back
A deployment that fails rolls itself back. When a container fails its health check, exits, or keeps restarting, the deployment stops, every node it changed returns to what it ran before, and the nodes it had not reached are never touched:
==> app-1: preparing
==> app-1: app
==> app-2: preparing
==> app-2: app
==> app-2: rolling back
==> app-1: rolling back
Deployment 85549657b9e8dbab7d79adda5e1cbeb919803cfcab243882043abfbae414ebb4 on production [production]
app-1 rolled back runs what it ran before
app-2 FAILED service 'web' of stack 'app': its health check fails; rolled back to what it ran before
last log lines of the failed container:
listening on :3000
cannot reach the database at db:5432
db-1 not started unchanged
To return from a deployment that finished:
mm -e production deploy rollback
It prints a plan for the commit the nodes ran before and applies it after you
confirm. mm -e production deploy status shows what every node runs, what it
ran before, and a rollout that was interrupted.
Volumes are not rolled back: the earlier version runs against the data the later one left. Plan schema changes so that the previous version still works with them. More in return to the previous deployment.
Connect services across nodes
Declare a network in mm.yaml and put services on it. It spans every node
that runs one of them:
networks:
backend: {}
modules:
shop:
services:
web: {stack: app, role: http, networks: [backend], placement: web}
db: {stack: app, role: database, networks: [backend], placement: db-1}
web on app-1 reaches the database at db:5432, although db runs on
db-1. The traffic travels inside the mesh, encrypted, and two containers
that share no network cannot reach each other.
| Name | Resolves to |
|---|---|
db | The healthy containers of db on the asking node, or on every node when it has none. |
tasks.db | Every healthy container of db, on every node. |
db.db-1 | The healthy container of db on node db-1. |
mm -e production service status lists what each name resolves to. A network
that only a Compose file declares stays local to its host.
Limits: a network spans at most 16 nodes, a node runs at most 253 containers of one network, and the default address range holds 256 networks. Addresses are IPv4. See networks across nodes and service names across nodes.
Share a volume between nodes
Use a shared volume for files that the containers of a service all need on
every node, such as uploads or generated assets. List the Compose volume under
shared-volumes on its stack:
stacks:
app:
files: [compose.yaml]
shared-volumes: [uploads]
Use a local volume for anything where performance matters: databases,
queues, search indexes, caches. A shared volume is a network mount. Next to
the storage node, creating a file takes 7 to 9 ms instead of 0.15 ms; on a
node 58 ms away, every file opened and every write flushed waits a round trip,
and a large read runs at 8 MB/s instead of 1 GB/s. It also does not give the
locking guarantees a database needs, and it stops answering while the node
that keeps it is down. Give such a service an ordinary volume and pin it to one
node with placement. mm config validate refuses a service with
role: database or role: queue on a shared volume; allow-shared-volumes: true on the service overrides that. The measurements
are in the remote host guide.
The Compose file stays as it is. One node, the first unless you set
cluster.storage, keeps the files under /var/lib/mm/storage and serves them
to the other nodes over the mesh with NFS. Nothing is installed on the nodes
that mount it. The deployment plan says which node keeps each volume and which
nodes mount it, and the deployment creates the directory before it replaces a
container.
Check it:
mm -e production target storage status
Target production: app-1 keeps shared volumes in /var/lib/mm/storage, 26.2 GiB free of 29.3 GiB
app/uploads 12.0 KiB in 3 files; mounted on app-1, app-2
app-1 keeps the volumes and serves them on 10.90.0.1
app-2 reaches app-1 in under 1 ms
db-1 reaches app-1 in 58 ms
Every node reaches the storage node.
What to know before you rely on it:
- The storage node is a single point of failure. While it is down, reads and
writes of shared files on the other nodes wait, and continue when it is
back.
mmdoes not replicate or back up the files: back up/var/lib/mm/storageon the storage node yourself. - A node that boots while the storage node is away starts its other services; the ones that mount a shared volume start once the storage node answers.
- Taking a volume out of
shared-volumes, a rollback, and a failed deployment never delete files.mm -e production target storage delete uploadsshows what it would remove and deletes it only with the hash it prints. - A volume that becomes shared starts empty on the storage node. Copy the files from the old local volume yourself.
- Changing
cluster.storageis refused while the old storage node holds files, with the steps to move them. - Shared volumes need the mesh
mmbuilds. A target that setscluster.mesh.interfaceserves none.
See shared volumes across nodes for where the files are, what happens while the storage node is down, and how the export is kept private.
Read logs and status
A service is all of its containers on all of its nodes, and every command means all of them unless you narrow it:
mm -e production service status
mm -e production service logs web --follow
mm -e production service logs web --node app-2
mm -e production service logs
service status asks every node at once and lists each service once, with a
row for each node's container:
STACK SERVICE STATE NODE CONTAINER HEALTH STATUS IMAGE
app web degraded 1/2 app-1 web-1 healthy Up 2 hours registry.example.com/shop/web:1.4.2
app-2 - - no answer -
app db running 1/1 db-1 db-1 healthy Up 2 hours postgres:17
app worker running 1/1 app-1 worker-1 running Up 2 hours registry.example.com/shop/worker:1.4.2
The count under STATE is the containers that run and pass their health
check, out of those the service should have. A container whose health check
has not passed yet makes the service read starting, as in the middle of a
rollout. A node that does not answer reads no answer, and the reason follows
the table:
node app-2: accepts SSH, but mmd is not running there (the relay found no daemon socket). Start it on the host with `systemctl start mmd`; `journalctl -u mmd` shows why it stopped
The other nodes' rows are shown all the same. Between the two, status lists what each service name resolves to on each network.
service logs web merges the lines of every container of web into one
stream in time order and starts each line with its node and container:
app-1 web-1 | GET / 200 12ms
app-2 web-1 | GET /assets/app.css 200 2ms
app-1 web-1 | GET /api/projects 200 18ms
Without a service name, service logs reads every service of the environment
the same way, and --node reads only the nodes you name. A node that does not
answer is one line in the stream, marked with !, and the command fails only
when no node answered. Lines are ordered by the timestamps the container
engines record, so two nodes whose clocks differ can swap lines that are
closer together than that difference: keep the nodes' clocks synchronised with
NTP or chrony. Status, logs, and commands has
every column and option.
Status and logs take no deployment lock, so they answer in the middle of a
rollout and while a node is down. mm service exec runs a command in the
service's container on the first node of its placement.
Harden the containers
One line in .config/mm.yaml gives every service of a deployment a read-only
root filesystem, no Linux capabilities, no new privileges, the engine's
default seccomp profile, and a writable /tmp:
hardening: strict
mm deploy plan prints what each service will run with, and the plan hash
covers it:
security: web: read-only root, no capabilities, no new privileges, default seccomp, tmpfs /tmp
A service that cannot take one of these defaults leaves it out with a reason, which the plan prints next to it:
modules:
shop:
services:
db:
stack: app
hardening:
except:
read-only: Postgres writes its socket under /var/run
It does not apply to mm dev on your machine. Images that start as root and
switch to their own user, such as the official postgres and nginx images,
need a few capabilities back with cap_add: in the Compose file. Each node's
/etc/mm/host.yaml can also offer OCI runtimes such as gVisor's runsc and
put every service that publishes a port under one of them. Service names,
shared volumes, and secret files work unchanged under a read-only root.
Ship configuration files
A file that is not a secret and not part of the image, such as a proxy
configuration or an init script, goes under the service's config-files.
Each node that runs the service writes it and mounts it read-only into the
container:
modules:
proxy:
path: proxy
services:
proxy:
stack: app
config-files:
/etc/nginx/conf.d/site.conf: { template: site.conf.tmpl }
file: ships a file from the module's directory as it is; template: replaces
each {{ KEY }} with the environment's binding of that key. When the file
changes, the plan says which services are replaced, and only those roll, one
node at a time. A config file is at most 1 MiB and never takes a secret.
Read secrets on the nodes
Secrets from mm's own store reach the nodes sealed during setup. Secrets
kept in an external OpenBao or Vault can be read by the nodes themselves, so
their values never pass through the machine that runs mm deploy. Give the
nodes an AppRole that may only read the environment's secrets, then connect
each node that runs a service using such a secret, once, as root on the node.
The command reads the role ID and the secret ID from stdin, one per line:
{
bao read -field=role_id auth/mm-approle/role/mm-host-production/role-id; echo
bao write -f -field=secret_id auth/mm-approle/role/mm-host-production/secret-id; echo
} | mmd secrets connect --project PROJECT_ID --environment production \
--provider team-secrets --address https://10.90.0.1:8200 \
--ca-file /etc/mm/bao-ca.pem
It logs in before it keeps anything and stores the credential encrypted, for root only. A plan then carries the version of every secret each node read, and a rotated secret replaces only the services that use it. A node asks the store only for the secrets of its own services, and a node whose services use none needs no connection. When the nodes reach the store at another address than the provider's, such as a node's mesh address, declare it on the target:
targets:
production:
providers:
team-secrets: {address: https://10.90.0.1:8200}
Deploy without git on the nodes
When the nodes may not reach your git server, have mm send each commit:
targets:
production:
delivery: push
nodes:
- {name: app-1, host: 203.0.113.11}
- {name: app-2, host: 203.0.113.12}
- {name: db-1, host: 203.0.113.13}
setup:
break-glass:
- {name: ops-recovery, key: ssh-ed25519 AAAA… ops@offline}
There is then no origin and no deploy key in setup:, and the nodes never
run git. mm deploy plan sends every node of the plan a bundle of the files
the commit tracks, read from your checkout's git objects, and each node checks
them against the commit before it keeps them. A node that already holds the
commit is sent nothing, so a rollback costs no upload. Images still come from
their registries, so the nodes need a route to those. With push delivery a
node cannot ask the git server which branches hold a commit: deploy-from is
checked against the branches your client states, so give deploy on such a
target only to CI and to people who may push to those branches.
Add or remove a node
Add the node to nodes:, or delete it, commit, and run setup again:
mm -e production target setup plan
mm -e production target setup apply --plan PLAN_HASH --yes
mm -e production target setup record
The plan shows the one mesh entry that changes on every other node. A new node
runs nothing until a service's placement names it and you deploy. Removing a
node drops it from the mesh; setup does not contact a host that is no longer
listed, and the plan says how to take the mesh off it. Move its services to
other nodes and deploy before you remove it. See add or remove a
node.
When a node goes down
Nothing is rescheduled. mm has no scheduler and no consensus store: the
services placed on a node run on that node and nowhere else.
- The other nodes keep running. Within about ten seconds they stop answering service names with the lost node's containers, so a service that also runs elsewhere keeps working by name.
- A service that runs only on the lost node is down until the node is back. For a service that has to survive a node, place it on more than one.
- When the node boots, the mesh, the firewall rules, and its services come back without a deployment.
- While it is down,
mm deploy planstops and names the node. To deploy to the others, pass--nodeswith their names; see when a node does not answer. - If the lost node is the storage node, reads and writes of shared volumes on the other nodes wait until it returns.
To replace a node for good, remove it from nodes:, add its replacement,
change the placement of its services, run setup, and deploy. Data in its
local volumes is not moved for you.
Scale a service
replicas runs a count of containers spread over the placement's nodes, and
autoscale lets the count follow the load between scale.min and
scale.max:
modules:
shop:
services:
web:
stack: app
placement: all
autoscale:
scale: { min: 1, max: 4 }
A container that leaves first drops out of the names and the ingress, drains, and then stops, so scaling down drops no request. Autoscale a service covers the metrics, the thresholds, and how to watch what the autoscaler decided.
Restart a service, run a job, route traffic in
mm service restartrestarts a service's containers one node at a time; see restart a service.- Jobs run on a cluster on their own, with a deployment, or once per environment; see run a job on a remote target.
ingress:routes reach the service's containers on every node; see route traffic into the cluster.
What a cluster does not do
- Nothing is rescheduled when a node goes down: its containers come back when the node does, and the other nodes keep answering for services that run on them too.
- A named Compose volume is local to its node. Keep a service that stores data pinned to one node, or use a shared volume for files where speed does not matter.
Known issues lists the remaining limits.