Run on a multi-node Swarm

mm can deploy to a Docker Swarm cluster with several managers. You choose where each service runs with one word in .config/mm.yaml. mm writes the Swarm constraints, keeps the node labels, and runs a built-in NFS server for volumes that every node must see.

Use a test cluster first.

Requirements#

• An active Swarm. Run mm on a manager, or from your machine through a manager's mmd (see remote hosts).
• Published images, as for any deployment. Every node pulls them.
• For shared volumes or HTTP routes: the nfsd kernel module on the storage node, and /var/lib/mm/storage on a normal filesystem such as ext4, xfs, or btrfs. Other nodes need nothing installed.
• For HTTP routes: ports 80 and 443 free on every ingress node.

Set up the nodes#

Install Docker Engine on every node with mm target setup and the docker-swarm runtime. On managers that developers reach through mmd, add the remote access flags; see remote hosts. Setup logs in through a private target for each node's admin account:

mm target add mgr-1-setup --host 10.0.0.11 --user admin \
  --host-key-fingerprint SHA256:REVIEWED_HOST_KEY_FINGERPRINT
mm --environment production --target mgr-1-setup --runtime docker-swarm \
  target setup plan \
  --origin git@github.com:acme/shop.git \
  --origin-host-key 'ssh-ed25519 AAAA…' \
  --break-glass "ops-recovery=$(cat ops-recovery.pub)"

Apply each plan with the command it prints, and repeat for every node. Setup installs the same packages as for Docker Compose and no host proxy, because Swarm runs Traefik itself.

Setup does not create the Swarm. Initialise it on the first manager and join the others by hand, on private addresses:

# On mgr-1:
docker swarm init --advertise-addr 10.0.0.11 --data-path-addr 10.0.0.11
docker swarm join-token manager
# On mgr-2 and mgr-3, with the token printed above:
docker swarm join --token TOKEN \
  --advertise-addr 10.0.0.12 --data-path-addr 10.0.0.12 10.0.0.11:2377

Check the cluster:

mm target swarm nodes

Swarm cluster m0lv578bbrlwkzwufcgiuwfnu (3 nodes, 3/3 managers reachable, quorum 2)
  mgr-1 (this node) leader active 10.0.0.11 ingress
  mgr-2 manager active 10.0.0.12 ingress
  mgr-3 manager active 10.0.0.13 ingress
  storage: not chosen yet; the first deployment with shared volumes uses the node it runs on
  ingress: every manager (default)

The command warns when a manager is down, when losing one more manager would break quorum, or when the cluster has an even number of managers.

Where cluster commands run#

mm target swarm nodes, group, storage, and ingress follow the selected target, like mm deploy:

• Through mmd. From a project checkout, when the environment's target (or --target) is an SSH target, mm sends the command to that manager's mmd. Nobody needs a shell on the manager.
• On the manager itself. With a local target, or outside a project with no --target, mm uses this machine's Docker engine.

mm --environment production target swarm nodes
mm --environment production target swarm group set edge mgr-2 mgr-3

Through mmd, reading nodes needs the view level in that environment, and changing groups, storage, or ingress needs admin (see remote access). A change is a reviewed plan, like a deploy:

1. The manager plans the label changes against the live cluster, stores the plan, and records it in /var/lib/mm/audit.jsonl.
2. mm shows the changes and asks before applying them (--yes skips the question).
3. The manager applies that stored plan once, within ten minutes, and only if the cluster still matches it. Otherwise it refuses, and you run the command again to review the new plan.

The audit log records who planned and applied each change, the plan hash, the request, and the label changes.

Node labels belong to the whole cluster, not to one environment. Anyone with admin in any environment a manager runs can change them, and the change affects every project on the cluster. Use the host's ceiling to keep admin for the people who run the cluster.

Run a command in a service#

mm service exec SERVICE -- COMMAND runs in that service's container wherever Swarm placed it. When the manager mm reached does not run the container, it names the node that does, and mm connects to that node's mmd (a target member of that name, checked against its pinned keys) and runs the command there. This needs the node to be listed in the target, as every node of a nodes: target is.

Pin a service to a node#

Add placement to the service:

modules:
  db:
    path: db
    services:
      postgres:
        stack: app
        placement: mgr-1

placement accepts:

Docker Compose, Podman Compose, and single-node Swarm ignore placement, so the same file still works on a laptop.

Group nodes#

A group lets a service run on any of several nodes, so Swarm can move it when a node fails:

mm target swarm group set edge mgr-2 mgr-3

mm shows the node label changes and asks before applying them. Use --yes in scripts. Then name the group in placement:

web:
  stack: app
  placement: edge

Groups are stored as io.mm.group.<name> labels on the nodes, so everyone who manages the cluster sees the same groups. mm only changes labels that start with io.mm.. Remove a group with mm target swarm group rm edge.

Set every label at once on a manager#

A host administrator can set the groups, storage node, and ingress nodes in one step with mmd swarm converge. Run it as root on a manager that runs the project's environment, and pass the whole label set as JSON on standard input. Name nodes by hostname:

echo '{"storage": "mgr-1", "groups": {"db": ["mgr-1"]}, "ingress": ["mgr-2", "mgr-3"]}' |
  mmd swarm converge --project PROJECT_ID --environment production

{
  "cluster-id": "m0lv578bbrlwkzwufcgiuwfnu",
  "changed": true,
  "changes": [
    { "node-id": "…", "hostname": "mgr-1", "add": { "io.mm.group.db": "true" } },
    { "node-id": "…", "hostname": "mgr-2", "add": { "io.mm.ingress": "true" } },
    { "node-id": "…", "hostname": "mgr-3", "add": { "io.mm.ingress": "true" } }
  ],
  "warnings": []
}

The declaration is the complete set of io.mm. labels:

• groups: each group gets exactly the listed nodes. Groups that are not listed lose their labels, and a warning names them.
• ingress: the nodes that run ingress. Leave it out, or pass an empty list, for every manager.
• storage: the storage node. Leave it out to keep the current one; moving storage does not copy data, so the command warns when it moves.

Running the command again with the same declaration prints "changed": false and an empty changes list. Changes are recorded in /var/lib/mm/audit.jsonl and appear in mmd access audit.

Share a volume between nodes#

A normal named volume lives on one node. On a cluster, mm requires a service that uses one to be pinned to exactly one node. If the service must run on several nodes, list the volume under its stack:

stacks:
  app:
    files: [compose.yaml]
    shared-volumes: [uploads]

Keep the volume declared in Compose as usual (volumes: { uploads: {} }) without driver, driver_opts, or external. On the first deployment that needs it, mm:

1. Chooses the node it runs on as the storage node, and says so in the plan.
2. Starts an NFS server there, limited to the cluster's node addresses.
3. Creates a directory for each shared volume under /var/lib/mm/storage.
4. Mounts that directory on every node that runs the service.

To use a different storage node, run this before the first deployment:

mm target swarm storage set mgr-2

Moving storage later does not copy data. Services start with empty shared volumes until you copy the files from the old node's /var/lib/mm/storage.

The NFS server mm starts is a container that needs to mount kernel filesystems. Docker's default AppArmor profile forbids that, and Swarm offers no way to change a service's profile, so on Ubuntu (AppArmor on by default) the container cannot start. A cluster target set up with mm target setup avoids this: setup installs the storage node's own nfs-kernel-server (NFSv4.2 only, exported to the nodes' private addresses) and labels the node io.mm.storage-server=host, and deploys then use it instead of the container. The volumes and their mount options are the same either way.

Serve HTTPS#

Routes work as on a single host: add ingress to a stack and a domain to the environment (see configuration). On a Swarm, mm runs Traefik on every ingress node (every manager unless you choose otherwise), listening directly on ports 80 and 443. Point DNS or your load balancer at those nodes; any of them can answer any request. No host proxy is installed.

Certificates come from Let's Encrypt automatically. mm runs one ACME instance that requests certificates; the other nodes forward Let's Encrypt's HTTP challenges to it and load the certificates it obtains. Route and certificate changes reach every node within a few seconds, without restarting Traefik. They live on the built-in shared storage, so the first route also starts it.

Serve your own certificate#

Behind a proxy that blocks Let's Encrypt's challenges, such as Cloudflare Access, serve a certificate you provide instead, for example a Cloudflare origin certificate. Store the PEM files as secrets, base64 on one line, and name them in the environment:

secrets:
  tls.origin-certificate:
    storage: {provider: age, location: local/tls-origin-certificate}
  tls.origin-key:
    storage: {provider: age, location: local/tls-origin-key}

environments:
  integration:
    tls: {certificate: tls.origin-certificate, key: tls.origin-key}

base64 -w0 cert.pem | mm -e integration secret set tls.origin-certificate --stdin
base64 -w0 key.pem | mm -e integration secret set tls.origin-key --stdin

Each deploy writes both files to the shared ingress storage, where every edge loads them, and that environment's routes stop asking ACME for certificates. The certificate has to cover every domain the environment routes. Provided certificates need Swarm ingress.

Run ingress on fewer nodes, for example those your load balancer targets:

mm target swarm ingress set mgr-2 mgr-3
mm target swarm ingress set   # every manager again

To test with Let's Encrypt staging or use an internal ACME server, set its directory, and the CA that signs its HTTPS certificate if public roots do not trust it:

mm target show local
mm target ingress configure local --image TRAEFIK_IMAGE \
  --acme-email ops@example.com \
  --acme-ca-server https://acme-staging-v02.api.letsencrypt.org/directory \
  --expected-fingerprint TARGET_FINGERPRINT

Add --acme-ca-certificate ca.pem for an internal server. Keep the image shown by mm target show.

Deploy#

Deploy as usual:

mm deploy plan
mm deploy apply --plan PLAN_HASH --yes

The plan lists where each service runs and the shared volumes:

  cluster: 3 nodes
    app/db -> mgr-1 (one node: no failover)
    app/web -> mgr-2, mgr-3
  shared storage: mgr-1 (10.0.0.11:/var/lib/mm/storage) - new
    app/uploads
  ingress edges: mgr-1, mgr-2, mgr-3 (ports 80 and 443)

The plan includes the nodes and mm labels it was made against. If a node joins or leaves, or a group, storage, or ingress label changes, deploy apply rejects the old plan. Create a new plan.

Fail over between managers#

When you reach the cluster through mmd, enroll the other managers as members of the same target. mm tries the target first, then each member in order.

For a target the repo declares under targets:, list the managers under members so every clone and CI runner gets them, and the keys they pin are reviewed with the rest of the repo. See Swarm managers as failover members.

For a target in your private catalog, add each member with mm target swarm member add:

mm target show production
mm target swarm member add production mgr-2 \
  --host 10.0.0.12 --user mm \
  --host-key-fingerprint SHA256:REVIEWED --host-key 'ssh-ed25519 REVIEWED_BASE64' \
  --daemon-fingerprint REVIEWED_BLAKE3 \
  --expected-fingerprint TARGET_FINGERPRINT

Set up each member with mm target setup and the same --origin and --break-glass flags, so every manager runs mmd with the same host policy. Take each member's --host-key and --daemon-fingerprint from its setup output. mmd fetches the project from its origin itself. mm moves to the next member only when it cannot connect. It never resends a request that reached a daemon.

Access on every manager#

Every manager enforces the same users, whichever one a command reaches.

When mm team sync or mm deploy apply changes an environment's access, the manager that runs it does three things before deploying anything:

1. It stores the reconciled users file in the Swarm as a config named mm-access-record-….
2. Each other manager's mmd that runs the environment picks up the newest record within a few seconds and updates its SSH keys. It then records an acknowledgment (mm-access-ack-…).
3. The command waits up to 20 seconds for those acknowledgments, and reports which managers confirmed.

Applied the users file of 3f2c9a1e5b7d4c6f8a0b2d4e6f8a1c3e5b7d9f0a to production
  access changes in production:
    carol: none -> deploy, +1 key(s)
  managers: mgr-1, mgr-2, mgr-3 confirmed (shared record 4)

A manager that does not confirm in time gets a warning line, and the change still holds. That manager applies it on its next check or when its mmd restarts. mmd checks the Swarm before every plan and apply, and at least every 30 seconds for other requests. When it cannot reach Docker to check, it refuses requests rather than act on an outdated copy.

If another manager changed the environment's access after you planned, apply refuses with "produce a new plan". Plan again and review the new change.

A newly set-up manager picks up the current users of every environment it runs, so you need no extra sync.

For the first sync of a new cluster, run mmd access sync --require-shared as root on one manager once the Swarm is formed (see apply the users file for the first time). It shares the record like a team sync. Every other manager needs only a running mmd with the project and environment enrolled in its /etc/mm/host.yaml: it adopts the record within a few seconds. Check with mmd access show on each manager; shared-record shows the same sequence.

After upgrading#

Access is shared once a manager runs this release. Upgrade every manager. Then run mm --environment ENVIRONMENT team sync once for each environment, so one shared record replaces the copies each manager kept on its own.

What is shared#

• The records hold user names, public keys, levels, deploy-from, the commit, and who applied it. They never hold secrets.
• mm keeps the newest three records per environment.
• When a manager leaves the Swarm, the next change drops its acknowledgments.
• Each manager still applies its own /etc/mm/host.yaml: break-glass keys, ceilings, and which environments it runs.
• Managers trust each other's records. Anyone who can use Docker on one manager already controls every node.

Limitations#

• Single storage node. Shared volumes are unavailable while the storage node is down. NFS mounts are hard, so writers wait instead of losing data. Remove application stacks before the storage node when you tear down a cluster.
• Unencrypted NFS. Shared storage needs every node on a private network: initialise and join the Swarm with --advertise-addr and --data-path-addr set to private addresses (or WireGuard). With more than one node, mm refuses to plan shared volumes or routes while any node address is public, and mm target swarm nodes warns about it. The storage node publishes port 2049 through Docker, which bypasses UFW and firewalld; to keep it off the public interface, add a DOCKER-USER rule such as iptables -I DOCKER-USER ! -i eth1 -p tcp -m conntrack --ctorigdstport 2049 --ctdir ORIGINAL -j DROP, with eth1 the private interface.
• No failover for single-node placement. A service pinned to one node stops when that node stops.
• Emergency blocks are per manager. mmd access block takes effect only on the manager where you run it. Block the key on every manager.
• Shared storage carries ingress. While the storage node is down, edges keep serving the routes and certificates they already loaded, but new route changes and certificate renewals wait for it.
• Local HTTPS. Routes that use mm's local certificate authority (.test and similar development domains) are not available on Swarm. Use a public domain, or https: false.
• Rolling ingress updates. Changing the Traefik image or ingress nodes restarts edges one node at a time; each node is briefly unavailable, so keep your load balancer's health checks on.
• Per-member history. Each mmd keeps its own deployment history. After failing over to another member, plan again; rollback uses that member's history.
• Cluster commands through mmd need a project. Access is granted per project and environment, so run them from the project's checkout. Outside one, they use this machine's Docker engine.