Set up a remote host

Remote hosts run mmd, the mm daemon. Developers connect over SSH to a single locked mm account whose only command is the relay to mmd. The daemon checks each request against the project's users file and runs the operation itself. Nobody needs a shell, sudo, or the host's Docker socket.

mm → SSH (account mm, relay only) → mmd → container runtime

mmd fetches each deployed commit from the project's git origin with its own read-only key, so a deploy runs exactly what is committed.

This is an advanced setup. mm target setup installs everything on the host in one reviewed step; you need root on the host only for that step.

Requirements#

On the host:

• Debian or Ubuntu with systemd and OpenSSH, and an admin account that can use sudo. Setup installs Docker Engine from Docker's own repository.
• /etc/ssh/sshd_config includes /etc/ssh/sshd_config.d/*.conf before its own settings. Debian and Ubuntu do this by default.

On your machine:

• ansible-playbook (in /usr/bin or /usr/local/bin) and ssh-keyscan.
• The admin account's key: either the target's own key file (mm target add --identity-file), or an SSH agent (SSH_AUTH_SOCK) holding the key.
• mm and mmd built for the host, from the same release. A Linux mm installed with --with-daemon has mmd beside it, and setup uses that pair by default. Otherwise, extract the host's Linux release archive and pass its directory with --host-binaries.
• A break-glass key: an Ed25519 key pair that is admin on every project the host runs. Keep the private half offline:
  shcopy

  ssh-keygen -t ed25519 -C ops@offline -f ops-recovery
  

Set up the host#

Add the host as a private target that logs in as the admin account. Only setup uses it; developers connect through the repo's target as mm:

mm target add prod-eu-setup --host prod-eu.example.com --user admin \
  --host-key-fingerprint SHA256:REVIEWED_HOST_KEY_FINGERPRINT

Plan the setup for the environment the host runs. --origin adds the remote access layer to the plan:

mm --environment production --target prod-eu-setup --runtime docker-compose \
  target setup plan \
  --origin git@github.com:acme/shop.git \
  --origin-host-key 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl' \
  --break-glass "ops-recovery=$(cat ops-recovery.pub)"

The plan lists the install-access step, the origin, the break-glass keys, and the path and digest of each binary it will install. Review it, then run the apply command it prints. That command repeats the same flags, because the plan hash covers them:

mm --environment production --target prod-eu-setup --runtime docker-compose \
  target setup apply --plan PLAN_HASH --origin … --break-glass … --host-binaries …

Setup first runs the project's bootstrap playbook, which installs the container runtime. Then it runs mm's own access playbook, which:

1. Installs git and the OpenSSH server.
2. Installs mm at /usr/bin/mm and mmd at /usr/libexec/mmd, and stops if they report different releases.
3. Creates the mm account. It has no password, its shell is the relay (/usr/libexec/mm-relay-shell), and its only group is mm-relay. Setup stops if the account is in any other group.
4. Writes /etc/mm/mmd.json and installs mmd.service.
5. Enrolls the project in /etc/mm/host.yaml with the origin, its pinned host key, the environment, and the break-glass keys. Creates the deploy key at /var/lib/mm/projects/PROJECT_ID/deploy-key, unless a key is already there.
6. Installs the sshd drop-in and checks it with sshd -t and sshd -T: the mm account must get only the relay, with no TTY or forwarding. If the check fails, setup removes a new drop-in and does not reload sshd.
7. Writes /etc/ssh/mm_authorized_keys, starts mmd, and checks it is running.

Expected result:

Bootstrapped target prod-eu-setup (…)
  …
  verified: remote-access
  mm and mmd: 0.1.0
  host key:   ssh-ed25519 AAAA…
  daemon:     0f3c…
  deploy key: ssh-ed25519 AAAA… mmd-deploy-019c527b-…

Add the deploy key to the git host as a read-only deploy key. Keep the host key and daemon identity for the repo target in the next section.

Run setup again to upgrade mm and mmd, to add another environment, or to enroll another project from that project's checkout. Setup keeps the other projects, ceilings, and exceptions in host.yaml, and adds break-glass keys without replacing existing ones. It rewrites host.yaml from its parsed contents, so comments in that file are lost.

Adjust the host policy#

/etc/mm/host.yaml is the host's own limit on every project. It is root-owned mode 0600. Projects cannot change it. Setup writes the project's name, origin, environments, and break-glass keys; add ceilings, the first deploy-from rule, and workload exceptions by hand:

schema-version: 2
break-glass: # admin everywhere; keep the private key offline
  - name: ops-recovery
    key: ssh-ed25519 AAAA… ops@offline
projects:
  019c527b-6242-7b43-bfd4-bd60e27c1de3: # the project's id from .config/mm.yaml
    name: shop
    origin: git@github.com:acme/shop.git
    origin-host-key: ssh-ed25519 AAAA… # the git host's key, pinned
    environments: [staging, production] # environments this host runs
    initial-deploy-from:
      production: [main] # until the repo sets deploy-from
    ceiling:
      production: { level: deploy } # the most the repo may grant here

• Break-glass keys always work, even when a users file locks everyone out.
• ceiling caps what the repo can grant per environment. Without one, the repo may grant up to admin.
• origin is the only place mmd fetches code from, with the key at /var/lib/mm/projects/PROJECT_ID/deploy-key.

After editing, check the file and restart the daemon:

mmd access validate
mmd access render-keys
systemctl restart mmd.service

Connect a client#

Commit the host's public connection facts to the repo, so every clone can reach the environment. Use the host key and daemon identity that setup printed. If someone else ran setup, get them through a trusted channel; on the host they are /etc/ssh/ssh_host_ed25519_key.pub and mmd --print-identity. Name the target differently from the private setup target, because a private target with the same name takes precedence:

# .config/mm.yaml
targets:
  prod-eu:
    host: prod-eu.example.com
    host-key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA…
    daemon-fingerprint: 0f3c… # 64 characters
environments:
  production:
    target: prod-eu
    deploy-from: [main]

These are public keys, not secrets. mm checks both on every connection and connects as mm. The first time a client connects it pins them; if a later commit changes either, the client refuses until you review the change:

mm target trust prod-eu          # shows pinned and new keys
mm target trust prod-eu --yes    # accept after checking with the host admin

Swarm managers as failover members#

For a multi-node Swarm where several managers run mmd, list the other managers under members. Each one takes the same facts as the target's own host; port defaults to 22:

targets:
  production:
    host: mgr-1.example.com
    host-key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA…
    daemon-fingerprint: 0f3c… # 64 characters
    members:
      - host: mgr-2.example.com
        host-key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA…
        daemon-fingerprint: 9b1e…
      - host: mgr-3.example.com
        port: 2222
        host-key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA…
        daemon-fingerprint: 4d72…

mm connects to host first, then to each member in list order. It moves to the next one only when it cannot connect, and never resends a request that reached a daemon. Messages name members by position: member-1 is the first entry.

Members' keys are pinned with the target's own keys. A commit that adds a member, or changes any member's host key or daemon identity, is refused until you review it with mm target trust. Removing or reordering members only drops keys you already trusted, so it needs no review. A removed member that comes back later needs review again.

Declare a cluster by node#

A target can list its hosts under nodes: instead of host and members. Each node names the hostname it gets and its address on the private network the Swarm uses:

targets:
  production:
    nodes:
      - name: app-1
        host: 203.0.113.11
        private-address: 10.77.0.1
        host-key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA…
        daemon-fingerprint: 0f3c… # 64 characters
      - {name: app-2, host: 203.0.113.12, private-address: 10.77.0.2, …}
      - {name: app-3, host: 203.0.113.13, private-address: 10.77.0.3, …}
    swarm:
      private-network:
        wireguard: {subnet: 10.77.0.0/24} # interface wg0, port 51820
        # or: interface: eth1
      address-pool: 10.200.0.0/16 # overlay networks; Docker's default is 10.0.0.0/8
      address-pool-mask: 24
      storage: app-1
      groups: {db: [app-1]}
      ingress: [app-2, app-3] # omit for every manager
    setup:
      user: root
      origin: git@github.com:example/app.git
      origin-host-key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA…
      deploy-key: {secret: origin.deploy-key, public-key: ssh-ed25519 AAAA… mm-deploy}
      break-glass:
        - {name: ops-recovery, key: ssh-ed25519 AAAA… ops}

Node order is failover order, as with members, and messages name each node. mm connects to a node only once its host-key and daemon-fingerprint are recorded; until then, commands for the environment stop and say so.

Set up a cluster from the declaration#

Create each node with this device's mm key (mm key show) authorized for the setup.user account, usually root. Then, from the repo:

mm -e production target setup deploy-key   # once: prints the public key
gh repo deploy-key add - --title 'mm production' <<<'ssh-ed25519 AAAA… mm-deploy-app-production'
# set setup.deploy-key.public-key to that line and commit
mm config split targets                     # once: record rewrites .config/mm/targets.yaml
mm -e production target setup plan          # scans host keys, binds secret revisions
mm -e production target setup apply --plan HASH --yes
mm -e production target setup record        # writes host keys and daemon identities
git add .config/mm/targets.yaml && git commit -m "Record production nodes" && git push

apply runs these steps in order, and each one is safe to repeat:

1. prepare-nodes: sets each hostname, writes /etc/mm/node.env (MM_NODE_NAME, MM_PRIVATE_INTERFACE, MM_PRIVATE_ADDRESS), and builds the WireGuard mesh when declared, then checks every node reaches the others.
2. provision-nodes: the project's .config/ansible/bootstrap.yml and the remote access layer on each node, as single-host setup runs them.
3. form-swarm: docker swarm init on the first node and join as managers on the others, on the private addresses and the overlay pool. A rebuilt node's stale entry is removed; a node in another Swarm stops setup.
4. serve-storage: when swarm.storage names a node, installs that node's kernel NFS server for shared volumes (NFSv4.2, exported only to the nodes' private addresses) and stops mm's NFS container if an earlier deploy started it.
5. push-secrets: every age secret set for the environment, and the deploy key, sealed to each node's own store with the same revisions.
6. label-nodes: storage, group, and ingress labels (mmd swarm converge), including io.mm.storage-server=host on the storage node.
7. sync-access: the first users-file sync as root on the first node (mmd access sync --require-shared) at HEAD, or --commit SHA. The commit must be pushed to a branch the environment deploys from.
8. verify: each node's daemon identity, host key, and version.

The plan binds each node's scanned SSH host key. A key that differs from the recorded one stops the plan; after rebuilding a node, plan again with --trust-new-host-keys and check the fingerprints it prints. The plan also binds each secret's revision, so changing a secret after review means planning again.

mm config validate checks that:

• node names are lowercase hostname labels, and hosts and private addresses are unique;
• private addresses are private, inside the WireGuard subnet when one is used, and outside the overlay address pool;
• storage, groups, and ingress name declared nodes;
• there are between one and seven managers (an even number is a warning);
• setup.deploy-key.secret names a secret declared under secrets:.

A private target with the same name (mm target add) takes precedence, for example to go through a jump host.

mm authenticates with this device's mm key (mm key create), never with ~/.ssh or an SSH agent. In CI, set MM_SSH_KEY to a machine key's private key instead.

Apply the users file for the first time#

Nobody but the break-glass key can connect until an environment reconciles the repo's users file. Push the commit, then reconcile it as root on the host, without the relay:

sudo mmd access sync --project PROJECT_ID --environment production \
  --commit 3f2c9a1e5b7d4c6f8a0b2d4e6f8a1c3e5b7d9f0a --dry-run   # review
sudo mmd access sync --project PROJECT_ID --environment production \
  --commit 3f2c9a1e5b7d4c6f8a0b2d4e6f8a1c3e5b7d9f0a

mmd access sync checks the commit exactly as mm team sync does: the host fetches it from the enrolled origin with the deploy key, it must be on a branch that deploy-from allows, and the host ceilings and Compose restrictions apply. It then writes the environment's users, rewrites authorized_keys, records host-root in the audit log, and prints a JSON summary:

{
  "project": "019c527b-6242-7b43-bfd4-bd60e27c1de3",
  "environment": "production",
  "commit": "3f2c9a1e5b7d4c6f8a0b2d4e6f8a1c3e5b7d9f0a",
  "previous-commit": null,
  "changed": true,
  "dry-run": false,
  "deploy-from": ["main"],
  "users": [
    { "user": "alice", "machine": false, "access": { "level": "deploy" }, "keys": ["SHA256:…"] }
  ],
  "changes": [{ "user": "alice", "after": { "level": "deploy" }, "keys-added": ["SHA256:…"] }],
  "sharing": null
}

--dry-run prints the same summary and writes no access state. Running the command again at the same commit prints "changed": false and changes nothing. It only moves access forward: it refuses a commit that the commit already applied is not an ancestor of. Go back with a reviewed mm team sync.

On a Swarm manager the record is shared like a team sync: sharing reports the shared record and which managers confirmed it, and the other managers' mmd adopt it on their own. Add --require-shared to refuse when the host is not a Swarm manager, so the record cannot stay on one host by mistake.

The break-glass key works too: MM_SSH_KEY="$(cat ops-recovery)" mm --environment production team sync.

After that, every mm deploy apply and mm team sync re-applies the users file from the commit it uses. See remote access.

Finally, remove personal login accounts and sudo from the host, and store the break-glass private key offline.

Use the host#

mm --environment production deploy plan             # the host fetches HEAD's commit
mm --environment production deploy apply --plan PLAN_HASH --yes
mm --environment production service status
mm --environment production service logs api

deploy plan sends a commit ID. The host fetches it from the origin, checks it is on an allowed branch, checks it out under /srv/mm/projects, and plans from that checkout. Push the commit before planning.

Remote hosts never build. Modules that mm builds deploy from a build receipt: publish the images, write the receipt with mm build receipt, and pass it to deploy plan --receipt. The host checks that the receipt is a build of the planned commit, either for this environment or for one whose configuration builds the same units. See deploy a build to a remote host.

Deploy from CI#

Build once in a CI environment, then deploy that build to each environment with a machine key. Declare a ci environment in .config/mm.yaml for the build job. In the build job:

mm --environment ci build --publish --yes
mm --environment ci build receipt build-receipt.json

Keep build-receipt.json as a job artifact. In each deploy job, from a checkout of the same commit, with MM_SSH_KEY set to a key the users file grants deploy on the environment:

PLAN=$(mm --output json --environment production deploy plan --receipt build-receipt.json | jq -r .binding.plan_hash)
mm --environment production deploy apply --plan "$PLAN" --yes

The host deploys the receipt's digests when production's configuration at that commit builds exactly the same units as ci did: the same modules, platforms, image repositories, variants, and build inputs. Otherwise it refuses and names the first module that differs. A module with a build secret can only be promoted when every secret is listed under credential-only-secrets, because other secrets can hold different values per environment. The plan's plan.build_receipt.environment and the host's audit log (build_receipt_environment) record the environment the receipt was built for.

The receipt holds image names, digests, and build settings, but no secret values.

To skip rebuilding unchanged modules on a fresh CI runner, import the previous publication's receipt before building. mm build --publish looks for reusable images only in mm's local state, not in the registry:

PLAN=$(mm --output json --environment ci build import previous-receipt.json --cache-only | jq -r .plan_hash)
mm --environment ci build import previous-receipt.json --cache-only --plan "$PLAN" --yes

Remote exec supports output and cancellation. It currently rejects --stdin, --tty, and interactive shell.

Secrets and the deploy key on the host#

The host resolves a deployment's age secrets from its own store, /var/lib/mm-home/secrets/PROJECT_ID.age, opened with the age identity in /var/lib/mm-home/identities/age.key. Two root commands copy secrets from the operator's store into it. They exchange a sealed bundle that mm builds on the operator's machine; you cannot write one by hand.

Print the host store's age recipient. The command creates the store and the identity if they are missing, and prints the same recipient every time:

mmd secrets recipient --project PROJECT_ID --environment integration

{ "recipient": "age1…" }

Import a bundle sealed to that recipient. It is read from stdin, either as ASCII-armored or as binary age data, up to 24 MiB:

mmd secrets import --project PROJECT_ID --environment integration < bundle.age

{
  "project": "019c527b-6242-7b43-bfd4-bd60e27c1de3",
  "environment": "integration",
  "secrets": {
    "age://database-password": { "revision": "4f0c…", "changed": true }
  },
  "deploy-key": {
    "public-key": "ssh-ed25519 AAAA… mm-deploy-integration",
    "changed": false
  }
}

• The host must already run the environment for the project (mmd access enroll). A bundle for another project or environment is refused.
• Each secret keeps the revision it has in the operator's store, so revisions on the host and on the operator's machine compare equal. A secret whose revision is already active is left unchanged. Secrets that the bundle does not list stay in the host store.
• A bundle can carry the environment's deploy key. The import checks that its private half matches its public line, then replaces /var/lib/mm/projects/PROJECT_ID/deploy-key and deploy-key.pub. Later setup runs keep that key, so one key on the git host serves every node.
• The output names references and revisions only, never values.

Recover or upgrade#

After a connection drops during a change, inspect service status and /var/lib/mm/audit.jsonl before retrying. A lost response does not mean the operation failed. Plans expire after ten minutes and cannot be replayed.

Upgrade mmd and mm together by running setup again with the new release; a protocol mismatch reports MM_REMOTE_UPGRADE_REQUIRED. Setup keeps /var/lib/mm, so the daemon identity it prints stays the same. On a Swarm, upgrade every manager, then sync each environment once (see access on every manager).

Hosts set up with the earlier per-developer accounts and access.yaml must move to this layout: mmd refuses a schema-1 access.yaml. Write host.yaml, add each developer to the users file with mm team join, sync, check with mm team whoami, and then delete the old mm-* accounts.