Operations Hardening

This page is the operator-facing checklist for the production Docker stack. It covers the parts that protect availability and blast radius: MariaDB durability, connection limits, Redis/cache ceilings, request limits, signup controls, and container hardening.

Database Resilience

The multi-container stack stores primary state in the mariadb-data named volume. The database is internal-only: it publishes no host port and is reachable only on the Compose network.

The MariaDB service now starts with explicit safety defaults:

The db healthcheck runs a real SELECT 1 against the configured application database instead of only checking that the MariaDB process is alive.

Connection Budget

The server container runs several Node services under supervisord. Each MySQL-backed package uses DB_CONNECTION_LIMIT; the current default is 20. For a single server container, DB_MAX_CONNECTIONS=150 leaves headroom for auth, syncing, revisions, websocket/legacy packages, migration/admin tasks, and a short overlap during restarts.

If you scale out more server containers, increase DB_MAX_CONNECTIONS and DB_MEM_LIMIT together. A practical starting formula is:

DB_MAX_CONNECTIONS >= (server replicas * DB-using processes per replica * DB_CONNECTION_LIMIT) + admin headroom

Do not raise the per-process pool first. Raise it only when live metrics show connection wait time is the bottleneck.

Backup And Restore

Take logical database backups while the stack is running:

docker compose exec db sh -c \
  'exec mariadb-dump -uroot -p"$MYSQL_ROOT_PASSWORD" --single-transaction --routines "$MYSQL_DATABASE"' \
  > backup.sql

Restore into a running stack:

docker compose exec -T db sh -c \
  'exec mariadb -uroot -p"$MYSQL_ROOT_PASSWORD" "$MYSQL_DATABASE"' \
  < backup.sql

Run the non-destructive restore drill before trusting a backup procedure:

node scripts/verify-backup-restore.mjs

The drill dumps the live MariaDB database, restores it into a temporary srn_restore_* database, compares the restored table list, row counts, and table checksums, then drops only the temporary database. Run it when the system is idle so live writes do not change the source database while the comparison is in progress. Use --keep-backup or --output backup.sql when you want to keep the generated dump for inspection.

flowchart LR
  Live[("Live MariaDB")]
  Dump["Logical SQL dump"]
  Restore[("Temporary srn_restore database")]
  Compare{"Tables, row counts,<br/>and checksums match?"}
  Pass["Report success"]
  Fail["Report mismatch"]
  Cleanup["Drop temporary database<br/>and remove temporary dump"]

  Live -->|"mariadb-dump"| Dump
  Dump -->|"restore"| Restore
  Live --> Compare
  Restore --> Compare
  Compare -->|yes| Pass
  Compare -->|no| Fail
  Pass --> Cleanup
  Fail --> Cleanup

Back up the uploads volume with the database if you use file attachments. Keep .env backed up separately. Losing server secrets can invalidate sessions or make server-side encrypted settings unreadable.

Restart Safety

The admin UI can restart Redis and MariaDB only when the optional ops profile is running and the server has SERVICE_CONTROL_DOCKER_ENABLED=true. The server never receives the raw Docker socket. It talks to docker-socket-proxy, which is configured to permit only container restart endpoints.

Database restarts are intentionally visible operations. Use them for controlled maintenance, not routine remediation. Back up before risky changes such as database image upgrades, memory reductions, or major version moves.

Operation Limits

The stack has several independent limits. They are intentionally layered because each protects a different failure mode.

Unauthenticated rate limits are Redis-backed and fail open if Redis is down, so a cache outage cannot lock legitimate users out of their notes. The tradeoff is that IP blocks and rate limits temporarily degrade during a Redis outage.

For public instances, consider setting:

REGISTRATION_INVITE_ONLY=true
REGISTRATION_APPROVAL_REQUIRED=true
REGISTRATION_SIGNUPS_PER_IP_MAX=5
REGISTRATION_SIGNUPS_PER_IP_WINDOW_HOURS=24
REGISTRATION_MAX_TOTAL_ACCOUNTS=<your planned capacity>

For expensive authenticated operations, consider:

RATE_LIMIT_USER_MAX=30
RATE_LIMIT_USER_WINDOW_SECONDS=60
ASSISTANT_DAILY_REQUEST_LIMIT=100
ASSISTANT_5H_TOKEN_LIMIT=200000
ASSISTANT_WEEKLY_TOKEN_LIMIT=1000000

Tune these to your hardware and user count. A small personal instance can keep most authenticated limits off; a public instance should set them deliberately.

Redis Limits

Redis is used for cache, rate-limit counters, transient operation state, and event plumbing. It is persisted with append-only files, but it is not the source of truth for notes.

The Compose service now sets:

CACHE_MEM_LIMIT=256m
CACHE_MAXMEMORY=192mb
CACHE_MAXMEMORY_POLICY=noeviction

CACHE_MAXMEMORY is below the container memory limit so Redis returns controlled write errors instead of being OOM-killed. noeviction avoids silently discarding keys that may represent rate-limit or transient operation state. If you operate a high-churn instance and accept eviction semantics, choose a policy explicitly.

WebSocket Sync Safety

Worker WebSocket sync is the preferred transport, with the durable HTTP command path as fallback. The socket transport is advertised only when three conditions hold: WEB_SOCKET_CONNECTION_TOKEN_SECRET is set, the WEBSOCKET_SYNC_ENABLED kill switch is not the exact string false, and the fleet-shared Redis ticket, command-lease and socket-budget stores report ready. An admissible browser origin is checked per upgrade, not at negotiation: an explicit WEBSOCKET_SYNC_ALLOWED_ORIGINS entry, an origin derived from PUBLIC_URL, or an origin equal to the upgrade’s own Host all pass.

The durable gRPC command port is a separate, fourth condition and gates only the SYNC_ITEMS operation. When it is unbound the socket opens and serves live collaboration, API RPC, invite events, the assistant stream and the files lane; only item sync uses HTTP. Operators read the unmet conditions as stable codes: WEB_SOCKET_CONNECTION_TOKEN_SECRET_MISSING, WEBSOCKET_SYNC_DISABLED_BY_CONFIGURATION, REDIS_UNBOUND and SYNCING_SERVER_GRPC_UNBOUND, each with a remedy naming a variable and never a value. This keeps a partial rollout from accepting work that another replica cannot resume safely, without letting one unmet server-to-server dependency close five capabilities that do not depend on it.

Single-container and LXC deployments accept only REDIS_HOST/REDIS_PORT for this socket plane; that connector has no Redis authentication or TLS support. Use it only on the same private trusted network. Leaving Redis unset there does not merely disable item sync over the socket: no realtime gateway is started at all, so push, live collaboration, presence, comments, realtime invites and push-approved multi-factor prompts are absent and only HTTP and periodic sync remain.

WEBSOCKET_GATEWAY_INTERNAL_SECRET is an internet-facing credential, not merely a server-internal one: whoever holds it can mint a legacy connection token for any user and session. The three shipped front doors blank X-Internal-Secret on /sockets, and the gateway refuses an internal mint that arrives proxied or from a non-loopback peer. Keep the value out of clients, browsers and build artefacts regardless.

That refusal is a required check rather than an assumption. The container lane runs a script inside the server container, the one vantage point that reaches both the public front door and loopback, and fails if a front-door mint carrying the real secret returns a token — or if the same mint from loopback does not. See Realtime proofs in the container lane.

On a multi-container upgrade from a keyless release, a normal setup rerun adds only this missing key using one atomic, permission-preserving .env migration and timestamped backup. It refuses malformed or duplicate assignments and does not rotate an existing valid key.

The browser first checks the unauthenticated capability document, then mints a one-use ticket over its authenticated HTTP session. The upgrade URL contains no credential or ticket; the ticket arrives in the first AUTH frame. A transport probe therefore needs only an exact allowed Origin and a valid WebSocket key:

curl -fsS "$PUBLIC_URL/v1/sockets/sync/capabilities"
# enabled: {"capabilities":[{"id":"ws-sync","version":1,"endpoint":"/sockets/sync"}]}

curl -sik --http1.1 --max-time 2 \
  -H "Origin: $PUBLIC_URL" \
  -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
  -H 'Sec-WebSocket-Version: 13' \
  -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
  "$PUBLIC_URL/sockets/sync"
# expect 101; the unauthenticated probe then closes because it sends no AUTH frame

A ticket request refused only because a shared store has not reported ready yet answers 503 with Retry-After: 5 and transient: true in the body, so a page load that races a restart retries instead of negotiating HTTP-only for the whole session. A refusal with a configuration cause carries no transient flag, because retrying cannot fix it.

Legacy upgrades are accepted on the exact path /sockets only. Any other path under that prefix is closed with 1008 unknown path; /sockets/sync remains the worker lane.

During shutdown the public capability provider is cleared before the gateway is drained, and the HTTP listener closes only after that drain. Monitor Redis latency/errors and empty capability responses together; HTTP sync should remain available while the socket plane is intentionally closed.

Docker Image And Runtime Hardening

The stack is designed so the app front door is the only publicly reachable Standard Red Notes service. The server, database, cache, queue emulator, MCP bridge, and docker socket proxy remain internal-only. The optional n8n profile publishes a loopback-only development port; production should remove that mapping and use a dedicated TLS proxy hostname.

Runtime hardening currently includes:

read_only is not globally enabled because several containers intentionally rewrite runtime config, write supervisor logs, write sqlite/uploads, or maintain database/cache state. Writable paths are constrained with named volumes and tmpfs where the current images support it safely.

Image Pinning

Compose supports image override variables for production pinning:

MARIADB_IMAGE=mariadb:12.3.2
REDIS_IMAGE=redis:8.8.0-alpine
FLOCI_IMAGE=floci/floci:1.5.33-compat
N8N_IMAGE=n8nio/n8n:2.32.6
DOCKER_SOCKET_PROXY_IMAGE=tecnativa/docker-socket-proxy:v0.4.2

For reproducible production pulls, replace mutable tags with exact tags or repo@sha256:<digest> values after your own image update review. This is especially important for optional services that otherwise track moving upstream tags, such as n8n.

Verification Commands

Check the rendered Compose model:

docker compose config
docker compose -f docker-compose.single.yml config

Check service health:

docker compose ps
docker compose exec db sh -c \
  'mariadb -uroot -p"$MYSQL_ROOT_PASSWORD" "$MYSQL_DATABASE" -e "SELECT 1"'

Check key MariaDB variables:

docker compose exec db sh -c \
  'mariadb -uroot -p"$MYSQL_ROOT_PASSWORD" "$MYSQL_DATABASE" -e "SHOW VARIABLES WHERE Variable_name IN (\"max_connections\", \"max_allowed_packet\", \"innodb_buffer_pool_size\", \"innodb_flush_log_at_trx_commit\", \"local_infile\")"'

Check Redis memory policy:

docker compose exec cache redis-cli CONFIG GET maxmemory
docker compose exec cache redis-cli CONFIG GET maxmemory-policy

Run the e2e safety gates after hardening changes:

$env:APP_URL = "http://localhost:3001"
npm --prefix e2e test -- app-opens.spec.ts --project=chromium
npm --prefix e2e test -- encryption-data-safety.spec.ts --project=chromium
npm --prefix e2e run test:ops-load
node scripts/verify-backup-restore.mjs

The ops load gate can be scaled without editing code:

$env:OPS_LOAD_NOTES = "250"
$env:OPS_LOAD_CLIENTS = "4"
$env:OPS_REDIS_WORKERS = "4"
$env:OPS_REDIS_OPS_PER_WORKER = "500"
npm --prefix e2e run test:ops-load

It registers a real account, pushes encrypted notes to the server, signs in parallel clients, verifies pulled note integrity, runs concurrent Redis SET/GET/INCR churn, checks Redis throughput, and confirms MariaDB persisted the expected note rows.