Skip to content
GuideForOperatorWith or without Foxden Admin

Check the server's health ​

This page lets you find out whether the server and its apps are working, read their logs, and gather what someone helping you needs.

In Admin ​

  • Home shows the server's processors, memory and disk, and lists what Needs your attention, most urgent first. Each row opens the page that fixes it.
  • Server › System runs the server's own check of itself under Health check: ports, disk, DNS, mail and each app's settings. What needs you comes first, each with what was seen and what to do; Good to know holds reminders that are not problems.
  • Server › Products shows each app's version, whether it answers, and under To finish what each app still needs before everything works.
  • Server › Backups shows each app's newest backup and whether its off-site copy arrived.

With the CLI ​

One verdict: checkup ​

Run foxden checkup. It runs the server check, the status of each app, an email deliverability check and a test that the newest backup restores, then gives one verdict:

VerdictExit codeMeaning
fit0Everything checks out.
running, but1The apps run, and something needs your attention. The reason follows.
not ready3The server is not ready: nothing is installed, or something it needs is missing.

Run it once a week, and whenever something feels wrong.

Every check: doctor ​

foxden doctor runs each check on its own and prints, for each one, what it saw and the fix. Narrow it down:

  • foxden doctor --product fennec: one app's checks (the server-wide checks always run).
  • foxden doctor --fail-only: only warnings and failures, with their fixes.
  • foxden doctor --check backup-fresh: one check by its name.
  • foxden doctor --domain example.com: also check that the domain points at this server.

Each line starts with the check's name, such as health, sizing, dns, backup-fresh or email-spf. The messages, and what to do about each, are on Fix server problems.

What is running: status ​

foxden status --product fennec lists the app's containers, the image each runs, its version and whether it is healthy. foxden status --watch redraws it every 2 seconds until you press q.

Logs ​

foxden logs asks which service to read, then whether to follow it, show errors, or search. Or say it in one line:

  • foxden logs app --follow --product fennec: follow the app live; Ctrl-C stops.
  • foxden logs --errors --product fennec: errors from every service of the app.
  • foxden logs app --since 1h: the last hour.
  • foxden logs --grep timeout -C 3: every line with "timeout", three lines either side.
  • foxden logs caddy --product arctic: the web server of Arctic installed on its own, where certificate errors show.

Each app's services are named in its compose file: usually app, worker, db, migrate and updater. Each app's page lists its own, for example Running Fennec.

The shared web server in front of the apps is not part of any app, so foxden logs does not read it. On the server, run docker compose --project-directory /opt/foxden/ingress logs caddy to see its certificate errors.

Backups log to /var/log/<stack>-backup.log on the server (for Fennec, /var/log/codefox-crm-backup.log), and the weekly restore test to /var/log/<stack>-restore-verify.log.

A file for a support thread ​

foxden support-bundle --product fennec writes one file, foxden-support-fennec-<time>.json, readable only by you: the checks and what they found, the health answer, which containers run and the exact image of each, the last 200 log lines of each container, the backup state, disk use, and the names of the settings in .env. It never includes a setting's value. It does include the log lines as they are, which can hold customer names, email addresses and request bodies; --lines 0 leaves them out.

Health addresses ​

Each app answers a health address on the server, which foxden and the updater read. It returns JSON with ok and the build it runs (version).

AppHealth address on the server
Foxden Adminhttp://127.0.0.1:3020/api/health
Fennechttp://127.0.0.1:3010/api/health
Arctichttp://127.0.0.1:3000/api/health
Corsachttp://127.0.0.1:3400/api/healthz
Pampashttp://127.0.0.1:3410/api/healthz
Culpeohttp://127.0.0.1:3024/api/healthz
Capehttp://127.0.0.1:3430/api/healthz (and /api/healthz/deploy, which also waits for the call engine)
Libraryhttp://127.0.0.1:3480/api/health

The same path answers on each app's public address, for example https://crm.example.com/api/health, so an uptime monitor outside the server can watch it. Arctic installed on its own is behind its sign-in gate, so its public address asks for the gate's password.

Foxden is open source under Apache-2.0.