Monitoring

Four places say how the service is doing: the control API's endpoints, the relay's metrics port, the relay's admin port, and the logs. On the managed service you use the first only; on your own hosts you have all four.

The control APICopy Anchor Link

Stream statsCopy Anchor Link

GET /v1/streams/{id}/stats is for a dashboard polling every few seconds. The player never calls it.

curl -s $API/v1/streams/$ID/stats -H "Authorization: Bearer $SK"

Illustrative example. The values below are made up to show the shape; they are not a measurement.

{ "viewers": { "seats": 1, "queued": 0, "cap": 100, "queue_max": 200, "peak": 1, "hls": 0 },
  "seats_with_tokens": 1, "turns": 0, "ingest_token_at": "2026-11-01T12:00:03Z",
  "ingest": { "live": true, "since": "2026-11-01T12:00:02Z", "width": 1920, "height": 1080,
              "fps": 60, "kbps": 5980, "reconnects": 0 } }

Reading it also frees seats whose heartbeat stopped, so a stale count corrects itself.

The relay's metricsCopy Anchor Link

Start the relay with --metrics-addr 127.0.0.1:9090 (the package's unit does). It then serves Prometheus text at /metrics (on any path, with no key; keep it on loopback or a private network and scrape it from there). Counters appear once they have fired.

Watch moq_relay_admission_busy to see an edge approach its limit: it is the relay's own headroom, and 0.85 is where it starts to refuse.

The relay's admin portCopy Anchor Link

--admin-addr 127.0.0.1:4480 --admin-key-file ... (loopback only). With the key as a bearer, GET /usage:

Illustrative example. The values below are made up to show the shape; they are not a measurement.

{ "scopes": { "/tn_...": { "sessions": 1, "publishers": 1, "egress_bps": 1702,
      "bytes_out_closed": 0, "sessions_closed": 0, "bytes_out_live": 7055 } },
  "admission": { "mode": "on", "saturated": false, "busy": 0.003, "delay_ms": 1.9,
      "sessions": 0, "budget": 50, "admitted": 0, "returned": 0, "places": 0,
      "refused_saturated": 0, "refused_paced": 0, "episodes": 0, "changed_at_s": 0,
      "enter_busy": 0.85, "leave_busy": 0.7, "retry_ms": 2000 } }

Per scope: live sessions and publishers, the send rate, and the bytes sent by closed and by live sessions since the relay started. It is the number to check usage against: bytes_out_closed on each relay should add up to the bytes the API's usage shows for the same period. Without the key the port answers 401. It also serves the node agent's heartbeat (Fleet).

The session log (--session-log) holds one JSON line per closed token session: scope, namespace, token_id, sub, role, start_ms, end_ms, bytes_out, token_exp and close (closed, expired, admin).

LogsCopy Anchor Link

Services log to stdout, so systemd keeps them (journalctl -u nearglass-relay). The control API logs at info; set RUST_LOG (for example nearglass_control_api=debug) to change it. Lines worth searching for:

CanariesCopy Anchor Link

A canary is a viewer that watches a stream all the time, so you know it works before a customer does. The repository has two, and both are our own tooling for our demo, not part of the packages:

• tools/canary runs a headless browser against a page with the player for a window every hour, and records frames on screen, how far behind live the picture is (median and 99th percentile), reconnects and held pictures every second. It raises an alarm when a window's median is under 50 frames a second or it reconnects more than twice in 10 minutes.
• tools/audio-canary is a native listener that subscribes to the audio track the way the player does, plays it into a null device, and reports every dropout and its cause every 10 s.

To canary your own deployment, publish a looping file to a stream of its own (nearglass publish --file ... --loop, with its own key), and watch it with a real browser on another network, on a schedule. Read the player's statechange events and stats() (Embed the player), and alert on: no live within a few seconds of load, a mode of ll-hls where you expect moq, or reconnects.

What to alert onCopy Anchor Link

What was run, and what was notCopy Anchor Link

Run on one host from the packages: GET /v1/health, /v1/status, /status, /v1/fleet, /v1/tenant, /v1/usage and the stream stats (including seats, a full stream and the line), the relay's /metrics with its metric names and the admin GET /usage, with and without the key, and the session log.