Fleet and regions

One relay is enough to start. A fleet is several relays in several regions, under one control API: viewers get the nearest healthy relay, a dead relay moves its viewers to the next, and you are told.

ShapeCopy Anchor Link

A region is an origin and its edges. Publishers connect to the home region's origin. Every other region's origin pulls from the home origin as an internal peer, and every edge pulls from its own region's origin. Viewers connect to edges only.

              publisher
                  |
   home:   [origin east] <---------- [origin west]
              |    |                    |    |
           [edge] [edge]             [edge] [edge]
              viewers                   viewers

One control API serves the whole fleet. It never connects to a relay. Each relay host runs nearglass-meter --node <id>, the relay's agent (the same program that ships usage). Every 2 s the agent:

• reads the relay's admin port (is it answering, live sessions and publishers per scope, send rate, whether the relay says it is saturated) and the relay's CPU and memory from /proc, and posts them to POST /v1/fleet/heartbeat;
• takes back token ids to close and closes them through the admin port, so a lost seat, a kick or an ended turn reaches every relay in every region within a heartbeat;
• takes back the scope file when it changed and writes it where the relay reads it, so a token the API minted is valid on every relay;
• takes back the routes (the relay's coordinator file, naming each stream's namespace at the relay's upstream) and merges them under the relay's own lock. A relay restarted with an empty file gets them again at the next beat.

The agent calls out, so it needs no inbound port and the API holds no relay credential. The relay's admin port stays on loopback.

The fleet fileCopy Anchor Link

NEARGLASS_API_FLEET_FILE names a JSON file, read at start. With it, the API needs no NEARGLASS_API_RELAYS.

{ "home": "east",
  "regions": [
    { "id": "east", "name": "East",
      "origin": { "id": "east-origin", "url": "https://origin.east.example.com/" },
      "edges": [ { "id": "east-edge1", "url": "https://edge1.east.example.com/" } ] },
    { "id": "west", "name": "West",
      "origin": { "id": "west-origin", "url": "https://origin.west.example.com/" },
      "edges": [ { "id": "west-edge1", "url": "https://edge1.west.example.com/" } ] } ],
  "heartbeat_s": 2, "down_after_s": 6, "per_region": 2 }

Every region needs an origin and at least one edge. A bad file stops the API with the reason, for example fleet: node id r1-edge1 twice or fleet: node r2-edge1: https:// only.

Start an agent on each relay host with the node's id:

nearglass-meter --log /var/log/nearglass-relay/sessions.jsonl --api https://api.nearglass.com/ \
  --key-file /etc/nearglass-control-api/meter.key --relay-id east-origin --node east-origin \
  --admin-url http://127.0.0.1:4480 --admin-key-file /etc/nearglass-control-api/relay-admin.key \
  --scope-file /etc/nearglass-relay/scope.json \
  --coordinator-file /run/nearglass-relay/coordinator.json \
  --relay-pid-file /run/nearglass-relay/relay.pid \
  --cpu-cores 4 --mem-limit-mb 3072

--relay-pid-file is the relay's process id, for its CPU and memory (without it the agent reports the host's). --cpu-cores is the cores the relay may use and --mem-limit-mb its memory bound, which the shares are measured against (CPU defaults to all the host's cores; without a memory limit the host's memory in use is reported). Start agents before relays, and upgrade the API before the agents: the heartbeat refuses fields it does not know.

The relays reach each other as internal peers: list each peer with --internal-peer <address> on the relay, an address your private network authenticates (see Self-hosting).

What a node isCopy Anchor Link

Each node is unknown (not heard from since the API started), up, saturated or down. A node is down when no heartbeat arrives for down_after_s, or when its agent says the relay's admin port does not answer. A node still starting gets 30 s from the API's start or its agent's first beat before it is called down, so a deploy that starts hosts one by one does not page.

A relay judges its own saturation (--admission, on by default): it measures how busy the runtime serving viewers is, refuses new viewers over --admission-busy (0.85) for 2 s, and admits again only under 0.70 for 10 s, so an edge near its limit does not flap. The agent carries that verdict and the API leaves the edge out of viewers' lists at the next beat. For a relay with its rule off, the CPU and memory thresholds above apply.

What a viewer getsCopy Anchor Link

A viewer's descriptor (from a viewer token, admit, or a token refresh) lists, for each region in the file's order, the least loaded edges that are up, at most per_region, each with its region and id. The player picks among them: see Relay selection.

An edge that is down or saturated is left out. A region whose origin is down is left out while another region can serve. With no edge up anywhere the list falls back to saturated edges, then to every edge: the relay still checks the token, so the list is a hint, never access.

AlertsCopy Anchor Link

publisher.gone is per account (scope), not per stream, and exists only with a fleet file.

The operator's channel is a webhook URL in a file, never in the environment or on a command line (the service refuses to start if NEARGLASS_API_OPERATOR_ALERT_URL is set):

NEARGLASS_API_OPERATOR_ALERT_URL_FILE=/run/credentials/nearglass-control-api.service/alert-webhook.url
NEARGLASS_API_OPERATOR_ALERT_FORMAT=discord      # or slack, or json
NEARGLASS_API_OPERATOR_ALERT_SECRET_FILE=...     # json only

The file is root-only (mode 600), its first line the URL, handed to the service with LoadCredential=alert-webhook.url:/etc/nearglass-control-api/alert-webhook.url. An empty file means no channel, with a warning in the log. The URL is never logged and is removed from delivery errors. Formats:

Every alert is also logged and the newest 200 are kept for GET /v1/fleet. Unlike a customer's webhook, the operator's URL may be a private address. An event as the json format delivers it:

{ "type": "relay.down", "at": "2026-11-01T19:18:06Z", "node": "east-origin", "region": "east",
  "role": "origin", "url": "https://origin.east.example.com/",
  "reason": "no heartbeat", "last_heartbeat_s_ago": 6 }

reason is never heard from for a node whose agent never beat.

Looking at itCopy Anchor Link

GET /v1/status (and GET /status, a page that refreshes from it every 5 s) needs no key. It shows the overall state (up, degraded, down), each region with its relays (role, state, CPU and memory shares, sessions, last heartbeat) and the recent relay incidents, and no URLs, scopes, accounts or spend:

{ "state": "degraded", "home": "east", "at": "2026-11-01T19:17:16Z",
  "regions": [ { "id": "east", "name": "East", "state": "up", "edges": 1, "edges_up": 1,
    "nodes": [ { "id": "east-origin", "role": "origin", "status": "up",
                 "cpu": 0.001, "mem": 0.223, "sessions": 0, "send_mbps": 0.0,
                 "last_seen_s": 0, "since_s": 4 } ] } ],
  "incidents": [ { "type": "relay.down", "node": "west-edge1", "region": "west", "at": "2026-11-01T19:17:40Z" } ] }

GET /v1/fleet with the operator key (ok_...) is the fuller view: each node's URL, admission state and publishers per scope, the recent alerts of every kind, and viewer_relays, the list a descriptor would carry now. The name on the page is NEARGLASS_API_STATUS_NAME.

What was run, and what was notCopy Anchor Link

Run on one host with the packaged binaries: a fleet file with two regions, an agent for one real relay, the other three nodes never started. The agent beat (GET /v1/fleet showed its CPU, memory and admission state), the other nodes were called down 30 s after the API's start (never heard from), killing the agent raised relay.down for its node 6 s later (no heartbeat), the alerts arrived at a local receiver signed and verified, the status page and /v1/status showed the regions and incidents, a heartbeat without the meter key was refused (401), and the agent wrote the scope file it was given.