Operate
Troubleshooting
From a symptom to its cause and fix, with the error text the service and the player produce.
Find the message you see. Errors from the API are JSON,
{"error": {"code": "...", "message": "..."}}, and the player shows them as
Cannot play: <call>: HTTP <status> <code>: <message>. Log lines are
quoted as they are written, minus the timestamp.
Publish
| You see | Cause | Fix |
|---|---|---|
nothing plays; stats shows ingest.live: false (or ingest_token_at: null) | no encoder is sending video to the gateway | check the stream's publish.rtmp or publish.srt address, the port is open, and the key is the stream's stk_... |
gateway: RTMP publish refused or SRT publish refused with refused=UnknownKey | the key is not any stream's, or the stream was deleted or its key rotated | take the key from the stream's create answer; a rotated key is shown once |
refused=Protocol | the key is not allowed on this protocol | the stream's ingest list lacks rtmp or srt; a stream's ingest cannot be changed (PATCH changes only name, seats, line and admission), create a new one |
refused=InUse | another publisher is live on this key | stop the other encoder; a second one takes over after the first has been silent 5 s |
refused=Expired | a keys-file entry (no API) passed its expires | make a new key |
the encoder connects, then picture never starts; gateway: no keyframe for N s: set the encoder's keyframe interval (1 s) | the encoder sends keyframes too rarely | set the keyframe interval to 1 s (OBS and SRT) |
encoder disconnected after you rotated the key or deleted the stream; gateway: stream key revoked (deleted or rotated): its session ends | as logged | use the new key |
409 no_stream_key: this stream publishes with tokens only (ingest: sdk) | the stream takes no encoder | publish with a publisher token (nearglass publish --api), or create a stream with rtmp or srt in ingest |
encoders are refused for minutes; gateway: control API unreachable for 120 s: every stream key dropped | the gateway cannot reach the API | fix the route; it recovers by itself (control API reachable again) |
nearglass publish: Session(Failed ... invalid peer certificate: UnknownIssuer) | the relay's certificate is not from a CA the publisher trusts | use a CA-issued certificate, or --tls-root ca.pem |
nearglass publish: Session(Ended ... session ended: Ok(Err(Decode(More(1)))) and it reconnects over and over; relay: session authorization failed, closing code=2 reason="unauthorized" | the relay refused the publisher's token | see Tokens |
nearglass publish: Session(Ended ... namespace ended: Ok(Err(Serve(Closed(1)))) | the token is not allowed to publish this namespace: a viewer token, or another namespace | mint a publisher token for the stream's namespace |
Error: token file ...: token: bad base64url length or padding bits | the token file is not a token (extra text, a wrapped line) | the file holds the token only, as nearglass token mint prints it |
400 invalid_request: ttl: 1 to 604800 s on a publisher token | a lifetime of 0 or over 7 days | pick one in range |
Play
The player's status line is on the left; the API's reason follows Cannot play:.
| You see | Cause | Fix |
|---|---|---|
Cannot play: admission: HTTP 401 unauthorized: missing or unknown publishable key | the page's publishable-key is missing, wrong, or a secret key | use the account's pk_.... A secret key must never be in a page |
HTTP 403 server_admission: this stream admits through its backend (admission: server) | the stream's admission is server, the page asked for public | give the player a token-url (Mint viewer tokens), or create the stream with "admission":"public" |
HTTP 404 not_found: no such stream | wrong id, deleted, or it belongs to another account (other accounts' streams are 404, never 403) | check the id and the key's account |
HTTP 402 spend_cap_reached: this month's spending cap is reached (...) | the monthly cap | raise it (PATCH /v1/tenant) or wait for the month; viewers already in keep watching (Usage) |
HTTP 429 rate_limited: too many requests from this address | public calls are limited per address (a burst of 20, then 2 a second) | many viewers behind one address (an office) need server admission; behind your own proxy set NEARGLASS_API_TRUST_FORWARDED=1 so the real address is used |
HTTP 429 rate_limited: too many tokens for this seat this minute | one seat asked for more than 6 tokens in a minute | a page loop; the player already waits, a custom one must too |
HTTP 503 clock_unsynchronized: the server clock is not synchronized; no token is issued until it is | the API host's clock | fix NTP on the API host |
The stream is full. Waiting for a seat... | the stream's seats and line, or the account's seats, are full, or every relay is full | raise viewer_cap on a new stream, or the account's limit; the player retries by itself |
In line, number N..., then A seat is free. Join within 10 s. | a normal waiting room | press Join, or set auto-join |
Your seat was given up. Reload to watch again. (HTTP 410 seat_lost: this seat is gone: ask for a new one) | no heartbeat for 20 s (60 s from a hidden tab), a kick, or the stream was deleted | reload; check the page can reach {play_url}/hb (NEARGLASS_API_PUBLIC_URL, CORS, a content blocker) |
stuck on Connecting... | the relay is not reachable from the browser | UDP 443 open to the relay, certificate valid for its name, the relay running |
the label says Compatibility mode, about 1 s behind | the browser could not use MoQ (no WebTransport, UDP blocked, or no relay session within 3 s) and fell back to LL-HLS | expected on Safari and some networks; if it happens everywhere, look at the relay's certificate and port |
This browser cannot play this stream. | no MoQ support and no LL-HLS playlist | the deployment has no LL-HLS (NEARGLASS_API_HLS_URL) |
black, status live, publisher is on | the encoder sends no keyframes, or a codec the browser cannot decode | H.264, no B-frames, keyframe every 1 s |
| viewers get in, then drop after a few minutes | their token expired and renewal failed | the page must be able to reach /v1/play/{id}/token (or your token-url); check the API's log |
Tokens
The relay answers a bad token with one log line and the same code whatever the reason:
session authorization failed, closing code=2 reason="unauthorized"Go down this list on the relay and the API host.
| Cause | Check | Fix |
|---|---|---|
| signed with a key the relay's scope does not list | the scope file's keys against the issuer: GET /v1/issuer gives the PEM | put the public key in the scope; with the API, let it write the scope file |
| issuer or audience differ | scope issuers and audiences against NEARGLASS_API_ISSUER and NEARGLASS_API_AUDIENCE (nearglass-token, nearglass-relay), or nearglass token mint --issuer --audience | make them equal |
| the token expired | the token's lifetime against the scope's clock_skew_secs (2 for accounts, 60 in the example) | mint closer to use; the player renews by itself |
| the relay's clock is not synchronized | moq_relay_clock_synchronized is 0, or the log says system clock is not synchronized: token scopes refuse every token until it is | run NTP on the relay |
| the relay's clock is behind the issuer's | log: refusing a token issued in the future | fix the relay's time |
| the token is for another namespace | the stream's namespace against the token's | use the answer's namespace |
| a viewer or controller token used to publish | the relay closes the session (Closed(1)) | publisher role |
| a second session on one token | moq_relay_seat_refused_total{reason="token_id_in_use"} | one token carries one session; the player asks for a fresh one for each connection, so a copied token fails |
| the scope's session limit | moq_relay_seat_refused_total{reason="scope_full"} | raise the account's limits |
| no token at all | a bare URL, a path with no entry in the scope file | a token, and the account's scope path in the URL |
the API's own 401 unauthorized | missing bearer key, unknown key | the Authorization: Bearer sk_... header; the right account's key; a rotated key is gone for good |
403 operator_only | a secret key on an operator call (name and limits) | use the ok_ key |
A viewer's session outlives its token: a joined viewer is closed at the
token's exp plus the skew plus the watch grace (NEARGLASS_API_WATCH_GRACE,
600 s), and the player has by then renewed on a new session.
Self-hosting
| You see | Cause | Fix |
|---|---|---|
API exits: NEARGLASS_API_STATE_FILE is required | no state file | set it, in a directory the unit can write |
NEARGLASS_API_SECRET_KEY_SHA256 (the default tenant) or NEARGLASS_API_OPERATOR_KEY_SHA256 (tenants) is required | no key at all | nearglass-control-api keygen operator |
tenants (NEARGLASS_API_OPERATOR_KEY_SHA256) need NEARGLASS_API_TENANT_KEY_DIR and NEARGLASS_API_SCOPE_FILE | accounts need somewhere to keep keys and the scope | set both |
NEARGLASS_API_RELAYS (or NEARGLASS_API_FLEET_FILE) is required | no relay to hand out | set one |
...: SHA-256 hex digests (nearglass-control-api keygen prints them), not keys | a key was put where its hash goes | use the sha256: line |
NEARGLASS_API_RELAY_ADMIN_URL and NEARGLASS_API_RELAY_ADMIN_KEY_FILE go together | one of the pair | set both or neither |
NEARGLASS_API_PUBLISHABLE_KEYS: pk_ keys only (a secret key there would be public) | a secret key in the list | use nearglass-control-api keygen publishable |
NEARGLASS_API_OPERATOR_ALERT_URL is not read (the URL is a secret) | the alert URL in the environment | a root-only file named by NEARGLASS_API_OPERATOR_ALERT_URL_FILE |
API log: no relay admin port: a lost seat's open sessions run until the relay's watch grace ends | no admin port configured | set NEARGLASS_API_RELAY_ADMIN_URL and _KEY_FILE; run the relay with --admin-addr and --admin-key-file |
relay exits: admin port failed; exiting error=admin port 127.0.0.1:4480 | the address is in use, or not a loopback address, or the key file is unreadable or shorter than 32 characters | free the port; check the file |
| the relay logs a warning at start that a send buffer was capped | the sysctl file is not applied | sysctl --system (Self-hosting) |
a new account's relay URL is refused (unauthorized) | the relay has not read the new scope file yet, or reads another file | the relay's --scope-config is the API's NEARGLASS_API_SCOPE_FILE; a new scope is read within a second |
| usage stays at zero; no spend alert | no nearglass-meter, or it points at a different log than the relay's --session-log | the usage agent; look for shipped records=N |
| usage is missing for a period after the session log was emptied | a log truncated in place loses what was not shipped yet | rotate the log by renaming it, never by truncating |
agent heartbeat refused with unknown field | the agent is newer than the API | upgrade the API first |
POST /v1/fleet/heartbeat answers 404 no_fleet: this API has no fleet file | no NEARGLASS_API_FLEET_FILE | set it and restart the API |
400 from the heartbeat for a node | the node id is not in the fleet file | --node must be an id of the file |
every relay shows relay.down right after a deploy | the agents had not started yet | the API waits 30 s from its start or the agent's first beat; start agents before relays |
| a webhook never arrives | the URL is not a public address, or your receiver answered non-2xx three times | Webhooks |
| the relay does not start under systemd and cannot open its key | the unit's dynamic user cannot read a root-only key.pem | pass it as a credential (Self-hosting) |
Still stuck
Collect three things before asking: the API's response body (the code and
message), the relay's log around the refusal, and
GET /v1/status or GET /v1/streams/{id}/stats. They answer most
questions.