Controller seats

An interactive stream carries commands back to the device that publishes it: one driver at a time steers (a robot, an excavator, a camera head) while everyone else watches. Commands and video share the relays and the connection the device already has: there is no second channel to run.

The parts, and who runs them:

The teleop sample is all three, small.

1. The streamCopy Anchor Link

curl -s $API/v1/streams -H "Authorization: Bearer $SK" \
  -d '{"name":"excavator-1","namespace":"site-a/excavator-1","mode":"interactive","ingest":["sdk"],"viewer_cap":20}'

"ingest":["sdk"] because the device publishes itself, with a publisher token, rather than through an encoder. namespace is optional; set it when the device already has a name.

2. The deviceCopy Anchor Link

Your backend mints the device's publisher token (ttl up to 7 days; the device refreshes it before it expires through its token callback):

curl -s $API/v1/streams/$ID/publisher-tokens -H "Authorization: Bearer $SK" \
  -d '{"ttl":86400,"subject":"excavator-1"}'

The answer is a descriptor: token, relays, namespace (and relay_cert_sha256 for a relay with a self-signed certificate). The device publishes its encoder's output and opens a turn when your backend says a driver is up:

let (bc, mut events) = Broadcast::start(connector, BroadcastConfig::new(namespace, video), encoders);
bc.start_turn("t-42", Duration::from_secs(90)); // this driver, at most 90 s
while let Some(e) = events.recv().await {
    match e {
        BroadcastEvent::Control(control::Event::Command { payload, .. }) => apply(&payload),
        BroadcastEvent::Control(control::Event::Stop(_reason)) => all_outputs_safe(),
        _ => {}
    }
}

In C: nearglass_pub_start_turn(p, "t-42", 90) and a callback set with nearglass_pub_set_control_callback that gets NEARGLASS_EVENT_COMMAND and NEARGLASS_EVENT_STOP.

The device is the safety authority. Its gate accepts commands only for the turn it opened, ends the turn at its own deadline, and raises a stop when commands stop arriving (the driver let go, the tab was hidden, the network went), when the turn ends, and when the connection drops. On a stop, put every output in its safe state. Nothing the network or a page does can keep a machine moving past that.

How the device learns the next turn id is between it and your backend (a call it polls, a message you already have). The id is any printable string without / or spaces, at most 59 characters.

3. The queue, in your backendCopy Anchor Link

For each driver's turn:

# The turn starts: a token for this driver, this turn only
curl -s $API/v1/streams/$ID/controller-tokens -H "Authorization: Bearer $SK" \
  -d '{"turn_id":"t-42","driver_ref":"user-1234","ttl":95}'
# The turn ends early (the driver quit, you moved them on)
curl -s -X DELETE $API/v1/streams/$ID/controller-tokens/t-42 -H "Authorization: Bearer $SK"

The controller token lets its holder watch the stream and send commands on control/<namespace>/t-42, nothing else. Its answer adds turn_id and control_namespace to the usual descriptor. Make ttl the turn plus a few seconds: it bounds when the driver can connect. Ending a turn closes the driver's control session at the relay at once. Minting a turn id that already has a live token answers 409 turn_exists; use a new id per turn.

Watchers who are not driving take ordinary viewer seats.

4. The driver's pageCopy Anchor Link

The driver's page plays the video on a viewer seat, like any watcher, and sends commands with the turn's controller token. Two tokens, because the relay allows one session per token: the player and the controller are two sessions.

import { createPlayer, createController } from '@nearglass/player';

// Your backend: { watch: a viewer-tokens answer, turn: a controller-tokens answer }
const { watch, turn } = await (await fetch('/api/drive/excavator-1', { method: 'POST' })).json();
const ns = watch.namespace.split('/');
const player = createPlayer({
  relay: watch.relays[0].url, certHash: watch.relay_cert_sha256, namespace: ns, track: 'video',
  tokens: async () => [{ value: watch.token }], canvas: document.querySelector('canvas'),
});
const ctl = createController({
  relay: turn.relays[0].url, certHash: turn.relay_cert_sha256, namespace: ns, turn: turn.turn_id,
  tokens: async () => [{ value: turn.token }],
  state: () => readJoystick(),   // a Uint8Array: the whole state, read at every send (at most 1024 bytes)
  player,                        // no new frame for 500 ms stops driving: never drive blind
});
ctl.on('connected', () => {
  holdButton.onpointerdown = () => ctl.press();   // the dead-man: hold to drive
  holdButton.onpointerup = () => ctl.release();   // let go: one stop, then silence
});

The controller sends the whole state 50 times a second while held. It sends one stop, then nothing, when the tab is hidden, the window loses focus, release() is called, or the video stalls; after a stop it waits for a fresh press(). press() answers false and does nothing when the controller is not connected yet, the tab is hidden, or the player has shown no frame for 500 ms: never drive blind.

Options of createController, besides the ones above: rateHz (default 50), stallMs (default 500), watchFocus (default true: stop on a hidden tab or a window that loses focus) and now (below). Events, with ctl.on(name, fn): connected, driving (after a press() that took) and stopped (with the reason: released, tab hidden, window lost focus, video stalled or closed). ctl.active says whether it is driving, ctl.stats() counts what was sent, and ctl.close() ends the session.

The viewer seat needs a heartbeat. The example above plays on a seat from viewer-tokens, and a seat nobody beats is freed after 20 s and its relay session closed. Beat it as the web player does: POST {watch.play_url}/hb with {"seat": watch.seat} every watch.heartbeat_ms, and POST {watch.play_url}/leave when the driver is done. (play() does this for you, and its controller's engine property is the player to pass to createController.)

The controller does not reconnect. It makes one connection with the token it is given. If the connection drops, driving has ended (the device raises a stop when commands stop arriving); make a new controller, with a fresh token from your backend, if the driver's turn is still on. The same goes for the player's tokens callback in the example: it hands back the one token it was given, so a long turn needs a callback that fetches a fresh one (see Token renewal).

Clocks. Each command carries the driver's send time, and the device drops a command that is older than 150 ms when it arrives, by the device's own clock, and any beyond 60 a second (the controller's 50 fits). The controller stamps commands with now(), by default the driver's own clock (Date.now). A driver and a device are different machines, so give the controller a clock on the device's time: createClock({ url }) (@nearglass/player/clock.js) reads it from an HTTP endpoint you run beside the device, answering GET with {"now": <unix ms>} on the clock the device uses, and keeps the offset; pass now: clock.now to createController. Without it the two clocks have to agree to within a few tens of milliseconds, or the device refuses the commands as stale. Our API does not run a time endpoint for you.

Measured, and notCopy Anchor Link

In this repository's tests (one host), commands from the driver example reach the device example through a relay at 50 Hz, and the device raises a stop when the driver lets go; the C device (nearglass.h) does the same. The device's gate stops after 300 ms without a valid command by default (nearglass-pub). The browser controller with a real joystick and a real machine over the internet is not part of these docs' checks.