Use Nearglass with AI and vision models

Nearglass is the low-latency video path between machines, models and people. A camera publishes once; your model, your operators and your viewers all subscribe to the same stream. This page shows the Python side: frames into your model as numpy arrays, frames out of Python, and a data track that carries each result tied to the exact frame it came from, so a player or another program draws it on the right picture.

InstallCopy Anchor Link

With Rust and Python 3.9 or newer:

python3 -m venv .venv && . .venv/bin/activate
pip install "maturin>=1.8,<2"
cd python && maturin develop --release

It brings numpy and PyAV from PyPI. PyAV carries its own FFmpeg, which does the H.264 decoding and encoding.

Frames into your modelCopy Anchor Link

import nearglass as ng

sub = ng.Subscriber("https://relay.nearglass.com:4443/tn_a", "live/st_1", token=VIEWER_TOKEN)
for frame in sub:
    result = model(frame.image)      # H x W x 3 uint8, RGB
    print(frame.pts_us, frame.id, frame.skipped)

Each frame is the newest picture. While your model works on one frame, the stream keeps arriving and being decoded; when the model asks again it gets the latest picture, not the next one in a queue, so a model at 5 frames a second on a 60 frame a second stream stays live instead of falling behind. frame.skipped says how many pictures went by unseen.

Subscriber(..., size=(640, 360)) scales on the way out. read(timeout) takes one frame instead of iterating. The token is a viewer token, from your backend the same way a web page gets one (Mint viewer tokens on your server).

Results tied to frames: the data trackCopy Anchor Link

out = ng.DataPublisher("https://relay.nearglass.com:4443/tn_a", "live/st_1/ai", "detections", token=PUBLISHER_TOKEN)
for frame in sub:
    boxes = model(frame.image)
    out.send(frame, {"boxes": boxes, "label": "person"})

send takes any JSON value (numpy arrays and numbers are converted) or bytes, and the frame it is about. The result travels with that frame's timestamp and id, so whoever reads it knows exactly which picture it describes. Keep results small: boxes and labels, not images.

Namespace and tokens. Publish results under the stream's namespace plus a suffix: the stream live/st_1, its results live/st_1/ai. Then the stream's own tokens cover them: a viewer token for live/st_1 reads live/st_1/ai, and a publisher token for live/st_1 publishes it. If your tokens carry a token id (one live session per id), give the model its own publisher token.

Reading results in another program, paired with their frames:

watch = ng.Subscriber(RELAY, "live/st_1", token=VIEWER_TOKEN, history=240)
results = ng.DataSubscriber(RELAY, "live/st_1/ai", "detections", token=VIEWER_TOKEN)
for r in results:
    frame = watch.frame_at(r.pts_us)  # the exact picture the result is about
    if frame is not None:
        save(frame.image, r.value)

history=N keeps the last N decoded pictures for frame_at.

Results on the video in a web pageCopy Anchor Link

createDataTrack from @nearglass/player reads a data track beside a player and tells you, on every frame the player shows, which result goes with it:

import { createPlayer, createDataTrack } from '@nearglass/player';

const player = createPlayer({ relay, tokens, namespace: ['live', 'st_1'], track: 'video', canvas });
const data = createDataTrack({ relay, tokens, namespace: ['live', 'st_1', 'ai'], track: 'detections', player });
data.on('sync', ({ exact, latest, behindMs }) => {
  // exact: the result for the frame on screen, if it has come
  // latest: that, or the newest result before it, behindMs older than the picture
  drawBoxes(latest ? latest.value.boxes : []);
});

A model's result is made after its frame, so in a live player it usually arrives just after that frame was shown. Draw latest and use behindMs to show or hide results that are too old; exact is there when the result won the race, and data.at(captureMs) finds the result for any frame you still hold. Every result also arrives as a 'data' event.

The ai-vision sample (samples/ai-vision/) is all of this end to end: detect.py runs a placeholder motion model on a stream and publishes its boxes, and overlay.html draws them over the video.

Frames out of PythonCopy Anchor Link

pub = ng.Publisher(RELAY, "live/robot-7", width=1280, height=720, fps=30, bitrate=2_000_000,
                   token=PUBLISHER_TOKEN, data_tracks=["detections"])
while True:
    img = render()                         # H x W x 3 uint8, RGB
    pts_us, frame_id = pub.send(img)
    pub.send_data((pts_us, frame_id), {"boxes": boxes})

Frames are encoded with H.264 tuned for delay (no B-frames, no lookahead, headers on every keyframe) and sent through the same publisher the other SDKs use: it reconnects by itself and sends a keyframe when a viewer joins. encoder="h264_nvenc" (or another FFmpeg H.264 encoder) uses hardware. data_tracks publishes results beside the video in the same namespace.

What it costsCopy Anchor Link

Decoding runs on a thread of its own; each frame you take is converted to numpy once, and frames your model never sees are never converted.

Limits todayCopy Anchor Link

• Tokens are fixed when a Subscriber or DataSubscriber is made: a viewer session ends when its token expires (plus the relay's grace). Make a new one with a fresh token.
• Video only: no audio into Python yet.
• Data tracks are not listed in the stream's catalog: subscribers name them.
• Linux x86_64 is the platform it was built and tested on.