Audio
Which sound the ingest carries, what happens to AAC and Opus on the way, and what to expect for lip sync.
A stream carries one audio track next to its video. Viewers always receive Opus (48 kHz, stereo); what your encoder sends decides how it gets there.
What is carried
| Your encoder sends | Over | What happens |
|---|---|---|
| Opus, mono or stereo | SRT (Opus in MPEG-TS), RTMP (Enhanced RTMP v2) | passed to viewers packet by packet, not decoded |
| AAC-LC, mono or stereo, any sampling rate | SRT (ADTS in MPEG-TS), RTMP (FLV AAC and Enhanced RTMP) | decoded and encoded again as Opus at the ingest, resampled to 48 kHz if needed |
| HE-AAC (SBR), more than two channels, AAC in LATM, MP3, AC-3 | any | not carried: the stream plays without sound, and the ingest logs which format it refused |
| Anything | WHIP | not carried yet: the stream plays without sound |
| Anything | nearglass publish --file or --encoder mpp, the C library | not carried: they publish video only |
| AAC-LC or Opus | an encoder into nearglass publish --srt-listen or --rtmp-listen | carried, as for the ingest |
AAC-LC is what OBS, ffmpeg's aac encoder and most hardware encoders send
by default, so for most people the default works. Mono is sent on as stereo,
so a viewer always gets two channels. The Opus the ingest makes is 128
kbit/s.
Some encoders cannot send Opus over some protocols. ffmpeg's FLV output cannot carry Opus, so over RTMP from ffmpeg use AAC. Opus over RTMP is taken but no encoder has been tested sending it.
Settings
- Codec: AAC-LC or Opus. In OBS, leave the audio encoder on AAC (not HE-AAC), and set mono or stereo.
- Bitrate: 128 kbit/s is a good default.
- Sampling rate: 48 kHz, or 44.1 kHz. Both are fine.
- ffmpeg over SRT:
-pes_payload_size 0sends each audio packet as soon as it is encoded. Without it ffmpeg gathers a fraction of a second of sound before sending. See ffmpeg.
Announce the sound before the picture
A viewer reads the stream's track list once, when it starts. The ingest puts
the audio track in that list only if the encoder's audio format is known
before the first video frame goes out: from the stream table on SRT, from
the sequence start on RTMP. An encoder that starts its audio late gets a
stream without sound until it reconnects, and the ingest logs it
(audio: announced after the stream started publishing). Most encoders
send both from the start.
Lip sync
Sound and picture are stamped from the same clock, so they start in step, and the player plays the sound in step with the picture it shows. The player never makes video wait for sound. If sound and picture drift in the encoder, the ingest keeps the encoder's own alignment and does not correct it.
What to expect:
- Sound normally plays a little after the picture. The player holds sound back rather than let it lead.
- AAC sound plays slightly later than Opus, because converting it needs a whole AAC frame first. The difference is small. If you can choose, Opus over SRT is the tightest path.
- The ingest waits for nothing when it converts: a frame is decoded the moment it arrives, and the Opus packet leaves as soon as it is full.
What the viewer sees
The player shows its sound control only when the stream carries audio, and a browser plays sound only after the viewer has interacted with the page, so the viewer presses Sound on to hear it. The LL-HLS fallback carries the sound too.
When there is no sound
| Check | Fix |
|---|---|
| Does the player show a sound control? | If not, the stream has no audio track: see the table above and the ingest's log, which names a refused format |
| Is the encoder's audio HE-AAC, 5.1, MP3 or AC-3? | Change it to AAC-LC or Opus, mono or stereo |
Does the stream publish over WHIP, nearglass publish --file, the camera or the C library? | They carry video only for now: use SRT or RTMP for a stream with sound |
| Did the sound start only after a reconnect? | The encoder announced its audio after its first video frame: start audio and video together |