The promised technical tour — how an AI process taps a mediasoup SFU, turns Opus and VP8 packets into PCM and pixels, runs them through models, and streams synthesized speech back into a live call.
This is the second of two posts. The first covered whatopensam lets you build and why an owned baseline matters. This one is the how: the plumbing, the codecs, and the loop.
The tap handshake: how a plain Python process becomes a participant in a mediasoup call — and the exact moment RTP starts flowing.
What this pipeline actually does, in one paragraph
opensam‘s real-time media AI pipeline is the code path between a browser’s WebRTC stream and an AI model: it taps a mediasoup SFU over a PlainTransport, parses raw RTP by hand into Opus audio and VP8 video, decodes both to PCM and pixel frames, runs them through STT/LLM/TTS or vision models, and re-encodes the result as RTP packets sent back into the room. No framework sits between you and any of those steps — everything below is the literal path the bytes take.
Why this layer is worth understanding
Every real-time media-AI framework eventually does what’s described below — it just does it behind an abstraction. The reason to look at it directly is simple: the abstraction is where your unusual requirement goes to die. When you need a video frame at a different cadence, a codec nobody templated, or media injected back on a path the framework didn’t anticipate, you need to know what’s actually on the wire.
opensam is small enough to hold in your head. Here’s the whole path through this real-time media AI pipeline.
1. The tap: getting RTP out of an SFU
A browser’s media reaches the SFU over WebRTC — DTLS-encrypted, ICE-negotiated, the full stack. An AI process doesn’t want any of that. It wants raw packets.
mediasoup’s answer is PlainTransport: a transport that speaks plain, unencrypted RTP over UDP. The server creates one per bot:
js
const transport = await room.router.createPlainTransport({
listenIp: { ip: '127.0.0.1' },
rtcpMux: true, // RTP and RTCP share one port
comedia: false, // we will tell the SFU where to send
});
comedia is the subtle knob, and opensam uses it in both directions:
Receiving (comedia: false) — the bot binds its own UDP socket on port 0 (the OS picks a free port), reads the assigned port back, and hands it to the server over WebSocket signalling. The server calls transport.connect({ ip, port }) and now knows exactly where to fire packets.
Sending (comedia: true) — the reverse. The SFU learns the bot’s source address from the first packets it receives. That’s why the bot fires five 440 Hz tone frames on startup: they’re a warm-up whose only job is to teach mediasoup where the bot lives. Without them, the SFU has no return address.
Media doesn’t flow at connect() though. It flows when a consumer is created against a producer and resumed:
js
const consumer = await entry.transport.consume({
producerId, rtpCapabilities: room.router.rtpCapabilities, paused: true,
});
// ...later
await consumer.resume(); // <- the exact moment RTP starts hitting the bot's socket
Consumers are created reactively: when a participant publishes, the server emits new-audio-producer / new-video-producer over signalling and the bot spins up a consumer for it. People joining late Just Work. The consumer’s rtpParameters come back with the SSRC and payload type — and the bot records ssrc → peerId, which is how it later knows who is speaking.
Crucially, audio and video consumers share the same plain transport — so both land on one UDP socket, interleaved, distinguished only by payload type.
2. One socket, two media types: parsing RTP by hand
There’s no RTP library here. The bot reads a datagram and walks the bytes.
RTCP shares the port. With rtcpMux: true, receiver reports and PLIs arrive on the same socket. They’re filtered by checking whether byte 1 falls in the RTCP packet-type range (200–206) and discarded.
The header is variable-length. Real size is 12 + (CC × 4), and if the X bit is set, plus 4 + (extension_length × 4). Getting this wrong hands your decoder garbage.
The header extension carries a gift. WebRTC negotiates the one-byte extension profile 0xBEDE, and inside it, extension ID 6 is ssrc-audio-level — the browser’s own measurement of how loud that packet is, 0 = loud, 127 = silence. The bot parses it and skips packets above a threshold of 40 before they ever reach the Opus decoder. It’s a free VAD pre-filter computed by the sender.
Dispatch is then one branch: payload type 101 → VP8, anything else → Opus.
3. The audio path: Opus → PCM → 16 kHz mono
mediasoup delivers Opus as the browser encoded it: 48 kHz, stereo, 20 ms frames — 960 samples per channel.
That’s 960 samples × 2 channels × 2 bytes = 3840 bytes of PCM per packet. A decoder is created per SSRC — each speaker gets their own, because Opus is stateful.
That statefulness drives the error handling. A corrupt or missing packet doesn’t just lose 20 ms; it poisons the decoder. So on a decode exception the bot rebuilds the decoder, returns a frame of silence, and skips the next three packets to let the stream resynchronize. Crude, but it prevents one bad packet from producing seconds of noise.
Speech models want mono at 16 kHz, so the final step is a downmix and resample:
Going down in rate is the direction that actually demands care: without a low-pass filter first, everything above 8 kHz folds back into the audible band as aliasing — and you’d be feeding that to your STT. soxr filters properly, and it keeps one resampler per SSRC, mirroring the per-SSRC Opus decoders, so one speaker’s filter state never bleeds into another’s.
Out the other side: clean 16 kHz mono PCM, tagged with the SSRC that produced it — which resolves to a participant name.
4. The listening loop: VAD → STT → LLM → TTS
PCM lands on an asyncio.Queue, and a forwarding task turns a stream of packets into utterances.
Silero VAD is picky: exactly 512 samples at 16 kHz — 1024 bytes, 32 ms. So the loop accumulates PCM into a buffer and drains it in exact 1024-byte chunks, converting each to a normalized float32 tensor and running the model (in an executor, so the event loop keeps breathing).
Around that sits a small state machine with hysteresis:
2 consecutive speech chunks confirm onset (~64 ms) — short enough to feel instant, long enough to reject a cough.
20 consecutive silence chunks (~640 ms) end the utterance and trigger finalize(), flushing the buffered speech to STT.
Speech chunks stream to Deepgram live (Whisper is the offline fallback). Transcripts below 0.70 confidence are dropped. Survivors get a wake-word check, then PII scrubbing before any text reaches the LLM.
The LLM stage streams from Claude — and the interesting trick is that it doesn’t wait for the full reply. It splits the token stream on phrase boundaries (.!?;:) and pushes each finished phrase onto a TTS queue immediately. The bot starts speaking the first clause while the model is still writing the rest. That single decision is worth more perceived latency than any model swap.
Tool use loops the conversation at most twice: Claude may call summarise_meeting or extract_action_items, the bot builds the context from its transcript log, feeds the result back, and Claude streams its spoken reply.
Barge-in is blunt and effective: any confirmed speech while the bot is talking cancels every active TTS task and drains the queue. The human always wins.
Wrapped around all of it: OpenTelemetry spans per stage, a circuit breaker that trips to a canned fallback line when the LLM misbehaves, and a per-room token/cost tracker.
5. The video path: VP8 → frames → vision
When payload type is 101, the bytes take a different road.
After the RTP header comes the VP8 payload descriptor (RFC 7741) — a variable-length preamble that is not part of the codec bitstream and must be removed:
Byte 0: X N S R PPPP — X says an extension byte follows.
Extension byte: I L T K RRRR.
If I: PictureID is 1 or 2 bytes, decided by the top bit of the first byte.
If L: TL0PICIDX, 1 byte. If T or K: they share 1 byte.
Only what’s left is VP8.
Then reassembly. A video frame is far bigger than a UDP datagram, so it arrives in fragments. The bot accumulates payloads per SSRC and treats the marker bit as “frame complete”:
python
self.frame_buffers[ssrc].extend(payload)
if marker == 0:
return [] # not done yet
complete_frame = bytes(self.frame_buffers[ssrc]); self.frame_buffers[ssrc].clear()
Before decoding, a keyframe gate. PyAV cannot decode inter frames without a reference, and a bot that joins mid-stream arrives in the middle of a GOP. VP8 marks frame type in bit 0 of the first byte (0 = keyframe), so inter frames are dropped until the first keyframe appears. To avoid waiting for the browser’s natural keyframe interval, the bot asks for one: a request-keyframe signal triggers consumer.requestKeyFrame() on the server, which sends a PLI upstream and the browser produces a keyframe immediately.
Decode is PyAV: av.CodecContext.create("vp8", "r"), wrap the bytes in an av.Packet, get VideoFrames. On error, reset the codec and re-arm the keyframe gate.
Raw frames are then sampled to ~1 fps — you don’t need 30 fps to summarize a slide, and vision tokens cost money — JPEG-encoded at quality 85, and handed to a vision chain that tries Claude Sonnet, falls back to on-prem LLaVA, then Haiku. Nothing runs unless the sharing participant has ticked the AI-consent box.
6. The return path: PCM → Opus → RTP → the room
Injection is the mirror image, hand-rolled — the last leg of the real-time media AI pipeline, and the one most frameworks hide completely.
TTS audio is normalized once per utterance — RMS measured, gain applied toward a target dBFS, capped and clipped — then handed to the sender, which:
Resamples to 48 kHz stereo, honoring the rate the frame declares.
Buffers and drains in exact 3840-byte frames — Opus will not accept a partial frame.
Encodes with opuslib.Encoder(48000, 2, APPLICATION_VOIP).
Builds a 12-byte RTP header by hand: byte0 = 0x80 (V=2), PT = 120, sequence +1, timestamp +960, fixed SSRC.
sendto() the SFU.
Step 1 deserves more respect than it usually gets, because of an invariant that runs through this whole layer: every RTP packet carries exactly 20 ms. The timestamp advances by a fixed 960 ticks whether or not the audio inside it is correct. Nothing downstream will tell you the rate was wrong — the stream stays perfectly well-formed and simply plays at the wrong speed.
TTS rates vary in practice: Deepgram and Cartesia render at whatever TTS_SAMPLE_RATE asks for, while Kokoro ignores the request and always renders at its native 24 kHz. So the input rate is a fact to be read, never assumed:
Two details make this correct rather than merely plausible:
The resampler is stateful. A resampler is a filter, and a filter has memory. Audio arrives in ~1024-sample chunks, and one restarted at every chunk boundary produces a click at every seam — 20-odd times a second. soxr.ResampleStream carries its state across calls, so the chunks join as one continuous signal:
python
self._resampler = soxr.ResampleStream(sample_rate, SAMPLE_RATE, 1,
dtype="int16", quality="HQ")
mono = self._resampler.resample_chunk(mono) # state persists between calls
Interpolation, not repetition. Repeating each sample N times (a zero-order hold) is only coincidentally right when the ratio is an exact integer, and it’s a stair-step waveform with audible aliasing even then. A real resampler interpolates and handles non-integer ratios — which is the whole point, since the code shouldn’t care whether it’s handed 16 kHz, 24 kHz, or 48 kHz.
The one thing to understand about a proper resampler is that it holds about 20 ms inside its filter. That reads like lost audio until you measure it: stream 10 seconds and you’re 20 ms short; stream 30 seconds and you’re still 20 ms short. It’s constant latency, not accumulating drift — and those samples aren’t lost, they emerge as the next chunk pushes them through (the comfort-noise worker guarantees there’s always a next chunk).
The payoff is a timing invariant you can actually assert: feed five seconds of audio at 16 kHz, 24 kHz, or 48 kHz, and five seconds of speech comes out — every time.
The declaration has to be true
A resampler that honors a declared rate is only as good as the declaration, and this is where it gets interesting. The obvious design — “read the rate from config” — is wrong, because config describes what was requested, not what was rendered. Kokoro ignores the request entirely.
Worse, the provider isn’t fixed for the lifetime of the process. Cartesia owns a fallback chain — Cartesia → Deepgram → Kokoro — so a single failed API call means this sentence comes back at 24 kHz while the previous one was at 16 kHz. Config can’t express that; only the object that produced the bytes knows.
So each provider reports its own truth, and the chain updates it to match whoever actually served the utterance:
kk = KokoroTTS()
self.sample_rate = kk.sample_rate # the chain adopts the fallback's rate
The consumer then reads the rate from the provider rather than from config — and, importantly, reads it after the stream is drained, when it reflects who really answered:
This is the general shape of the lesson, and it outlives this codebase: in a media pipeline, metadata must travel with the payload. The moment a rate, a codec, or a channel count is inferred from configuration instead of read from the thing that produced the bytes, you’ve built a system that is correct only until the first fallback fires — and it will fail silently, because a wrong sample rate produces a perfectly well-formed stream.
The server was told what to expect via make_rtp_parameters(ssrc, payload_type=120) at create-bot-producer time — the bot’s declaration of its own codec, clock rate, channels, and SSRC. A comfort-noise worker keeps silent frames flowing when nobody’s talking, so the stream never goes stale.
A baseline earns trust by being honest about its shortcuts. These are deliberate trades for readability, and the first things to revisit if you push this real-time media AI pipeline past a POC:
Payload type 101 is hardcoded for VP8, and everything else is assumed to be Opus. The server does return the real payload type in rtpParameters — the bot just doesn’t read it. It works because this router’s two-codec config lands there; add a codec or enable RTX and audio/video misroute.
Video decodes inline on the audio loop. Audio reads go through an executor, but VP8 decode and JPEG encode run in the same coroutine — under heavy screen share, that’s audio jitter.
No jitter buffer, no sequence reordering. Frames are reassembled by concatenation and the marker bit. UDP reordering or loss yields a corrupt frame and a decoder reset.
A fixed SSRC of 12345678 identifies the bot’s outgoing stream. Fine for one bot; not fine for many in a room.
The bot’s own audio is never echo-cancelled. It works because the bot’s speech goes out on a different path than the audio it consumes — but put a real speaker and mic in the loop and you’ll want AEC.
None of these are mysteries. That’s the point: you can see every one of them, because the code is 400 lines you can read, not a framework you must trust.
POC now, production later
Disclaimer.opensam is intended for proof-of-concept and educational use. It is a reference baseline for understanding and prototyping real-time media AI — deliberately minimal and readable. It is not hardened for production: it does not guarantee the latency budgets, jitter resilience, packet-loss recovery, multi-tenant scale, or security posture that live products require.
Read this stack to learn exactly where the bytes go and what your product actually needs. When the POC earns the right to ship, the Samvyo media stack delivers these same capabilities — transport, raw media access, audio and vision pipelines, media injection — engineered for the latency, scale, resilience, and compliance production demands. Prototype this real-time media AI pipeline on the open baseline; productionize on the stack built for it.
Read the code
The whole media path is a handful of files on the development branch under v2ai/ai-bot/src/pipeline/:
File
What it owns
signalling.py
The tap handshake — transports, consumers, comedia, PLI
rtp_receiver.py
RTP parsing, RTCP filtering, audio-level extension, Opus decode, VP8 dispatch
VAD state machine, STT, streaming LLM, barge-in, TTS orchestration
rtp_sender.py
Opus encode, hand-built RTP headers, injection back into the SFU
bash
git clone git@github.com:Samvyo/opensam.git && cd opensam && git checkout development
Fork it, instrument it, break it. Once you can see the packets, nothing about a real-time media AI pipeline is magic anymore.
Frequently asked questions about opensam’s RTP-to-AI pipeline
What is a real-time media AI pipeline?
A real-time media AI pipeline is the code path that moves a live audio or video stream from a call into an AI model and back — parsing the transport protocol (RTP), decoding the codec (Opus for audio, VP8 for video) into raw PCM or pixel frames, running inference, then re-encoding and injecting the result back into the call, all with latency low enough to feel live.
What is mediasoup’s PlainTransport and why does opensam use it?
PlainTransport is a mediasoup transport that sends and receives plain, unencrypted RTP over UDP instead of the DTLS-encrypted, ICE-negotiated path a browser uses. opensam’s AI bot creates one PlainTransport per bot so it can read raw RTP packets directly, without implementing the full WebRTC handshake just to access media it will immediately decode.
What does comedia do in mediasoup, and how does opensam use it?
comedia controls whether a PlainTransport learns its remote address automatically. opensam sets comedia: false when receiving — the bot explicitly tells the SFU its UDP port via signalling — and effectively comedia: true behavior when sending, firing five silent warm-up tone frames so the SFU learns the bot’s source address from the first packets it sees.
How does opensam decode Opus RTP audio into PCM in Python?
opensam creates one opuslib.Decoder(48000, 2) per SSRC (per speaker, since Opus is stateful), decodes each 20 ms packet into 3840 bytes of 48 kHz stereo PCM, downmixes to mono, then resamples to 16 kHz with soxr — a proper filtering resampler, since a naive downsample would alias frequencies above 8 kHz into the audible band before the audio reaches speech-to-text.
How does opensam decode VP8 video from RTP packets?
opensam strips the VP8 payload descriptor defined in RFC 7741, reassembles fragmented RTP payloads into a complete frame using the RTP marker bit, drops inter frames until a keyframe arrives (requesting one via PLI if needed), and decodes the result with PyAV’s av.CodecContext.create("vp8", "r") into raw video frames.
Why does opensam parse RTP by hand instead of using an RTP library?
Parsing RTP directly — reading the 12-byte header, handling the variable-length extension, filtering RTCP by packet type — keeps the entire real-time media AI pipeline visible and debuggable in a few hundred lines of Python, rather than hidden inside a general-purpose library’s abstractions. It also lets opensam extract non-standard data, like the browser’s ssrc-audio-level header extension, as a free VAD pre-filter.
How does opensam send AI-generated speech back into a mediasoup call?
Text-to-speech audio is resampled to 48 kHz stereo with a stateful soxr.ResampleStream, buffered into exact 3840-byte Opus frames, encoded with opuslib.Encoder(48000, 2, APPLICATION_VOIP), wrapped in a hand-built 12-byte RTP header with an incrementing sequence number and timestamp, and sent over UDP to the SFU via sendto().
Why does the TTS sample rate matter so much in a real-time media AI pipeline?
Every RTP packet in this pipeline is assumed to carry exactly 20 ms of audio; the timestamp advances by a fixed amount regardless of whether the sample rate is correct. Because TTS providers can render at different native rates (and fail over to one another mid-stream), opensam reads the sample rate from whichever provider actually produced the audio rather than from static configuration — otherwise the stream stays well-formed but plays back at the wrong speed, silently.
What are opensam’s known limitations in this pipeline?
opensam hardcodes payload type 101 for VP8, decodes video inline on the audio coroutine (risking jitter under heavy screen share), has no jitter buffer or RTP sequence reordering, uses a fixed outgoing SSRC (fine for one bot, not many), and does not echo-cancel its own audio. These are documented, deliberate trade-offs for a proof-of-concept baseline, not hidden defects.
Where is opensam’s RTP pipeline code?
The pipeline lives under v2ai/ai-bot/src/pipeline/ on the development branch at github.com/Samvyo/opensam, split across signalling.py (the tap handshake), rtp_receiver.py (RTP parsing and Opus decode), vp8_decoder.py (VP8 reassembly and decode), bot.py (the VAD/STT/LLM/TTS loop), and rtp_sender.py (encoding and injection back into the SFU).
How the opensam open-source stack gives developers a from-first-principles alternative to frameworks like Pipecat — and the raw building blocks to prototype real-time media AI for voice and video use cases nobody else has templated.
Extract, process, inject — the whole idea in one picture. opensam solves the hard, undifferentiated plumbing — pulling raw media off a live call and pushing results back in. What model runs in the middle is your product.
This is the first of two posts. Here we look at what opensam lets you build and why owning this baseline matters. A follow-up will go deep on the how — the RTP plumbing, codec handling, and pipeline internals.
What is opensam, in one paragraph?
opensam is a dependency-light, open-source baseline for real-time media AI — the plumbing that pulls raw audio and video off a live call, hands it to any AI model in plain Python, and pushes the result back in. Unlike audio-first frameworks such as Pipecat or LiveKit Agents, opensam treats video as a first-class citizen and exposes the transport layer directly, so you can build voice agents, vision pipelines, or both on the same foundation — on-prem if you need to.
TL;DR
Most real-time voice-AI frameworks hand you a fast on-ramp and a set of opinions. That’s a great trade when your product looks like the demo. It’s a frustrating one the moment you need something the framework never imagined — processing video frames, injecting transformed media back into a live call, running on-prem models for compliance, or wiring AI into a dashcam feed instead of a support call.
opensam takes the opposite stance. It’s a dependency-light, transport-level baseline that shows you the entire path — from a browser’s microphone and camera, through a media server, into a plain Python process where any AI model can run, and back out to participants. You own every layer. Nothing is hidden behind a framework abstraction.
That makes it ideal for one job in particular: standing up a credible real-time media AI proof of concept, fast, for a use case that doesn’t fit the standard voice-agent mold.
Why a baseline you own beats a real-time media AI framework you rent
Frameworks like Pipecat, LiveKit Agents, or Vapi are genuinely good. They compress weeks of glue code into an afternoon. But they come with structural assumptions:
They’re audio-first. The mental model is speech in → LLM → speech out. Video is, at best, a bolt-on. If your product is fundamentally about pixels — telematics, surveillance, avatars — you’re fighting the grain.
They own the loop. The pipeline runner, the interruption logic, the turn-taking — that’s the framework’s territory. When you need to change how a frame is scheduled or how media is muxed, you’re patching someone else’s engine.
They abstract the transport. You rarely touch the actual RTP. That’s convenient until you need to, and then there’s a wall between you and the bytes.
Provider and deployment opinions. Swapping a model, running fully on-prem for a regulated customer, or metering per-room cost often means working around the framework.
opensam inverts all four. It is not a framework you extend; it’s a reference implementation you read, fork, and reshape. The AI loop is a few hundred lines of plain asyncio you can see end to end. The media path is explicit. Every provider is swappable because nothing depends on a provider-specific base class. And the whole thing is small enough to fully understand in an afternoon — which is exactly what you want under a proof of concept you’ll have to defend to a customer.
The point isn’t “frameworks bad.” It’s that a baseline — a clean, minimal, fully-owned starting point — is a different and complementary tool. When the requirement is novel, you want to start from bedrock, not from someone else’s opinions.
The stack, functionally
opensam ships three composable layers. You can start at whichever one matches your latency, scale, and control needs.
Layer
What it is
When you reach for it
full-mesh
Pure peer-to-peer WebRTC calling (React + Node signalling + TURN).
Smallest footprint, 2–4 participants, no media server to run.
mediasoup
The same experience on a mediasoup SFU — clients upload once, the server fans out. Redis, Prometheus, Grafana included.
Larger rooms, scale, and — crucially — a server-side tap point for media.
v2ai
An AI participant that joins a mediasoup room, pulls raw audio and video off the wire, runs it through AI, and speaks/acts back.
Anywhere you want a model inside a live call.
The magic layer is v2ai, and the reason is a single architectural idea worth stating plainly.
The one pattern that makes real-time media AI general: extract → process → inject
Every real-time media AI product, no matter how exotic, is a variation on the same loop:
Extract raw media from a live stream.
Process it with a model.
Inject a result back — as audio, as an event, or as transformed media.
opensam implements this loop against a real media server rather than a toy file. On the extract side, the AI bot asks the SFU to copy a participant’s RTP to a plain transport, then receives:
Audio — Opus RTP, decoded to clean 16 kHz PCM, ready for any speech model.
Video — VP8 RTP, reassembled and decoded to raw frames, ready for any vision model.
On the process side, the repo demonstrates a full audio brain — voice-activity detection, speech-to-text, an LLM with tool use, and text-to-speech — plus a vision path that samples frames, encodes them, and sends them to a vision model.
On the inject side, it produces synthesized audio back into the live call as a normal participant would.
Here’s why that matters: once you can pull raw frames out and push media back in, you are no longer limited to “voice assistant.” Swap the model in the middle and the exact same skeleton becomes a driver-monitoring system, a security analyst, or a live video filter. The transport, the decode, the timing, the return path — the genuinely hard, undifferentiated plumbing of real-time media AI — is already solved and visible.
What’s in the toolbox
From a functional standpoint, a developer picking up opensam inherits:
Media transport, both topologies. Mesh for the simplest path; SFU for scale and a clean server-side media tap.
Raw media access. Decoded audio PCM and decoded video frames handed to you in Python — the thing frameworks hide and telematics/vision products desperately need.
A working audio brain. Wake-word gating, Silero VAD, streaming STT (Deepgram with a Whisper fallback), an LLM layer with streaming and tool calls, and multi-provider TTS (Cartesia / Deepgram / a local Kokoro backup). Barge-in / interruption is handled — the bot stops talking when a human starts.
A working vision path. Frame sampling (throttle the firehose to the rate your model can afford), JPEG encoding, and a vision layer that falls back across Claude and an on-prem LLaVA.
Media output. A proven path for putting AI-generated audio back into the room — the template you mirror for any “inject” step.
The unglamorous production-adjacent bits. Consent gating before any AI touches a participant’s video, PII scrubbing before text hits an LLM, OpenTelemetry tracing, Prometheus/Grafana dashboards, per-room cost tracking, and circuit breakers around flaky upstreams.
These are the things that make a proof of concept demoable to a real customer rather than a notebook. Everything is swappable because nothing is welded to a framework. Don’t want Deepgram? The STT boundary is one class. Need a local LLM for a data-residency clause? The LLM call is one function. That freedom is the whole point of a baseline.
Real-time media AI example use cases
The following illustrate how the same extract → process → inject loop retargets to very different products. They show where the repo already does the heavy lifting and where you’d plug in your own model.
1. Video telematics & driver safety
The idea: a camera in a vehicle (or a stream from an edge device) becomes a live feed that an AI watches continuously — flagging drowsiness, distraction, phone use, tailgating, or road hazards, and speaking warnings back to the driver in real time.
What opensam already gives you: the driver’s video arrives as decoded frames; the frame sampler throttles them to a sane rate (you don’t need 30 fps to catch a microsleep); the consent and cost machinery is in place; and the return path can speak an alert — “Eyes on the road” — through the same audio producer the meeting bot uses.
What you add: your detection model in the process step (a drowsiness/attention classifier instead of a slide summarizer). The event → voice-alert wiring is already the pattern; you’re changing what triggers the alert, not how it’s delivered.
2. Physical safety & security monitoring
The idea: a live camera feed — a site entrance, a factory floor, a retail aisle — is analyzed in real time for intrusion, missing PPE, weapons, loitering, or crowd density, raising events the moment something is seen rather than after the fact.
What opensam already gives you: continuous frame extraction from a live RTC stream, sampling to control model spend, a multi-provider vision path with an on-prem LLaVA fallback (critical when footage can’t leave the building), consent gating for privacy compliance, PII scrubbing for anything textual, and observability to prove the pipeline is alive.
What you add: the security-specific detection model and your alerting sink — a dashboard event, a webhook, an SMS. Because the vision boundary is explicit, running everything air-gapped on local models is a configuration choice, not a rewrite.
3. Face swap & live avatars in streaming
The idea: transform a participant’s video in real time — swap a face, apply a stylized avatar, blur a background, anonymize a whistleblower — and stream the modified video back into the call.
Why it’s the purest demonstration of the pattern: this is extract → transform → inject in its most literal form. opensam already implements the first two-thirds against real media: it pulls VP8 off the SFU and decodes it to raw frames you can hand to a face-swap or restyle model. The audio path then shows you exactly how to produce media back into mediasoup as a participant.
What you add: the transform model, plus the video-return leg — encoding processed frames and publishing them the way the repo already publishes audio. In other words, opensam hands you the ingestion, decode, and a working blueprint for the injection side; you supply the model and mirror the producer pattern for video.
That’s a real proof of concept on a foundation that would otherwise take weeks to build.
On latency: live face swap is the most demanding of the three — every frame is on the critical path, not sampled. The opensam baseline is where you prove the concept and the pipeline shape; hitting broadcast-grade frame budgets is a hardening exercise (see the disclaimer below).
What you’re expected to build on top
Being honest about the seams is part of what makes a real-time media AI baseline trustworthy:
The “inject transformed video” leg is a blueprint, not a finished feature. Audio-return is implemented; video-return you build by following the same producer pattern.
The AI models are examples, not the product. The value is the scaffolding around them. Your differentiation is the model you drop into the process step.
Robustness is proof-of-concept-grade. Packet reordering, jitter buffering, multi-speaker scaling, and tight frame-timing are deliberately simple so the code stays readable.
None of these are surprises once you see the shape of the code — and each is exactly the kind of thing you’d expect to own when you choose a baseline over a framework.
From proof of concept to production
Disclaimer.opensam is intended for proof-of-concept and educational use. It is a reference baseline for understanding and prototyping real-time media AI — deliberately minimal, readable, and un-opinionated. It is not hardened for production traffic: it does not, out of the box, guarantee the latency budgets, jitter resilience, multi-tenant scale, security posture, or reliability that live products require.
When a proof of concept earns the right to become a product, you don’t want to spend the next two quarters rebuilding the plumbing into something production-grade. That’s precisely the gap the Samvyo media stack fills: the same capabilities — real-time transport, raw media access, audio and vision AI pipelines, media injection — engineered for the latency, scale, resilience, and compliance that production demands. opensam lets you prove what to build and that it works; Samvyo is the credible path to shipping it.
Think of it as a deliberate two-step: prototype real-time media AI on the open baseline, productionize on the stack built for it.
Getting started
Everything lives on the development branch, each project in its own folder with a dedicated README:
git clone git@github.com:Samvyo/opensam.git
cd opensam
git checkout development
Start with full-mesh to see real-time media with the least ceremony.
Move to mediasoup when you need scale and a server-side media tap.
Open v2ai to watch an AI join a live call, see the world, and speak back — then swap the model in the middle for whatever your product needs to see.
Fork it, read it, break it, and make it yours. That’s what a baseline is for.
Next post: a byte-level tour of how opensam extracts RTP audio and VP8 video from a mediasoup transport and runs it through the AI pipelines.
Frequently asked questions about real-time media AI and opensam
What is opensam?
opensam is an open-source, dependency-light baseline for real-time media AI. It extracts raw audio and video from a live call through a mediasoup media server, hands it to any AI model in plain Python, and injects the result back into the call — giving developers a transport layer they fully own instead of a framework abstraction.
How is opensam different from Pipecat or LiveKit Agents?
Pipecat and LiveKit Agents are audio-first frameworks that own the pipeline loop and abstract away the RTP transport. opensam is transport-level and treats video as a first-class citizen: it exposes the raw extract → process → inject loop directly, so developers can build both voice and video real-time media AI without fighting a framework built for speech-only use cases.
Can opensam process live video, not just audio?
Yes. opensam decodes VP8 RTP from the SFU into raw video frames in Python, samples them at a configurable rate, and feeds them to any vision model — the same pattern it uses for audio. This makes it suited to video-first real-time media AI use cases like driver monitoring, security analytics, and live face swap.
What media server does opensam use?
opensam runs on mediasoup, an open-source SFU (Selective Forwarding Unit). Clients upload media once, the SFU fans it out to participants, and an AI bot can tap a participant’s RTP stream server-side — the mechanism that makes real-time media AI possible without a browser-side plugin.
Is opensam production-ready?
No. opensam is explicitly a proof-of-concept and educational baseline. It does not guarantee production latency budgets, jitter resilience, multi-tenant scale, or security hardening out of the box. Teams that need to productionize a real-time media AI pilot typically move from opensam to a hardened stack like Samvyo, which implements the same extract-process-inject pattern with production-grade reliability.
What AI providers does opensam support by default?
The reference implementation wires up Deepgram (with a Whisper fallback) for speech-to-text, Cartesia, Deepgram, and a local Kokoro backup for text-to-speech, an LLM layer with streaming and tool calls, and a vision path that falls back across Claude and an on-prem LLaVA model. Every provider boundary is a single swappable class.
What can I build with opensam besides a voice assistant?
Because opensam exposes raw decoded audio and video rather than hiding it behind a voice-agent abstraction, the same extract → process → inject skeleton supports driver-safety telematics, physical security and PPE monitoring, live face swap and avatar streaming, and any other real-time media AI product built around a live camera or microphone feed.
Does opensam support on-premise or air-gapped deployment?
Yes — this is one of its core design goals. Because no component is welded to a specific provider’s base class, swapping a cloud STT, LLM, or vision model for a local equivalent (e.g., on-prem LLaVA) is a configuration change, not a rewrite, which matters for regulated industries and any environment where footage or audio cannot leave the building.
How do I get started with opensam?
Clone the repository (git clone git@github.com:Samvyo/opensam.git), check out the development branch, and work through the three layers in order: full-mesh for the simplest peer-to-peer demo, mediasoup for an SFU with a server-side media tap, then v2ai to see an AI bot join a live call and process real-time media AI end to end.
Who built opensam and why?
opensam is an open-source project from the team behind Samvyo, an enterprise real-time communications platform. opensam exists so developers can prototype novel real-time media AI use cases on a baseline they fully understand, before deciding whether to productionize on a hardened stack.
Autoscaling WebRTC apps are not at all easy. A lot of discussion on building large-scale WebRTC apps gets stuck on how to scale. There are no straightforward answers available to this question yet. For this reason, we at CentEdge have developed CWLB, a general-purpose WebRTC load balancer using mediasoup as the media server at its core.
When a customer connects with us to help them build WebRTC apps, the conversation goes something similar to this.
Customer: We want to integrate video conferencing capabilities in our existing web app.
Us : Sure. We can help you with that.
Customer: Our requirement is to have 15 person(max) conferencing rooms with recording capabilities.
Us: Sure. We can help you with that as well.
Customer: We want the solution to be super scalable so that even a million rooms can be started at the same time. We have our own data centre and you can run Kubernetes clusters there. We hope this will be fine for scaling requirements.
Us: No. Kubernetes clusters may not be sufficient to scale WebRTC apps considering the stateful nature of them. Also memory and cpu usage may not be the right indicators for indicating server load in this case.
Customer: What is stateful nature? Why memory and cpu usage are not the right indicators?
Us:….
The conversation goes on where we make our customers fully understand the nature of WebRTC calls and the media server parameters which indicate correctly the current load. Towards the end of the conversation, this question of how should the scaling problem be solved, used to remain open for further discussions as we did not have a ready-made answer for this.
After going through a similar conversation several times, we decided to do something about it. 1st June 2021 is when we started working on a general-purpose load balancer to auto-scale WebRTC apps. After a year of considerable effort, we have successfully developed the load balancer to auto-scale WebRTC apps. We call it CWLB, which stands for Centedge WebRTC Load Balancer. CWLB supports both horizontal as well as vertical autoscaling. Mediasoup is the media server used behind the load balancer to scale WebRTC apps and it currently supports AWS as the cloud provider to create/delete on-demand mediasoup media servers.
Before moving on to discuss more CWLB, we will elaborate on some keywords which we mentioned in the above para for a better understanding of the context.
Why Autoscaling?
The first important question is, why does one need autoscaling? Because one needs more video rooms simultaneously which is beyond the capabilities of a single media server. Let’s look at an example.
A c5.2xLarge instance of aws (8vCPU & 16Gb RAM) can handle either one large 50-person conference room or 10 small 5-person conference rooms. Once the server is on full load, it can’t cater to any new room creation requests until the rooms running in it are closed. One option is to run multiple simultaneous servers to handle more load irrespective of whether new room requests are coming or not. In this case, it will be huge wastage of resources as one has to pay the server bills while the servers are idle most of the time.
There may also be instances where servers may be required only at a specific time but not all the time. In this case, one needs to manually create new servers just before they are needed and create a mechanism to route new room creation requests to the newly created servers. Once the need is over, again the servers are needed to be shut down and closed manually. This is still okay if the demand for video room creation is predictable as one will get the time for the creation of new servers but it is nearly impossible if the room creation time s highly unpredictable. An example of a predictable load is a church prayer service that happens every day at the same time or a scheduled board meeting that happens every week / every month on a specific date and time. These kinds of services give one ample time to create new servers to cater to these prescheduled demands. An example of an unpredictable service is a teaching-learning app where any teacher can log in at any time to start a room. In this case, the room creation requests are so random that one won’t get any time to create new servers. Therefore it is impossible to scale manually in case of an unpredictable load.
To solve the above-mentioned problems, a load balancer is used in front of the media servers whose job is to distribute the incoming load among the available servers based on a predefined algorithm. If no more media servers are available, then create new ones. If some of the media servers are idle, then delete them so that valuable resources can be saved. A load balancer is a must to cater to unpredictable load scenarios. Also, it is good to have for predictable load scenarios because it saves a lot of manual effort while minimizing the chance of error happening from the manual effort.
Is Autoscaling mandatory?
No. It is not mandatory for all kinds of WebRTC applications. When an application has a finite amount of load and also the load is predictable, then autoscaling may not be needed in this case.
Example:
If you have a small school with 100 students in 5 grades which makes 20 students (approx.) in each grade. In this case, an 8vCPU X 16GB RAM server running for 24 X 7 should be economical as well as sufficient enough to handle the peak load of all 5 grades running their classrooms simultaneously. For this kind of use case, adding a load balancer will add a lot of complexity and cost rather than saving it.
Why mediasoup?
Because mediasoup is one of the most capable media servers available out there today with high-performance metrics. It has many cutting edge features like
Simulcast & SVC
Congestion control
Multi-stream (ability to send multiple streams over a single peer connection)
Sender & Receiver side bandwidth estimation
A tiny Nodejs module for easy integration with existing large Nodejs applications
super low-level APIs to provide minute control over media stream flows
Features like ice restarts and prioritization provide application flexibility
We have used the majority of the capabilities provided by mediasoup in our load balancer to provide enough flexibility to our customers who will be using our load balancer to build their super scalable applications on top of it.
Why aws?
Because aws is the leading cloud provider today it is used by many enterprises, and startups as well as individuals for hobby projects. It also has best-in-class uptime and trust among its users. It has very elaborated and easy-to-follow documentation for developer adoption. Also, their critical APIs which are used by our load balancer to scale media servers, are stable with less change frequency. For all of the above reasons, we choose aws as our first cloud provider for CWLB. We will eventually plan to support all leading cloud providers including Microsoft Azure, Google Cloud, Oracle Cloud, Digital Ocean, OVH cloud, etc., once our aws offering is complete and stable.
There can be 4 possible strategies using which one can auto-scale a webrtc application.
Horizontal scaling
Vertical scaling
Hybrid scaling
Hybrid+ scaling
Horizontal Scaling
This is the suitable mode of scaling if your use case needs smaller meeting room sizes of 2-5 users in each but a lot of such rooms are needed simultaneously.
A good example will be of a video contact center where 100+ customer support agents attend daily calls from customers. It is primarily an one to one call between the agent and the customer until the agent’s supervisor and /or manager decide to join the call. In this case, there will be a maximum of 4 users in the conferencing room at any point in time but there will be 100+ / 500+ such rooms running at any point of time.
In this case horizontal load balancer can be used to distribute the load from first media server to second media server as soon as the load on the first server reached it’s peak. The load balancer would keep track of the real time usage and release resources whenever the load on first sever is reduced. This way the load balancer can upscale / downscale media server resources based on the real time load.
Vertical Scaling
This is the suitable mode of scaling if your use case needs larger meeting room sizes of 20 – 60 users in each but a smaller number of such rooms are needed simultaneously.
A good example will be of a school / educational institution where only 10 teachers conduct daily sessions for their respective classes. In this case, though relatively there will be more number of students in each of the sessions but a maximum of 10 such rooms for 10 teachers need to be run at any point in time.
In this case a vertical load balancer can be used to distribute the load from the first core of the media server to other available cores as soon as the load on the first core reached it’s peak. In this case, though only one media server maybe sufficient to cater to the whole school but effectively distributing load between all the available cores of the media server will be key to achieve the desired output from the media server. Here the load-balancer’s job would be to keep track of the real time usage and release resources whenever the load on each individual core of the media server is reduced.
The two load balancing strategies mentioned here are the two basic forms of media server load balancing in WebRTC. The other two approaches are advanced uses cases which needs more advanced load balancing with fine grain control. They are described in the second part of this blog series. the link to the 2nd part of the post is here.
CWLB
Introducing CWLB (Centedge WebRTC Load Balancer), a general-purpose WebRTC load balancer designed using mediasoup as the media server. It has been designed from scratch to cater to the demands of those enterprises who don’t want to use a video API vendor for certain reasons but want to use a dependable managed video infra with a dedicated support team, along with the possibility of customization of even the core media flows.
Features
Mediasoup as the media server
AWS/DigitalOcean as the cloud provider
Hybrid+ scaling
Highly flexible yet resource-efficient
An advanced load distribution algorithm with 85% efficiency (approx.)
Note: Currently with the CWLB v2 release, the efficiency of CWLB is 85%(approx.). Our goal is to reach >90% efficiency by the v3 release of CWLB.
Now we also have a production grade scalable in-house video conferencing solution named Meetnow on top of CWLB. It has been designed to truly unify your organization’s external and internal communication in the today’s remote first world. Some of the unique features are 2- 100 user room with different modes of one to one, conferencing and event, Complete meeting and attendance analytics, and last but not the least, pay only for real usage without any monthly / yearly commitments until you are sure about switching on to our Enterprise plan.
If you have mediasoup based open source project like mediasoup demo or edumeet which currently works great but does not autoscale then this is for you. If you have a BBB(bigbluebutton) / jitsi implementation currently in production which does not autoscale then this is for you. If you have any other open-source/custom-built video implementation in production which doesn’t autoscale, then this is for you. Even if your current production video setup is working fine but you may need something like this in near future Or you are just curious to know more about CWLB, feel free to drop us a note at hello@centedge.io / sp@centedge.io to know more about how we can help you. If you wish to schedule a free 30 mins discussion for your use case with one of our senior/principal consultants, feel free to do so using this link.
As the first post of 2023, Wishing everybody a wonderful new year 2023.
If you are here, you most probably are facing issues related to scaling with your WebRTC application or you are just exploring with some future plans to build a production grade WebRTC app. In both the cases, you are at the right place. This post is going to be a continuation from the previous post we wrote on this topic a couple of months ago. The previous post described in details regarding when auto-scaling is necessary and when it is not. If you are not sure if your solution needs WebRTC autoscaling or not, you should read the previous post here before reading further.
In the last post we discussed about horizontal OR vertical scaling as a strategic option to scale mediasoup media servers based on the use case. In this post, we are going to discuss about another way of auto-scaling and its use case. We also are going to discuss interesting new enhancements to CWLB.
The third WebRTC scaling strategy
The third approach is a combination of vertical and horizontal scaling combined in as one. It can be called a hybrid scaling approach. Here the vertical scaling approach is used to scale one room to all available cores in a mediasoup instance in case of a need.Once this mediasoup instance is totally occupied, but still the same room needs more resources, the horizontal scaling is used to scale to different mediasoup instance located in separate host. For all the new resource allocation requests for the same room, the new server is then used according to the vertical scaling strategy until and unless the first server has free resources to spare. This hybrid approach is typically useful for very large rooms like large event rooms where the load-balancer needs to cater to 100s / 1000s of concurrent users in one room in a complete just in time resource request mode.
Lets understand the 2 important key words mentioned in the above paragraph.
Resource requests: A request is made to the media server to allocate some resources to the user so that the user can send / receive audio / video / screen-share media streams.
Just in time request: This load-balancer strategy is used when the load-balancer has no previous information about the size of the rooms so that it can pre-allocate and reserve the resources.Here the load-balancer has to work really hard to keep track of real time resource usage of each media server and allocate / free resources in real time as the user joins / leaves a room. This type of implementation is relative complex to a pre-allocation and reservation based load-balancing strategy.
The Hybrid+ WebRTC scaling strategy
The hybrid+ scaling strategy has all the things that is there in the hybrid scaling strategy. In addition, it also has some other important aspects which makes this strategy a really good choice for medium / large scale deployments.
An additional relay server between the client and the media server to make a media server completely stateless i.e. the media server will not contain any kind of business logic.
Capable of creating / destroying on demand media servers using APIs of cloud providers in a completely automated manner with least manual intervention
Capable of utilizing advanced techniques like media server cascading for keeping the latency to the minimum while catering to a global user base. Media servers in different geographic locations need to run simultaneously to enable media server cascading.
Capable of HA(High Availability) setup where stand by media servers can take up load when primary media servers fail while in use. Additional standby media servers need to run to ensure HA.
CWLB 2.0
CWLB 1.0 which was released in June 2022 had vertical scaling, horizontal scaling which used AWS EC2 instances for auto-scaling of media servers. This was good enough for small and medium use cases. But for large and very large use cases like large scale event, it had 2 disadvantages. The first is that the load-balancer used to take more media server resources than the number of media servers ideally it should be consuming and the second is that the data transfer costs each room was incurring while using AWS ec2 instances.
In CWLB 2.0 , we have now addressed these 2 points along with many other improvements.
First, the core load-balancer algorithm is now fully JIT request compatible. It means it now uses media server resources very efficiently by keeping track of each media server usage in the real time and allocate / de-allocate resources based on real time user resource consumption demands. It now has all strategies enabled i.e. vertical scaling, horizontal scaling, and a mix of both aka hybrid scaling.
Second, we have integrated another cloud provider, DigitalOcean into the load-balancer which has relatively less data transfer costs than AWS ec2. Lets take an edtech use case as an example to compare the data transfer costs between AWS ec2 and DigitalOcean for your reference so that you can understand why this is important.
Example
A maths tutoring company in India runs online maths tutoring classes for high school students. Here each maths teacher teaches high school maths to1000 students in one online session. They conduct 6 such sessions every day for 6 days a week with each session being conducted for 90mins. Lets try to calculate an approximate data transfer cost for a month. Here we will be using some assumptions to look more realistic.
Lets calculate the amount of data being transferred from the media servers in the cloud to students who have joined the class.
The teacher is speaking while either sharing his/her camera / screen for whole of the class time i.e. 90mins.
Lets assume that the audio is consuming 40Kb/second and the video / screen share is consuming 500Kb/second of internet bandwidth. So each student is consuming 540Kb/second of data.
Here is how the maths looks like.
540 * 60 * 90 = 2.78Gb is what one student consumes for whole 90mins session.
It there are 1000 students in that session, the total data consumption for that session would be 2780.9 Gb or 2.71 Tb.
If there are 6 such sessions happen each day, then the data transfer amount for each day would be 16.29 Tb.
Considering that these sessions happen for 6 days a week, the data transfer amount for the week would be, 97.76 Tb.
Considering 4 weeks in a month, the data transfer amount for the whole month would be, 391.06Tb. That’s a lot of data being transferred!
Now lets look at the cost. AWS ec2 charges $0.08/Gb for outbound data transfers from AWS ec2 o public internet. It essentially means AWS doesn’t charge for the teacher who is sending his / her audio and video streams to the media server but it charges for the students who are listening to the audio and video streams relayed by media server hosted in AWS ec2.
The maths looks like this.
391.06 * 1024 * 0.08 = $32,036
The amount of data consumed per month in Tb which is converted to Gb by multiplying 1024 along with AWS data transfer cost per Gb. This is the cost only for the data transfer and it doesn’t include the cost for running AWS ec2 instances for media servers. That cost will be be added to this cost on the actual usage basis.
Now lets look at the maths for running the same amount of maths tutoring sessions with media servers running on DigitalOcean.
There will be no change on the total amount of data transfers which is 391.06Tb.
The maths will look like this.
391.06 * 1024 * 0.01= $4004
This cost will further come down as there is free data transfer bundled with each DigitalOcean droplets. For example a 4 vCPU , 8GB CPU optimised instance comes with 5TB of free data transfer per month. With DigitalOcean, we can consider the final cost to be in the range of $3200 ~ $3500.
Due to this difference in the data transfer costs, we integrated DigitalOcean into CWLB 2.0 to provide an alternative to AWS ec2 to run media server with lesser cost. But this is purely optional and configurable from the loadbalancer settings of the admin dashboard.
Any organization admin can change their cloud vendor in the dashboard from AWS to DO or vice versa with a button click and the media servers will run the desired cloud as selected by the admin. The default cloud setting for running the media servers is now DO(DigitlOcean).It can be changed to AWS EC2 any time in the loadbalancer settings.
Some other important updates in CWLB 2.0 are as below.
Loadbalancing recording servers
Like media servers, the servers responsible for handling meeting recording can get exhausted quickly if there is a lot of demand for recordings. In order to solve this, we have now integrated recording server autoscaling to the load-balancer. Now the load-balancer can not only auto scale media servers but also recording servers in a fully automated manner.
Loadbalancing breakout rooms
Breakout rooms were already available in CWLB 1.0 but they were not very resource efficient. The customers had to use the same amount of credits to use breakout rooms as the main room. With CWLB 2.0, the breakout rooms are fully integrated in the JIT request handling mode into the load balancer so that the customers need not pay anything extra for using breakout rooms. It’s completely dynamic based on the actual usage of the breakout rooms irrespective of the main room size.
Due to current work pressure, we are not able to write an exhaustive list of all the updates that happened in CWLB 2.0 though we would love to write a exhaustive list when time permits. Until then if you have any query / suggestion related to CWLB 2.0, please feel free to drop us a mail at hello@centedge.io.
A real-life incident that happened with one of our customers.
A customer of ours having offices in the US and EU has a nice & innovative video conferencing application with some really cool features for collaborative meetings. They came to us for helping them fix some critical bugs and load balance their video backend. A piece of Interesting information we came to know is that they were running only one media server but a really huge one with 72 cores! The reason for running such a large server was that they wanted a lag-free & smooth video experience for all. In the beginning, when they had a small server, they were facing issues with video quality. Therefore, they took the biggest possible server for consistent video quality for all without even realizing that the video quality issue was due to the server. After digging deep, we made some interesting discoveries about their architecture and suggested some changes to their video infrastructure which includes downgrading the media server to an 8-core media server and having a horizontal load balancer to distribute the load effectively. After the suggested changes, their video infra bill was down by ~80%.
Here is the comparison.
Before:
A 72-core instance in AWS in the EU Frankfurt region costs $3.492/hour which becomes $2514.24 per month.
After:
An 8-core instance in AWS AWS in the EU Frankfurt region costs $ 0.348/hour which becomes $250.56
A horizontal load balancer instance also costs approximately the same, i.e. $250 /month.
So the total becomes $500/ month. A savings of ~80% per month on the cloud server bill!
When the CEO of the company got to know of the media server bill, he was skeptical about the business viability of the service because of the cloud bill that used to be paid every month. After the change, the prospect of the service seems more promising to him for business viability.
Load balancing WebRTC Media Servers, The Need
The rush for creating video conferencing apps is going to stay especially using WebRTC. As WebRTC 1.0 is already standardized by the Internet Engineering Task Force (IETF) by the date this post is being written, it is going to become mainstream in the coming times with the advent of 5G. Having said that, building a video conferencing app still is much more complicated than building a pure web app. Why? Because too many things need to be taken care of to create a production-ready video conferencing app. Those too many things can broadly be divided into 2 major parts. One is to code the app and test it on the local network(LAN ). Once it is successfully tested locally, it is time to take it to the cloud to make it available to a host of other users through the Internet. This is where dev-ops plays a critical role.
Now let’s understand why it is so important.
Let’s assume you have built the service to cater to 50 users in conferencing mode in each room. Now if you have taken a good VPS like c5-xLarge on a cloud provider like AWS, let’s assume it can support up to 10 conference rooms. What will happen if there is a need for an 11th room? In this case, you need to run another server that can handle another 10 rooms. But how will you know when the 11th room request will come? If don’t want to check manually every time a new room creation request comes, then there are 2 options. Either you tell your user who is requesting the 11th room that the server capacity is full and wait until a room becomes free OR create a logic so that a new server can be created magically whenever the new room creation request comes!! Now this situation is called auto-scaling and this is the magical effect of doing proper dev-ops on your cloud provider. The point to note here is that the way you are creating new servers as the demand grows, similarly you have to delete the servers when the demand reduces. Else the bill from your cloud vendor will go over the roof!!
Here is a brief summary of how a typical load-balancing mechanism works. I am not going to discuss the core logic of when to scale as that can be completely dependent on the business requirement. If there is a need to be up-scaled or down-scaled( short form for creating or deleting servers on demand, programmatically) according to dynamic demand, then there has to be a control mechanism inside the application to let the cloud know that there is more demand for rooms, that’s why more number of servers need to be created now to cater to the demand surge. Then the cloud has to be informed about the details of the VPS needed to be created like instance type, EBS volume needed, etc along with other needed parameters for the cloud to be able to create the server. Once the server is created, the cloud has to inform the application server back that the VPS has been created and is ready for use. Then the application server will use the newly created server for the newly created room and thus cater to the new room creation request successfully. A similar but opposite approach has to be taken when the rooms get released after usage. In this case, we need to let the cloud know that we don’t need some specific servers and they need to be deleted as they won’t be used until a new room creation request comes. When a new room creation request comes, one can again ask the cloud to create new servers and cater to the request for creating a new room successfully. This is how one will typically manage their dev ops to dynamically create and delete VPS according to the real-time need.
WebRTC auto-scaling/load-balancing, the strategies
Now that we understand what is DevOps in brief, let us also understand the general strategies to follow to do the dev ops, especially for the video conferencing use case. It can be broadly divided into 2 scenarios based on varied levels of automation that can be brought in to satisfy one’s business requirement. Though there can be a lot of variations of automation that can be brought in, let me describe 2 strategies for the sake of simplicity that can satisfy a majority of the business requirements.
In this strategy, the point is to automate the load distribution mechanism effectively to up-scale and down-scale the media servers while keeping the media servers in a cloud-agnostic manner. In this strategy, media server creation and deletion are not the scopes of load balancing. They can be independently created and updated in the load balancer in some manner so that there are enough servers always available to cater to when there is a surge in demand.
Pros:
Multi-cloud strategy
Better command and control
Less complex to implement
Cons:
Lesser automation
Strategy-2: Uni cloud Fully automatic load balancing
In this strategy, the point is to automate the load distribution mechanism effectively upscale and downscale while bringing in more automation while tightly coupling to a cloud provider.
In this, a cloud provider’s APIs can be integrated to create and destroy servers in a completely on-demand manner, without much manual intervention. In this approach, the load balancer can create servers from a specific cloud using APIs in case of an upscaling need and delete a server whenever the load decreases.
Pros:
Greater automation
Highly resource-efficient
Cons:
More complex to implement
Dependent on a single cloud vendor
There is no general rule that one should follow a specific load-balancing approach. It completely depends on the business requirement for which one needs load balancing. One should properly understand one’s business requirements and then decide the kind of load-balancing strategy that will be suitable. If you need help in deciding a good load-balancing strategy for your video infrastructure, feel free to have an instant meeting or a scheduled one with one of our core technical guys using this link.
Note: The load balancer mentioned in the above real-life incident is a WebRTC-specific stateful load balancer developed from scratch by us only for the purpose of auto-scaling WebRTC media servers. It is known as CWLB and more details about it can be found here.
In today’s digital age, communication has become the lifeblood of businesses and individuals alike. As organizations strive to connect with their remote teams, engage with customers, and collaborate with partners worldwide, video communication has emerged as a powerful tool. When it comes to building a video communication app, there are two main options: building from scratch or integrating Communication Platform as a Service (CPaaS) video API providers. While the latter may seem like an attractive choice due to its convenience, there are several compelling reasons why building your own video communication app is a better long-term investment.
First and foremost, building a video communication app gives you full control over the user experience. By developing your own app, you have the freedom to customize every aspect of the platform to align with your brand identity and specific requirements. From the user interface to the features and functionalities, you can tailor the app to create a seamless and intuitive experience for your users. This level of control is crucial for building strong brand recognition and fostering user loyalty.
Secondly, building your own video communication app allows you to prioritize data privacy and security. With increasing concerns about data breaches and privacy issues, having control over the infrastructure and data handling processes becomes paramount. By building your own app, you can implement robust security measures, encryption protocols, and data storage practices to safeguard sensitive information. This not only protects your users but also builds trust and credibility in your brand, setting you apart from competitors who rely on third-party providers.
Moreover, building a video communication app provides scalability and flexibility. As your business grows and evolves, you have the freedom to add or modify features, scale up the infrastructure, and adapt the app to changing market demands. This agility is crucial for staying ahead in a rapidly evolving digital landscape. On the other hand, integrating a CPaaS video API provider might limit your ability to customize or scale the app according to your unique requirements, potentially hindering your growth potential.
Building your own video communication app also offers cost-effectiveness in the long run. While integrating CPaaS video API providers may seem like a quick and cost-efficient solution initially, the subscription fees and usage charges can add up significantly over time. By building your own app, you have the opportunity to make a one-time investment in development and infrastructure setup, reducing ongoing expenses in the form of API usage fees. This allows you to have better control over your budget and allocate resources more efficiently.
Last but not least, building a video communication app provides a competitive edge. In a market saturated with generic communication tools, having a unique and tailored app sets you apart from competitors. It allows you to differentiate your brand and offer a distinctive user experience that aligns with your specific value proposition. By investing in building your own app, you position yourself as an innovative and forward-thinking organization, attracting users and potential partners who value a premium communication experience.
Let’s take the example of an virtual events company to understand the numbers between build vs buy strategy.
An virtual events company hosts 100 events a month with the average of 500 participants attending each of those events. Lets assume that each event is 6-8 hours long out of which 4 hours of audio/video is being used by all the participants and the virtual events company is using a video cPAAS provider to provide audio / video sharing capabilities to it’s participants.
The maths for 1 event would look like as below.
4 *60 = 240 minutes *200 participants * $.004/video minute= $192 (considering 200 participants sharing their video and audio)
Total $257(approx.) is what the cPAAS provider charges for 1 event.
If 100 such events happen, then the total cPAAS bill would be $25,700 for the month! If the virtual events company has been using the cPAAS services for last 2 years while hosting similar kind of events each month, total cPAAS bill would have been $616,800.
Now lets do some maths to find out what the amount would if the virtual event service provider would have decided to build it from day 1.
For building a really scalable video back-end which can replace their cPAAS offering, it should take 8-12 months with a cost of $150,000(approx.). The front-end and all other costs would stay the same.
The next important cost is the server and the data transfer cost.
Lets try to calculate the data transfer costs for 1 event with 500 people with 4 hours of audio / video usage.
1 participant consuming video and audio at 540Kb/s for 4 hours
540 * 60*60*4= 7.41Gb of data consumption for 4 hours
if 200 participants are using video, then total data consumption is 200 * 7.41 = 1483Gb
1 participant consuming audio at 40Kbps for 4 hours
40*60*60*4 = 0.54Gb of data consumption for 4 hours
if 300 participants are using audio, then total data consumption is 300 * 0.54 = 164Gb
Therefor total data transfer cost becomes 1648 Gb.
In case of AWS, the data transfer costs become = 1648* $.08/Gb = $132
In case of DigitalOcean, the data transfer costs become = 1648* $.01/Gb = $16.48
Considering that the virtual event provider is running all it’s services in DigitalOcean as it is cheaper, the total data transfer cost per month for all the 100 events would be = $1648. The server cost can be considered as included in this cost as we haven’t considered the free data transfer provided by DigitalOcean as a combined package with the servers.
cPAAS cost of $25,700 vs self managed infra cost of $1648, the difference becomes $24052.
According to this calculation, it seems that the virtual event company can recover the development cost of the self managed video back-end in less than 6 months and keep on saving at least $15000 a month considering people needed to manage and enhance the video back-end would cost monthly $9052.
A similar analysis can be done for any segment based on the above example. While integrating CPaaS video API providers might offer convenience in the short term, building your own video communication app presents numerous advantages in terms of user experience, data security, scalability, cost-effectiveness, and competitive differentiation. It empowers you to create a platform that truly reflects your brand and caters to the unique needs of your users. In a world where effective communication is paramount, investing in building your own video communication app is a strategic decision that sets you on a path of long-term success and growth. We are here to help you build a cutting edge video conferencing back-end / application for your unique use case. Feel free to drop us a email at hello@centedge.io or use this link to have an instant video meeting with us.
