Real-Time Media AI, Not Rented: An Open-Source Baseline You Actually Own
How the opensam open-source stack gives developers a from-first-principles alternative to frameworks like Pipecat — and the raw building blocks to prototype real-time media AI for voice and video use cases nobody else has templated.
Extract, process, inject — the whole idea in one picture. opensam solves the hard, undifferentiated plumbing — pulling raw media off a live call and pushing results back in. What model runs in the middle is your product.
This is the first of two posts. Here we look at what opensam lets you build and why owning this baseline matters. A follow-up will go deep on the how — the RTP plumbing, codec handling, and pipeline internals.
What is opensam, in one paragraph?
opensam is a dependency-light, open-source baseline for real-time media AI — the plumbing that pulls raw audio and video off a live call, hands it to any AI model in plain Python, and pushes the result back in. Unlike audio-first frameworks such as Pipecat or LiveKit Agents, opensam treats video as a first-class citizen and exposes the transport layer directly, so you can build voice agents, vision pipelines, or both on the same foundation — on-prem if you need to.
TL;DR
Most real-time voice-AI frameworks hand you a fast on-ramp and a set of opinions. That’s a great trade when your product looks like the demo. It’s a frustrating one the moment you need something the framework never imagined — processing video frames, injecting transformed media back into a live call, running on-prem models for compliance, or wiring AI into a dashcam feed instead of a support call.
opensam takes the opposite stance. It’s a dependency-light, transport-level baseline that shows you the entire path — from a browser’s microphone and camera, through a media server, into a plain Python process where any AI model can run, and back out to participants. You own every layer. Nothing is hidden behind a framework abstraction.
That makes it ideal for one job in particular: standing up a credible real-time media AI proof of concept, fast, for a use case that doesn’t fit the standard voice-agent mold.
Why a baseline you own beats a real-time media AI framework you rent
Frameworks like Pipecat, LiveKit Agents, or Vapi are genuinely good. They compress weeks of glue code into an afternoon. But they come with structural assumptions:
- They’re audio-first. The mental model is speech in → LLM → speech out. Video is, at best, a bolt-on. If your product is fundamentally about pixels — telematics, surveillance, avatars — you’re fighting the grain.
- They own the loop. The pipeline runner, the interruption logic, the turn-taking — that’s the framework’s territory. When you need to change how a frame is scheduled or how media is muxed, you’re patching someone else’s engine.
- They abstract the transport. You rarely touch the actual RTP. That’s convenient until you need to, and then there’s a wall between you and the bytes.
- Provider and deployment opinions. Swapping a model, running fully on-prem for a regulated customer, or metering per-room cost often means working around the framework.
opensam inverts all four. It is not a framework you extend; it’s a reference implementation you read, fork, and reshape. The AI loop is a few hundred lines of plain asyncio you can see end to end. The media path is explicit. Every provider is swappable because nothing depends on a provider-specific base class. And the whole thing is small enough to fully understand in an afternoon — which is exactly what you want under a proof of concept you’ll have to defend to a customer.
The point isn’t “frameworks bad.” It’s that a baseline — a clean, minimal, fully-owned starting point — is a different and complementary tool. When the requirement is novel, you want to start from bedrock, not from someone else’s opinions.
The stack, functionally
opensam ships three composable layers. You can start at whichever one matches your latency, scale, and control needs.
| Layer | What it is | When you reach for it |
|---|---|---|
| full-mesh | Pure peer-to-peer WebRTC calling (React + Node signalling + TURN). | Smallest footprint, 2–4 participants, no media server to run. |
| mediasoup | The same experience on a mediasoup SFU — clients upload once, the server fans out. Redis, Prometheus, Grafana included. | Larger rooms, scale, and — crucially — a server-side tap point for media. |
| v2ai | An AI participant that joins a mediasoup room, pulls raw audio and video off the wire, runs it through AI, and speaks/acts back. | Anywhere you want a model inside a live call. |
The magic layer is v2ai, and the reason is a single architectural idea worth stating plainly.
The one pattern that makes real-time media AI general: extract → process → inject
Every real-time media AI product, no matter how exotic, is a variation on the same loop:
- Extract raw media from a live stream.
- Process it with a model.
- Inject a result back — as audio, as an event, or as transformed media.
opensam implements this loop against a real media server rather than a toy file. On the extract side, the AI bot asks the SFU to copy a participant’s RTP to a plain transport, then receives:
- Audio — Opus RTP, decoded to clean 16 kHz PCM, ready for any speech model.
- Video — VP8 RTP, reassembled and decoded to raw frames, ready for any vision model.
On the process side, the repo demonstrates a full audio brain — voice-activity detection, speech-to-text, an LLM with tool use, and text-to-speech — plus a vision path that samples frames, encodes them, and sends them to a vision model.
On the inject side, it produces synthesized audio back into the live call as a normal participant would.
Here’s why that matters: once you can pull raw frames out and push media back in, you are no longer limited to “voice assistant.” Swap the model in the middle and the exact same skeleton becomes a driver-monitoring system, a security analyst, or a live video filter. The transport, the decode, the timing, the return path — the genuinely hard, undifferentiated plumbing of real-time media AI — is already solved and visible.
What’s in the toolbox
From a functional standpoint, a developer picking up opensam inherits:
- Media transport, both topologies. Mesh for the simplest path; SFU for scale and a clean server-side media tap.
- Raw media access. Decoded audio PCM and decoded video frames handed to you in Python — the thing frameworks hide and telematics/vision products desperately need.
- A working audio brain. Wake-word gating, Silero VAD, streaming STT (Deepgram with a Whisper fallback), an LLM layer with streaming and tool calls, and multi-provider TTS (Cartesia / Deepgram / a local Kokoro backup). Barge-in / interruption is handled — the bot stops talking when a human starts.
- A working vision path. Frame sampling (throttle the firehose to the rate your model can afford), JPEG encoding, and a vision layer that falls back across Claude and an on-prem LLaVA.
- Media output. A proven path for putting AI-generated audio back into the room — the template you mirror for any “inject” step.
- The unglamorous production-adjacent bits. Consent gating before any AI touches a participant’s video, PII scrubbing before text hits an LLM, OpenTelemetry tracing, Prometheus/Grafana dashboards, per-room cost tracking, and circuit breakers around flaky upstreams.
These are the things that make a proof of concept demoable to a real customer rather than a notebook. Everything is swappable because nothing is welded to a framework. Don’t want Deepgram? The STT boundary is one class. Need a local LLM for a data-residency clause? The LLM call is one function. That freedom is the whole point of a baseline.
Real-time media AI example use cases
The following illustrate how the same extract → process → inject loop retargets to very different products. They show where the repo already does the heavy lifting and where you’d plug in your own model.
1. Video telematics & driver safety
The idea: a camera in a vehicle (or a stream from an edge device) becomes a live feed that an AI watches continuously — flagging drowsiness, distraction, phone use, tailgating, or road hazards, and speaking warnings back to the driver in real time.
What opensam already gives you: the driver’s video arrives as decoded frames; the frame sampler throttles them to a sane rate (you don’t need 30 fps to catch a microsleep); the consent and cost machinery is in place; and the return path can speak an alert — “Eyes on the road” — through the same audio producer the meeting bot uses.
What you add: your detection model in the process step (a drowsiness/attention classifier instead of a slide summarizer). The event → voice-alert wiring is already the pattern; you’re changing what triggers the alert, not how it’s delivered.
2. Physical safety & security monitoring
The idea: a live camera feed — a site entrance, a factory floor, a retail aisle — is analyzed in real time for intrusion, missing PPE, weapons, loitering, or crowd density, raising events the moment something is seen rather than after the fact.
What opensam already gives you: continuous frame extraction from a live RTC stream, sampling to control model spend, a multi-provider vision path with an on-prem LLaVA fallback (critical when footage can’t leave the building), consent gating for privacy compliance, PII scrubbing for anything textual, and observability to prove the pipeline is alive.
What you add: the security-specific detection model and your alerting sink — a dashboard event, a webhook, an SMS. Because the vision boundary is explicit, running everything air-gapped on local models is a configuration choice, not a rewrite.
3. Face swap & live avatars in streaming
The idea: transform a participant’s video in real time — swap a face, apply a stylized avatar, blur a background, anonymize a whistleblower — and stream the modified video back into the call.
Why it’s the purest demonstration of the pattern: this is extract → transform → inject in its most literal form. opensam already implements the first two-thirds against real media: it pulls VP8 off the SFU and decodes it to raw frames you can hand to a face-swap or restyle model. The audio path then shows you exactly how to produce media back into mediasoup as a participant.
What you add: the transform model, plus the video-return leg — encoding processed frames and publishing them the way the repo already publishes audio. In other words, opensam hands you the ingestion, decode, and a working blueprint for the injection side; you supply the model and mirror the producer pattern for video.
That’s a real proof of concept on a foundation that would otherwise take weeks to build.
On latency: live face swap is the most demanding of the three — every frame is on the critical path, not sampled. The opensam baseline is where you prove the concept and the pipeline shape; hitting broadcast-grade frame budgets is a hardening exercise (see the disclaimer below).
What you’re expected to build on top
Being honest about the seams is part of what makes a real-time media AI baseline trustworthy:
- The “inject transformed video” leg is a blueprint, not a finished feature. Audio-return is implemented; video-return you build by following the same producer pattern.
- The AI models are examples, not the product. The value is the scaffolding around them. Your differentiation is the model you drop into the process step.
- Robustness is proof-of-concept-grade. Packet reordering, jitter buffering, multi-speaker scaling, and tight frame-timing are deliberately simple so the code stays readable.
None of these are surprises once you see the shape of the code — and each is exactly the kind of thing you’d expect to own when you choose a baseline over a framework.
From proof of concept to production
Disclaimer. opensam is intended for proof-of-concept and educational use. It is a reference baseline for understanding and prototyping real-time media AI — deliberately minimal, readable, and un-opinionated. It is not hardened for production traffic: it does not, out of the box, guarantee the latency budgets, jitter resilience, multi-tenant scale, security posture, or reliability that live products require.
When a proof of concept earns the right to become a product, you don’t want to spend the next two quarters rebuilding the plumbing into something production-grade. That’s precisely the gap the Samvyo media stack fills: the same capabilities — real-time transport, raw media access, audio and vision AI pipelines, media injection — engineered for the latency, scale, resilience, and compliance that production demands. opensam lets you prove what to build and that it works; Samvyo is the credible path to shipping it.
Think of it as a deliberate two-step: prototype real-time media AI on the open baseline, productionize on the stack built for it.
Getting started
Everything lives on the development branch, each project in its own folder with a dedicated README:
git clone git@github.com:Samvyo/opensam.git
cd opensam
git checkout development- Start with full-mesh to see real-time media with the least ceremony.
- Move to mediasoup when you need scale and a server-side media tap.
- Open v2ai to watch an AI join a live call, see the world, and speak back — then swap the model in the middle for whatever your product needs to see.
Fork it, read it, break it, and make it yours. That’s what a baseline is for.
Next post: a byte-level tour of how opensam extracts RTP audio and VP8 video from a mediasoup transport and runs it through the AI pipelines.
Frequently asked questions about real-time media AI and opensam
What is opensam?
opensam is an open-source, dependency-light baseline for real-time media AI. It extracts raw audio and video from a live call through a mediasoup media server, hands it to any AI model in plain Python, and injects the result back into the call — giving developers a transport layer they fully own instead of a framework abstraction.
How is opensam different from Pipecat or LiveKit Agents?
Pipecat and LiveKit Agents are audio-first frameworks that own the pipeline loop and abstract away the RTP transport. opensam is transport-level and treats video as a first-class citizen: it exposes the raw extract → process → inject loop directly, so developers can build both voice and video real-time media AI without fighting a framework built for speech-only use cases.
Can opensam process live video, not just audio?
Yes. opensam decodes VP8 RTP from the SFU into raw video frames in Python, samples them at a configurable rate, and feeds them to any vision model — the same pattern it uses for audio. This makes it suited to video-first real-time media AI use cases like driver monitoring, security analytics, and live face swap.
What media server does opensam use?
opensam runs on mediasoup, an open-source SFU (Selective Forwarding Unit). Clients upload media once, the SFU fans it out to participants, and an AI bot can tap a participant’s RTP stream server-side — the mechanism that makes real-time media AI possible without a browser-side plugin.
Is opensam production-ready?
No. opensam is explicitly a proof-of-concept and educational baseline. It does not guarantee production latency budgets, jitter resilience, multi-tenant scale, or security hardening out of the box. Teams that need to productionize a real-time media AI pilot typically move from opensam to a hardened stack like Samvyo, which implements the same extract-process-inject pattern with production-grade reliability.
What AI providers does opensam support by default?
The reference implementation wires up Deepgram (with a Whisper fallback) for speech-to-text, Cartesia, Deepgram, and a local Kokoro backup for text-to-speech, an LLM layer with streaming and tool calls, and a vision path that falls back across Claude and an on-prem LLaVA model. Every provider boundary is a single swappable class.
What can I build with opensam besides a voice assistant?
Because opensam exposes raw decoded audio and video rather than hiding it behind a voice-agent abstraction, the same extract → process → inject skeleton supports driver-safety telematics, physical security and PPE monitoring, live face swap and avatar streaming, and any other real-time media AI product built around a live camera or microphone feed.
Does opensam support on-premise or air-gapped deployment?
Yes — this is one of its core design goals. Because no component is welded to a specific provider’s base class, swapping a cloud STT, LLM, or vision model for a local equivalent (e.g., on-prem LLaVA) is a configuration change, not a rewrite, which matters for regulated industries and any environment where footage or audio cannot leave the building.
How do I get started with opensam?
Clone the repository (git clone git@github.com:Samvyo/opensam.git), check out the development branch, and work through the three layers in order: full-mesh for the simplest peer-to-peer demo, mediasoup for an SFU with a server-side media tap, then v2ai to see an AI bot join a live call and process real-time media AI end to end.
Who built opensam and why?
opensam is an open-source project from the team behind Samvyo, an enterprise real-time communications platform. opensam exists so developers can prototype novel real-time media AI use cases on a baseline they fully understand, before deciding whether to productionize on a hardened stack.