Make the AI Wait: Build Explicit Floor Control for a Tencent RTC Voice Companion

Make the AI Wait: Build Explicit Floor Control for a Tencent RTC Voice Companion

Tópico: Make the AI Wait: Build Explicit Floor Control for a Tencent RTC Voice Companion
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
A real-time voice companion can produce an answer quickly and still be exhausting to use.

The problem appears when a user pauses to think. Automatic endpoint detection interprets the silence as the end of a turn, sends an incomplete thought to the LLM, and starts speaking just as the user finds the next sentence.

For developers using AI as a rubber duck, tutor, or creative companion, this creates a surprisingly personal tension: the machine's fluency starts setting the pace. A pause feels like a mistake even though pausing is often where the useful reasoning happens.

The counterintuitive fix is not to make every stage faster. It is to give the user explicit control of the conversational floor.

In this tutorial, we'll build a TypeScript coordinator with two input modes:

•
Automatic: an endpoint event can submit the current transcript.

•
Hold: silence does nothing. The companion waits until the user selects Ask AI.

The implementation will also reject stale LLM responses, stop output on interruption, preserve drafts across disconnects, and expose recoverable failures instead of silently resetting the conversation.

Tencent RTC provides the real-time voice layer for Conversational AI scenarios and can be connected to multiple LLM providers. Its Conversational AI overview is the starting point for the media and AI integration. This tutorial keeps the application policy independent from provider-specific callbacks.

Define the behavior before touching an SDK

These acceptance cases are more useful than a vague requirement such as "support natural turn-taking":

Situation
Required behavior

User pauses in Automatic mode
A verified endpoint may submit the draft

User pauses in Hold mode
Keep collecting; do not call the LLM

User selects Ask AI
Commit exactly the visible draft

User speaks while the agent is answering
Stop playback and invalidate the old turn

An invalidated LLM response arrives later
Ignore it

The network disconnects
Cancel active work but retain the unsubmitted draft

The LLM fails
Show Retry and Edit controls

Speech synthesis fails
Keep the accepted text available for reading or replay

The important distinction is that silence is an observation, not always consent to submit.

Keep the voice pipeline separated

A production voice companion has several independently fallible parts:

microphone
│
▼
RTC/media transport
│
▼
speech recognition ──► application turn coordinator
│
▼
moderation/policy
│
▼
OpenAI or another LLM
│
▼
speech synthesis
│
▼
RTC playback

Do not represent this entire pipeline with one isTalking boolean.

The RTC layer transports media. Speech recognition produces text and endpoint observations. The LLM generates a candidate answer. Speech synthesis produces output audio. Your application owns whether a draft may be submitted and whether a late result still belongs to the active turn.

Tencent RTC's Large Language Model configuration guide documents connecting OpenAI-compatible models and using request identification for routing and observability. Keep those identifiers aligned with your application turn IDs, but do not let provider configuration become your source of conversational state.

Create the project

mkdir patient-voice-companion
cd patient-voice-companion
npm init -y
npm install openai
npm install --save-dev typescript tsx @types/node
npx tsc --init
mkdir src
npm pkg set scripts.test="tsx --test src/*.test.ts"

The coordinator below has no dependency on a specific RTC, recognition, or synthesis callback name. Integration adapters will translate provider events into this application-owned vocabulary.

Model floor ownership as state

Create src/turn.ts:

export type InputMode = "automatic" | "hold";
export type Phase =
| "offline"
| "capturing"
| "thinking"
| "speaking"
| "recoverable";

export type PendingTurn = {
epoch: number;
prompt: string;
requestId: string;
attempt: number;
answer?: string;
};

export type TurnState = {
sessionId: string;
phase: Phase;
mode: InputMode;
epoch: number;
draft: string;
pending?: PendingTurn;
error?: "llm" | "synthesis";
};

export type Event =
| { type: "CONNECTED" }
| { type: "DISCONNECTED" }
| { type: "SET_MODE"; mode: InputMode }
| { type: "TRANSCRIPT_FINAL"; text: string }
| { type: "ENDPOINT_DETECTED" }
| { type: "ASK" }
| { type: "USER_SPEECH_STARTED" }
| { type: "LLM_OK"; epoch: number; answer: string }
| { type: "LLM_FAILED"; epoch: number }
| { type: "SPEECH_FINISHED"; epoch: number }
| { type: "SPEECH_FAILED"; epoch: number }
| { type: "RETRY" }
| { type: "DISCARD" };

export type Effect =
| {
kind: "requestLLM";
epoch: number;
prompt: string;
requestId: string;
}
| { kind: "cancelLLM"; epoch: number }
| { kind: "speak"; epoch: number; text: string }
| { kind: "stopPlayback" }
| { kind: "publishStatus"; message: string };

export type Transition = {
state: TurnState;
effects: Effect[];
};

export const initialState = (sessionId: string): TurnState => ({
sessionId,
phase: "offline",
mode: "automatic",
epoch: 0,
draft: ""
});

epoch is the admission token for asynchronous work. An LLM or synthesis result is valid only when its epoch matches the current state.

A request ID is for routing and diagnostics; the epoch determines whether an answer may still be used. Those are related concerns, but they are not interchangeable.

Implement the transition policy

Continue in src/turn.ts:

function submit(state: TurnState): Transition {
const prompt = state.draft.trim();

if (!prompt) {
return {
state,
effects: [{ kind: "publishStatus", message: "Nothing to ask yet." }]
};
}

const epoch = state.epoch + 1;
const requestId = `${state.sessionId}:turn-${epoch}:attempt-1`;

return {
state: {
...state,
phase: "thinking",
epoch,
draft: "",
error: undefined,
pending: { epoch, prompt, requestId, attempt: 1 }
},
effects: [{ kind: "requestLLM", epoch, prompt, requestId }]
};
}

function interrupt(state: TurnState): Transition {
const effects: Effect[] = [];

if (state.phase === "thinking" && state.pending) {
effects.push({ kind: "cancelLLM", epoch: state.pending.epoch });
}

if (state.phase === "speaking") {
effects.push({ kind: "stopPlayback" });
}

return {
state: {
...state,
phase: "capturing",
epoch: state.epoch + 1,
pending: undefined,
error: undefined
},
effects
};
}

export function transition(state: TurnState, event: Event): Transition {
switch (event.type) {
case "CONNECTED":
return { state: { ...state, phase: "capturing" }, effects: [] };

case "DISCONNECTED": {
const stopped = interrupt(state);
return {
state: { ...stopped.state, phase: "offline" },
effects: stopped.effects
};
}

case "SET_MODE":
return {
state: { ...state, mode: event.mode },
effects: [
{
kind: "publishStatus",
message:
event.mode === "hold"
? "Holding the floor. Silence will not submit."
: "Automatic turn submission enabled."
}
]
};

case "TRANSCRIPT_FINAL":
return {
state: {
...state,
draft: [state.draft, event.text.trim()].filter(Boolean).join(" ")
},
effects: []
};

case "ENDPOINT_DETECTED":
return state.mode === "automatic"
? submit(state)
: { state, effects: [] };

case "ASK":
return submit(state);

case "USER_SPEECH_STARTED":
return state.phase === "thinking" || state.phase === "speaking"
? interrupt(state)
: { state, effects: [] };

case "LLM_OK":
if (event.epoch !== state.epoch || !state.pending) {
return { state, effects: [] };
}

return {
state: {
...state,
phase: "speaking",
pending: { ...state.pending, answer: event.answer }
},
effects: [{ kind: "speak", epoch: event.epoch, text: event.answer }]
};

case "LLM_FAILED":
if (event.epoch !== state.epoch || !state.pending) {
return { state, effects: [] };
}

return {
state: { ...state, phase: "recoverable", error: "llm" },
effects: [
{
kind: "publishStatus",
message: "The AI did not answer. Retry, edit, or discard this turn."
}
]
};

case "SPEECH_FINISHED":
if (event.epoch !== state.epoch) return { state, effects: [] };
return {
state: {
...state,
phase: "capturing",
pending: undefined,
error: undefined
},
effects: []
};

case "SPEECH_FAILED":
if (event.epoch !== state.epoch || !state.pending?.answer) {
return { state, effects: [] };
}

return {
state: { ...state, phase: "recoverable", error: "synthesis" },
effects: [
{
kind: "publishStatus",
message: "Audio playback failed. The text answer is still available."
}
]
};

case "RETRY": {
if (!state.pending || state.error !== "llm") {
return { state, effects: [] };
}

const attempt = state.pending.attempt + 1;
const requestId = `${state.sessionId}:turn-${state.epoch}:attempt-${attempt}`;
const pending = { ...state.pending, attempt, requestId };

return {
state: {
...state,
phase: "thinking",
pending,
error: undefined
},
effects: [
{
kind: "requestLLM",
epoch: state.epoch,
prompt: pending.prompt,
requestId
}
]
};
}

case "DISCARD":
return interrupt(state);
}
}

There are two deliberate asymmetries here:

• A transcript remains editable until submission.

• Once submitted, a turn can be invalidated, but a late callback cannot reactivate it.

That second rule matters because cancellation is not proof that remote computation stopped. The response may still arrive.

Execute effects outside the state machine

The effect runner connects the deterministic policy to real services:

export interface VoicePorts {
complete(input: {
prompt: string;
requestId: string;
}): Promise<string>;

cancelCompletion(epoch: number): Promise<void>;
speak(input: { text: string; epoch: number }): Promise<void>;
stopPlayback(): Promise<void>;
publishStatus(message: string): void;
}

export async function runEffect(
effect: Effect,
ports: VoicePorts,
dispatch: (event: Event) => void
): Promise<void> {
switch (effect.kind) {
case "requestLLM":
try {
const answer = await ports.complete({
prompt: effect.prompt,
requestId: effect.requestId
});
dispatch({ type: "LLM_OK", epoch: effect.epoch, answer });
} catch {
dispatch({ type: "LLM_FAILED", epoch: effect.epoch });
}
return;

case "cancelLLM":
await ports.cancelCompletion(effect.epoch).catch(() => undefined);
return;

case "speak":
try {
await ports.speak({ text: effect.text, epoch: effect.epoch });
dispatch({ type: "SPEECH_FINISHED", epoch: effect.epoch });
} catch {
dispatch({ type: "SPEECH_FAILED", epoch: effect.epoch });
}
return;

case "stopPlayback":
await ports.stopPlayback().catch(() => undefined);
return;

case "publishStatus":
ports.publishStatus(effect.message);
}
}

Notice that failed cancellation is tolerated. Safety comes from rejecting the stale result, not from assuming cancellation always wins the race.

Put OpenAI behind the completion port

If your selected OpenAI-compatible endpoint supports the Chat Completions shape, a server-side adapter can look like this:

import OpenAI from "openai";
import type { VoicePorts } from "./turn.js";

const client = new OpenAI({
apiKey: process.env.LLM_API_KEY,
baseURL: process.env.LLM_BASE_URL
});

export const llmPort: Pick<VoicePorts, "complete"> = {
async complete({ prompt, requestId }) {
console.info("llm.request", { requestId });

const result = await client.chat.completions.create({
model: process.env.LLM_MODEL!,
messages: [
{
role: "system",
content:
"You are a voice companion. Answer the submitted thought; do not pretend to have heard unsent audio."
},
{ role: "user", content: prompt }
]
});

const answer = result.choices[0]?.message?.content?.trim();
if (!answer) throw new Error("Empty model response");

console.info("llm.response", { requestId });
return answer;
}
};

Keep credentials on the server. Confirm the exact model and request format supported by the endpoint you configure; "OpenAI-compatible" should not be treated as a promise that every optional OpenAI feature behaves identically.

The requestId above is application-owned correlation data. Carry the same identifier into the routing and observability configuration supported by your selected Tencent RTC Conversational AI setup rather than placing entire transcripts in logs.

Wire normalized voice events into the coordinator

Do not copy hypothetical callback names into your integration. Map the actual events from your Tencent RTC, recognition, and synthesis setup into the coordinator:

// Application-level mappings, not Tencent RTC API names.
recognitionAdapter.onFinalText(text =>
dispatch({ type: "TRANSCRIPT_FINAL", text })
);

recognitionAdapter.onEndpoint(() =>
dispatch({ type: "ENDPOINT_DETECTED" })
);

recognitionAdapter.onSpeechStarted(() =>
dispatch({ type: "USER_SPEECH_STARTED" })
);

ui.onHoldChanged(hold =>
dispatch({ type: "SET_MODE", mode: hold ? "hold" : "automatic" })
);

ui.onAsk(() => dispatch({ type: "ASK" }));
ui.onRetry(() => dispatch({ type: "RETRY" }));
ui.onDiscard(() => dispatch({ type: "DISCARD" }));

A visible control is preferable to relying only on phrases such as "let me think." Recognition can mishear the phrase, and users should not have to remember a magic incantation to control whether their speech is submitted.

A voice command can be an additional convenience, but the current mode should remain visible and directly reversible.

Reproduce the important races locally

Create src/turn.test.ts:

import test from "node:test";
import assert from "node:assert/strict";
import { initialState, transition } from "./turn.js";

function connected() {
return transition(initialState("session-a"), { type: "CONNECTED" }).state;
}

test("silence does not submit while the user holds the floor", () => {
let state = connected();
state = transition(state, { type: "SET_MODE", mode: "hold" }).state;
state = transition(state, {
type: "TRANSCRIPT_FINAL",
text: "The race might be in the cache"
}).state;

const result = transition(state, { type: "ENDPOINT_DETECTED" });

assert.equal(result.state.phase, "capturing");
assert.equal(result.state.draft, "The race might be in the cache");
assert.deepEqual(result.effects, []);
});

test("Ask submits the exact accumulated draft", () => {
let state = connected();
state = transition(state, { type: "SET_MODE", mode: "hold" }).state;
state = transition(state, {
type: "TRANSCRIPT_FINAL",
text: "First part."
}).state;
state = transition(state, {
type: "TRANSCRIPT_FINAL",
text: "Second part."
}).state;

const result = transition(state, { type: "ASK" });

assert.equal(result.state.phase, "thinking");
assert.equal(result.state.pending?.prompt, "First part. Second part.");
assert.equal(result.effects[0]?.kind, "requestLLM");
});

test("a response from an interrupted turn is ignored", () => {
let state = connected();
state = transition(state, {
type: "TRANSCRIPT_FINAL",
text: "Explain this design"
}).state;
state = transition(state, { type: "ASK" }).state;
const oldEpoch = state.epoch;

state = transition(state, { type: "USER_SPEECH_STARTED" }).state;
const result = transition(state, {
type: "LLM_OK",
epoch: oldEpoch,
answer: "This answer is now stale"
});

assert.equal(result.state.phase, "capturing");
assert.equal(result.state.pending, undefined);
assert.deepEqual(result.effects, []);
});

test("disconnect retains an unsubmitted draft", () => {
let state = connected();
state = transition(state, {
type: "TRANSCRIPT_FINAL",
text: "Do not lose this thought"
}).state;

state = transition(state, { type: "DISCONNECTED" }).state;

assert.equal(state.phase, "offline");
assert.equal(state.draft, "Do not lose this thought");
});

Run the suite:

npm test

These tests do not prove microphone quality or network behavior. They verify the application invariant under callback orderings that are difficult to reproduce manually.

Choosing between Automatic and Hold mode

Explicit handoff is not universally better. It exchanges conversational speed for control.

Experience
Better default
Reason

Short factual assistant commands
Automatic
The expected turn is brief and bounded

Coding rubber duck
Hold
Developers often pause inside one thought

Language pronunciation drill
Automatic
Fast repetition may be part of the exercise

Reflective or wellbeing companion
Hold
Silence may be intentional and sensitive

Hands-busy interaction
Automatic, with an accessible voice override
A screen control may be unavailable

Noisy social environment
Hold or push-to-talk
Endpoint observations may be unreliable

The hidden cost of Hold mode is interaction overhead. The hidden cost of Automatic mode is accidental submission, interruption, and pressure to speak continuously.

Measure both rather than optimizing only model latency:

• Time from explicit submission to first audible output

• Number of submissions immediately interrupted by the user

• Number of times users switch to Hold mode

• Drafts discarded before submission

• LLM, recognition, and synthesis failures by stage

• Recovery actions selected after a failure

Avoid storing raw audio or complete transcripts merely to obtain these measurements. Use consent, retention limits, bounded identifiers, and stage-level events. Users should be told when audio is being processed by AI and should always have visible mute, stop, and exit controls.

Failure drills worth running in staging

Endpoint events arrive after Hold mode is enabled

They must be ignored for submission. Switching modes should change policy immediately; it should not depend on restarting recognition.

The user interrupts while synthesis is starting

Both LLM_OK and USER_SPEECH_STARTED may be queued close together. Test both callback orders. Only the current epoch may reach audible playback.

Cancellation fails

The remote OpenAI-compatible request may continue. Keep the stale-result guard even if your provider exposes cancellation.

Recognition emits duplicated final segments

The simple accumulator in this tutorial will duplicate them. In production, normalize recognition results using stable segment identity supplied by the recognition layer, if available. Do not deduplicate only by text; a user can intentionally repeat a sentence.

The LLM fails after the draft was cleared

The accepted prompt remains in pending, so Retry can use the same text. The UI should also offer Edit, which copies that prompt back into a draft and invalidates the failed turn.

Synthesis fails after the LLM succeeds

Do not ask the LLM again. Preserve and display pending.answer, then offer text reading or a synthesis-only retry.

The connection drops while the user is thinking

Keep the unsubmitted draft locally for a bounded period, mark the session offline, and require a successful reconnection before submission. Do not let reconnection automatically send speech the user never explicitly committed.

What the AI demonstrates—and what it does not

A companion that returns a polished explanation demonstrates language generation over the context it received. It does not demonstrate that it heard the unsent part of a thought, understood why the user paused, or made a better engineering judgment than the user.

That distinction matters for developer confidence. Fast generated prose can make slower human reasoning feel obsolete, but the durable engineering work is elsewhere:

• deciding when enough evidence exists to commit a turn;

• separating provider output from application authority;

• preserving user intent across failures;

• testing races instead of trusting demo timing;

• choosing privacy and recovery behavior;

• deciding when not to automate.

The practical next step is small: add a visible Hold control and measure accidental submissions before changing models or chasing lower latency. If interruptions fall but users dislike the extra action, offer both modes and remember the preference with clear consent.

Release verification checklist

Before shipping, verify that:

• [ ] Hold mode is visible while active.

• [ ] Silence cannot submit in Hold mode.

• [ ] Ask AI submits the visible draft once.

• [ ] Barge-in stops audible output.

• [ ] Late LLM and synthesis callbacks are rejected by epoch.

• [ ] Disconnects do not erase unsubmitted speech.

• [ ] LLM retry does not rerun successful synthesis work, and synthesis retry does not rerun the LLM.

• [ ] Request IDs correlate stages without exposing transcript content.

• [ ] The UI distinguishes listening, thinking, speaking, offline, and recoverable states.

• [ ] Users can mute, stop, discard, and leave the experience.

• [ ] AI processing, transcript retention, and moderation behavior are disclosed.

• [ ] Automatic and Hold modes are tested with actual pauses, noise, and interruption timing.

Tencent RTC also describes AI virtual companions and character dialogue within its Social Entertainment solution. The floor-control policy here is especially relevant to those longer, less command-like conversations: natural interaction does not mean removing control. Sometimes the most useful thing a voice companion can do is wait.

Discussion

Where would you default to explicit handoff rather than automatic endpointing? More importantly, what evidence would persuade you to change that default?

Disclosure: I have a content relationship with Tencent RTC. Official Tencent RTC documentation was used as the implementation reference for this article.