Audio examplesPaper · arXiv link coming soon

Reference-Guided Turn-Taking Style Transfer for Full-Duplex Spoken Language Models

Every set of full-duplex weights speaks with one turn-taking personality. We turn it into a knob you can set per person.

Same model, same conversation, no training and no retraining. A five-minute recording of a speaker is enough to move the model's turn-taking toward that speaker.

a Extract from the model's own conversations User Model 80-ms frames onset 800 ms silent frames onset frames residual h at L30 μ− μ+ u initiative vector L31 L30 L0 audio in b Read and shift at L30 read out write in Read h · u onset Shift h′ = h + α s u speech ratio default −2a −a 0 +a +2a α
Fig. 1 · Method One direction, read and shifted. (a) From the model's own unsteered conversations, with no training and no annotation: among frames where the model has been silent, average the L30 residual before an onset within 800 ms (μ+) and before continued silence (μ−); u = (μ+ − μ−)/‖μ+ − μ−‖. (b) The projection h·u rises before the model takes the floor; adding h′ = h + α s u at every 80-ms frame shifts how much it speaks. s is the listen-to-speak swing of the projection, so doses compare across weights.
Control range
6–12% → 42–61%
Speech ratio across doses −2a … +2a, mean of 500 held-out conversations per weight.
Speaker-specific
92.7 / 98.0 / 95.6%
of reference pairs where giving a speaker someone else's dose makes the match worse (568 held-out speakers).
Calibration
93–99.5%
of the best achievable per-speaker gain, reached with a dose picked from the five-minute recording alone (568 held-out speakers).

01 · Sounds like you

One conversation, two speakers, opposite directions

Same partner (left channel), same model. Only the five-minute reference changes: a quiet speaker pulls the model below its default, a talkative one pushes it above. The model output (right channel) moves toward each speaker. Doses are steps along the initiative vector (see 02).

02 · Hear it change

Five doses, one conversation, one seed

Same input, same random seed; only the dose added along the initiative vector changes, from −2a to +2a. The model goes from barely speaking to holding the floor. The same direction and scale carry over to a second corpus, CANDOR, without refitting.

03 · It is this direction

Same displacement, random direction: the model does not take the floor

Same conversation as the dose sweep. All four clips share input, seed and displacement (+2a); only the direction differs. Swap the initiative vector for random unit directions, injected at the same layer with the same norm: the model does not start taking the floor. Only the initiative vector pushes it up.

04 · It knows before it speaks

The projection rises before the model takes the floor

Unsteered model on a representative held-out conversation. The curve is the per-frame projection of the L30 residual on the initiative vector, shown while the model is silent (the frames the readout is scored on); colored spans are where the model speaks. The projection climbs while the model is still silent and peaks as it takes the floor, anticipating onsets up to 800 ms ahead. The same direction that shifts the decision also reads it before any audio comes out.

How these examples were chosen