Speaker Diarization: How 'Who Said What' Actually Works

Transcription answers what was said. Diarization answers who said it — and it is the part that quietly breaks on interviews, panels and meetings. Here is how it works, why it mislabels people, and how to get clean speaker labels.

By TranscribeVideo.ai Editorial TeamJuly 27, 2026

A transcript of a two-person interview with no speaker labels is close to useless. You can read every word and still not know which of them made the claim you wanted to quote. The feature that fixes this has a name most people never encounter until it goes wrong: speaker diarization.

It is a separate problem from transcription, it fails in its own particular ways, and almost all of those failures are decided by the recording rather than the software. This guide explains what it actually does, why it splits one person into three speakers, and what to change to get labels you can trust.

What speaker diarization is

Diarization answers the question who spoke when. Given a single audio file with several people in it, it divides the timeline into segments and groups those segments by voice, so the output reads:

Speaker 1  00:00–00:14   So the deadline moved to the fifteenth.
Speaker 2  00:14–00:21   The fifteenth of next month, or this one?
Speaker 1  00:21–00:25   This one. That is the problem.

Note what it does not do: it has no idea that Speaker 1 is Maria. It knows only that two distinct voices are present and which stretches belong to each. Putting real names on the labels is a separate step, and usually a manual one.

Diarization vs speaker recognition

These get conflated constantly, and the difference matters when you are choosing a tool.

Diarization separates voices without identifying them. No prior knowledge of the speakers is needed. This is what transcription tools almost universally provide.
Speaker recognition matches a voice against an enrolled profile and returns an actual identity. It requires voice samples collected in advance, and it carries consent and privacy obligations that diarization does not.

For interviews, meetings and podcasts, diarization plus a two-minute renaming pass is almost always the right answer. Recognition is worth the setup only when you are processing a large volume of recordings featuring the same fixed set of people.

How it works, roughly

You do not need the maths, but the outline explains every failure mode below.

Voice activity detection. The system first separates speech from silence, music and noise, so it only analyses the parts where someone is talking.
Segmentation. The speech is cut at likely speaker-change points — pauses, and abrupt shifts in acoustic character.
Embedding. Each segment is converted into a numerical fingerprint capturing the character of the voice: pitch, timbre, resonance, speaking style.
Clustering. Segments with similar fingerprints are grouped. Each cluster becomes a speaker label.

The critical thing to understand is step four. Diarization is a clustering problem, not a recognition problem. It is grouping things that sound alike. That is why anything which changes how a voice sounds mid-recording — distance from the mic, a bad connection, a cold — can push the same person into two clusters.

Why your speaker labels are wrong

One person appears as two or three speakers

The most common complaint, and it is nearly always an acoustic change rather than a software bug. Someone leans back from the microphone, turns to face another person, moves to a different room, or their connection quality drops mid-call. The fingerprint shifts, the clustering sees an outsider, and a new label appears. Long gaps between a person’s contributions make it more likely still.

Two people are merged into one speaker

The mirror image, and it happens when voices genuinely are similar — similar pitch, similar accent, similar recording conditions. Two colleagues of the same age and gender sharing one room microphone are a classic case. Compressed audio makes it worse, because compression removes exactly the fine detail that distinguishes them.

Interruptions get attributed to the wrong person

When two people speak simultaneously, the microphone captures one blended signal. Most systems must assign that stretch to a single speaker, so one person’s words land under the other’s label. Panels, heated discussions and enthusiastic meetings produce the worst labels for this reason, and no setting fixes it — the information was lost at capture.

Short interjections vanish

A one-word “exactly” or “no” may be too brief to fingerprint reliably, so it gets folded into whoever was speaking around it. Usually harmless, occasionally significant if the interjection was a disagreement.

How to get clean speaker labels

Ranked by how much difference they make. The first one dwarfs everything else.

Record separate tracks if you possibly can. If each participant is on their own channel, diarization becomes trivial — the channel is the label. Most podcast setups and several conferencing tools can do this. It is the single highest-impact change available.
Use a real microphone.A laptop’s built-in mic sitting two metres from a table of people is the worst case: everyone at a different distance, room reverb on all of it, and the acoustic differences flattened.
Control the room. Background noise, air conditioning, café clatter and music all degrade the fingerprints the clustering depends on.
Discourage crosstalk. On a recorded call, asking people to leave a beat before responding measurably improves the output. It sounds fussy; it works.
Tell the tool how many speakers to expect, where that option exists. An accurate count constrains the clustering and prevents one person fragmenting into several.
Keep a consistent distance from the mic. Moving around mid-recording is a leading cause of the same person being split.

Fixing labels afterwards

Perfect automatic diarization on a difficult recording is not a realistic target, and chasing it wastes more time than correcting the output does.

The efficient workflow is to accept the machine pass and then do one editing sweep: rename Speaker 1 and Speaker 2 to real names, and scan the turn boundaries, where errors concentrate. On a clean two-person interview that takes a couple of minutes. Compared with transcribing an hour of audio by hand, it is not a close contest.

A practical tip for interviews: have each participant say their name at the start. It gives you an unambiguous anchor for each voice, and it makes the renaming pass instant.

When speaker labels really matter

Interviews and journalism — attribution is the whole point, and quoting the wrong person is a serious error.
Research— separating the interviewer’s framing from the participant’s answer is fundamental to the analysis.
Meetings — action items are worthless if you cannot tell who committed to them.
Podcasts — show notes, pull quotes and repurposed clips all depend on knowing who said what.
Anything with legal or compliance weight — here the machine output is a starting point that a human should verify, not a record to rely on.

The short version

Diarization clusters voices that sound alike; it does not know who anyone is. Its accuracy is set mostly at the moment of recording, not by the software you run afterwards. Separate tracks solve it outright, a decent microphone and a quiet room get you most of the way, and a short manual pass closes the rest.

Once the labels are right, the transcript becomes genuinely useful — see how to transcribe an interview for the full workflow, extracting quotes from video for turning it into publishable material, and creating show notes if you are working with podcasts.

Frequently asked questions

What is speaker diarization?

Speaker diarization is the process of working out who spoke when in an audio recording. It splits the audio into segments and groups them by voice, producing labels like Speaker 1 and Speaker 2 against each stretch of speech. It runs alongside transcription — transcription answers what was said, diarization answers who said it — and the two are separate problems that can fail independently.

What is the difference between diarization and speaker recognition?

Diarization tells you that two different people spoke, without knowing who either of them is. Speaker recognition (or identification) matches a voice against a known enrolled profile and can put a real name to it. Almost every transcription tool does diarization; far fewer do recognition, because it requires you to enrol voice samples in advance and carries privacy and consent implications.

Why does my transcript label the same person as two speakers?

Usually because something changed about how their voice reached the microphone. Moving closer or further away, turning their head, a change in background noise, or a drop in call quality can all shift the acoustic signature enough that the system treats it as a new voice. It also happens when someone is on a poor connection, and when a long silence separates two stretches of the same person talking.

Why does diarization fail on interruptions?

Because two voices arriving at the same time produce one mixed signal, and separating overlapping speech is a much harder problem than segmenting turn-taking. Most systems handle a clean back-and-forth well and degrade sharply on crosstalk. Meetings where people talk over each other are the single most common cause of poor speaker labels.

How many speakers can diarization handle?

Accuracy is usually good for two to four speakers and falls off as the number grows, because the chance that two voices sound similar to the model rises with every person added. If your tool lets you specify the expected number of speakers, setting it correctly typically improves results noticeably — guessing high tends to fragment one person into several labels.

How do I improve speaker labels in a transcript?

Improve the recording first, because no amount of processing recovers information that was never captured. Record each participant on a separate channel or track where you can, since separate tracks make the problem trivial. Otherwise: use a decent microphone rather than a laptop's built-in one, reduce background noise, discourage crosstalk, and ask people to stay roughly the same distance from the mic. Where the tool allows it, tell it how many speakers to expect.

Does diarization work on phone calls?

It works, but less well than on in-person or good-quality video calls. Phone audio is heavily compressed and band-limited, which strips out much of the acoustic detail diarization depends on. Two similar-sounding voices on a compressed line are far harder to separate than the same two voices recorded well.

Can I fix speaker labels after transcription?

Yes, and for most real work that is the fastest route. Machine diarization gets you most of the way, then a quick pass renaming Speaker 1 to the real name and correcting a handful of mislabelled segments takes minutes rather than the hours a manual transcript would. Fix the labels at the point where the speaker changes — errors tend to cluster around turn boundaries and interruptions.