Speaker Diarization Explained: How AI Knows Who Said What

What is speaker diarization?

Speaker diarization is the process of segmenting an audio recording by speaker — answering the question "who spoke when?" without knowing anything about the speakers in advance. The output is a set of time-stamped segments, each labeled with a speaker identifier (usually Speaker 1, Speaker 2, etc.) or a name if you've provided reference audio.

It's the technology behind the "Alice: Let's kick off the Q3 review" style transcript format you see in tools like Gregnote. Without it, you get a wall of text with no attribution, which is almost useless for meeting notes.

How it works under the hood

Modern speaker diarization combines three steps:

1. Voice activity detection (VAD)

Before doing anything with speaker identity, the system needs to find the speech segments — cutting out silences, background noise, music, and anything that isn't a human speaking. VAD models are typically trained on large corpora of speech and non-speech audio and are quite accurate for meeting audio.

The challenge is overlapping speech. When two people talk at once, VAD identifies the segment as "speech" but the diarization step has to decide whether to attribute it to one speaker or split it.

2. Speaker embedding extraction

For each speech segment, the model extracts a speaker embedding — a compact vector representation of the speaker's voice characteristics. These embeddings encode things like pitch, timbre, speaking rate, and resonance. The key property is that two segments from the same speaker should have similar embeddings, and segments from different speakers should have different embeddings.

The most widely used embedding models are based on ECAPA-TDNN and related architectures. Gregnote uses a fine-tuned variant of this family, trained specifically on meeting audio where there's more background noise, microphone variation, and codec compression than in clean studio recordings.

3. Clustering

Once you have embeddings for all speech segments, you need to group them into speakers. This is a clustering problem — find N groups (where N is unknown) such that segments from the same speaker are in the same group.

The clustering step is where most errors come from. Common failure modes:

Over-clustering: The model creates too many speaker clusters. A single person is split into two or more "speakers" because their voice changed significantly across the meeting (different mic position, got a coffee, turned away from the camera).

Under-clustering: Two speakers are merged into one. This happens when speakers have similar voice characteristics — same gender, same accent, similar speaking rate.

Overlap confusion: Segments where two people are talking simultaneously get attributed to one speaker or fragmented incorrectly.

Why diarization accuracy varies

Diarization accuracy is highly dependent on recording conditions. In controlled, ideal conditions (one microphone per speaker, low background noise, no overlapping speech), state-of-the-art systems achieve diarization error rates (DER) below 5%. In real meeting conditions, DER of 10–20% is more typical.

The factors that hurt accuracy most:

Microphone type: A single laptop microphone picks up everyone's voice at different distances and angles, creating level imbalances. USB conference speakers (like Jabra) are significantly better. Headsets are best.

Room acoustics: Reverberant rooms (echo-y conference rooms, large open spaces) make voice characteristics less distinct after the room's impulse response is applied.

Call platform compression: Platforms like Google Meet and Teams apply aggressive audio compression (opus codec, typically 16–48kbps). This removes high-frequency content that's important for speaker identity.

Number of speakers: Accuracy degrades as you add more speakers. Two speakers is easy. Eight speakers in one call is hard.

What developers can do to get better results

You have more influence than you might think. Here's what moves the needle:

1. Encourage headset use for important calls: This is the single biggest improvement available. A 16kbps opus stream from a headset sounds better for diarization than a 48kbps stream from a laptop mic with reverb.

2. Pass speaker count hints when you know them: If you're booking a meeting through your product, you often know how many people are invited. Gregnote accepts a speaker_count hint that constrains the clustering step:

{
  "meeting_url": "...",
  "hints": {
    "speaker_count": 4
  }
}

Even an approximate count (3–5 speakers) reduces over-clustering significantly.

3. Provide speaker names when available: If you know which participants joined (from a calendar integration, for example), you can pass their names and Gregnote will attempt to match voice clusters to names. This doesn't improve underlying accuracy but makes the output immediately useful.

4. Post-process the diarization: If your use case requires high accuracy (legal transcription, medical documentation), consider a human review step. Diarization errors in the transcript surface quickly when you read it — "Speaker 1: I agree with myself" is an obvious sign of over-clustering.

How diarization is evaluated

The standard metric is DER (Diarization Error Rate), which combines three error types:

• Missed speech: Regions labeled as non-speech that actually contain speech
• False alarm: Regions labeled as speech that don't contain speech
• Speaker confusion: Regions attributed to the wrong speaker

DER is calculated as: (Missed + False Alarm + Confusion) / Total Speech Duration

A DER of 10% means 10% of the meeting duration is incorrectly labeled in some way. For a 60-minute meeting, that's 6 minutes of errors — some of which you'll notice (a confused speaker label mid-sentence) and some you won't (a 2-second missed pause).

Diarization vs transcription accuracy

These are independent axes. You can have excellent diarization (correct speaker attribution) with poor transcription (wrong words) or vice versa. Gregnote's transcription pipeline uses Whisper-class models fine-tuned on meeting audio — transcription accuracy is generally high, around 95%+ word error rate on clean English. Diarization accuracy is good but will vary more with recording conditions.

Related reading

To understand how the full transcript gets from a meeting to your application, read our guide on meeting bot architecture at scale. If you're building a product that needs to display transcripts to end users, building meeting intelligence into your SaaS product covers the UX patterns that work.