Speaker Diarization Explained: How AI Identifies Who Said What

What is speaker diarization?

Speaker diarization is the process of splitting an audio recording into segments and labeling each segment with the speaker who produced it — "who spoke when."

A raw transcript without diarization looks like this:

"Let's start the call. Sure, sounds good. Before we dive in, I want to flag the deadline."

With diarization:

Alice: "Let's start the call."

Bob: "Sure, sounds good."

Alice: "Before we dive in, I want to flag the deadline."

The second version is the one your users can actually read and act on.

How it works

Modern diarization pipelines run in two stages. First, voice activity detection segments the audio into regions of speech and silence. Second, a speaker embedding model (typically a neural network trained on thousands of speakers) extracts a fixed-length vector from each speech segment and clusters similar vectors together. Segments in the same cluster are labeled as the same speaker.

The model doesn't know the speakers' names — it only knows that speaker A in segment 3 sounds the same as speaker A in segment 12. If the meeting platform provides display names (as Google Meet does), those are matched to the diarized segments.

Why it matters for meeting APIs

A transcript without speaker labels forces whoever reads it to figure out context manually. That might be acceptable for a one-on-one call. It's unusable for a 10-person board meeting.

More practically, if you're building downstream features — action item extraction, CRM note generation, sales coaching — you need to know whether a statement came from a prospect or from your sales rep. Diarization is what makes that distinction possible.

What good diarization looks like in a webhook payload

{
  "segments": [
    { "speaker": "Alice Chen",   "start": 0.0,  "end": 3.4,  "text": "Let's start." },
    { "speaker": "Bob Nguyen",   "start": 3.8,  "end": 8.1,  "text": "Agreed. First item..." },
    { "speaker": "Gregnote Bot", "start": 0.0,  "end": 0.0,  "text": "", "is_bot": true }
  ],
  "speakers": [
    { "display_name": "Alice Chen", "is_bot": false },
    { "display_name": "Bob Nguyen", "is_bot": false }
  ]
}

The is_bot field lets you filter out the notetaker's own audio trivially — no regex, no post-processing.