Which Parts of a Conversation Reveal What Someone Is Thinking?

NLP

Bayesian modeling

meta-analysis

measurement validation

Python

Observable

Six studies and 20 years of recorded conversations. Suggestions revealed more about what speakers were thinking than questions did, and the most recent words carried the most information.

Published

April 10, 2026

Modified

September 23, 2026

In brief. When you try to work out what someone is thinking, which of their words should you pay attention to? I combined six studies, recorded over roughly 20 years, in which people watched recorded conversations and wrote down what they thought a speaker was thinking; the speakers had also written down what they were actually thinking. I transcribed 74 hours of video, labeled each unique utterance as a dialogue act (e.g., question, suggestion, statement, and so on), and measured how closely each one matched the guesser’s guesses and the speaker’s real thoughts. Three results emerged: Suggestions matched what speakers were thinking more closely, on average, than any other kind of utterance, questions included. The most recent words carried the most information. And it made no difference whether those words came from the speaker or from their conversation partner.

20 years of recordings74 hours of video6 studies1,346 observers11,906 written guessessentence embeddingsBayesian modelsPythonR

traj = FileAttachment("data/trajectories.csv").csv({typed: true})
viewof side = Inputs.radio(new Map([["what observers guessed", "perceiver"], ["what speakers said they were thinking", "target"]]), {label: "Compare speech with", value: "perceiver"})
viewof overlay = Inputs.toggle({label: "Show each study", value: true})

pooled = traj.filter(d => d.side === side && d.level === "pooled")
studiesSel = traj.filter(d => d.side === side && d.level === "study")
speakerColor = ({domain: ["Target", "Interactant"], range: ["#123F36", "#C49A45"]})
speakerName = new Map([["Target", "the speaker being judged"], ["Interactant", "their conversation partner"]])
Plot.plot({
  width: 720, height: 360, marginLeft: 55, marginRight: 20,
  x: {label: "Position in the conversation segment →", domain: [0, 1], tickFormat: d => d === 0 ? "start" : d === 1 ? "end" : d},
  y: {label: "↑ Similarity between speech and " + (side === "perceiver" ? "the observer's guess" : "the speaker's thought"), grid: true, domain: [0, 0.3]},
  color: {...speakerColor, legend: true, tickFormat: d => "Words spoken by " + speakerName.get(d)},
  marks: [
    overlay ? Plot.line(studiesSel, {x: "time", y: "mean", z: d => d.study + d.speaker, stroke: "speaker", strokeOpacity: 0.3, strokeWidth: 1.2, tip: true, title: d => `${d.study} (${d.paradigm})\nwords by ${speakerName.get(d.speaker)}\nsimilarity ${d.mean.toFixed(3)} at ${Math.round(d.time * 100)}% of segment`}) : null,
    Plot.areaY(pooled, {x: "time", y1: "lower", y2: "upper", fill: "speaker", fillOpacity: 0.15}),
    Plot.line(pooled, {x: "time", y: "mean", stroke: "speaker", strokeWidth: 3, tip: true, title: d => `All studies pooled\nwords by ${speakerName.get(d.speaker)}\nsimilarity ${d.mean.toFixed(3)} (90% interval ${d.lower.toFixed(3)} to ${d.upper.toFixed(3)})\nat ${Math.round(d.time * 100)}% of segment`})
  ]
})

Each conversation was paused every few minutes so observers could write a guess and speakers could write down their actual thought. Left to right runs from the start of a segment to the pause; higher means what was said at that moment more closely matched what was written down. Both lines climb toward the pause, and the green (speaker) and gold (partner) lines sit on top of each other.

The question

Earlier research showed that people read each other more accurately when they can hear what is said. It had not asked which words, said when, and by whom. Research on conversation also treats questions as the best way to draw out what someone is thinking (Graesser & Person, 1994), so I expected questions to be the most revealing.

What I did

Turned video into text. I used a transcription model (Whisper) to transcribe the videos and coached a team of 15 research assistants to review transcriptions for accuracy I then hand-coded 5,000 utterances to train a model that split speech into single utterances; on a holdout sample utterances it had never seen, it agreed with human coding with 96.8% accuracy.

Labeled every utterance. I used a set of four language models to vote on whether each utterance was a question, suggestion, statement, agreement etc. The combined vote matched human coding 93.4% of the time, better than any published single model on this task.

Measured how close two sentences are. Every utterance, guess, and thought was converted into a sentence embedding so I could score how similar any two were in meaning.

How a computer compares two sentences

A sentence embedding turns a sentence into a long list of numbers that captures its meaning. “I’m nervous about the exam” and “this test is stressing me out” don’t share any words, but their lists of numbers end up close together because the underlying meaning is similar. Comparing two using the cosine of the angle between the vector of numbers gives a similarity score ranging from -1 to +1 where higher means closer in meaning. That let me ask, for every single utterance, how close it came to what the speaker was actually thinking.

Modeled change over time. Bayesian models traced how similarity rose and fell over the course of each conversation, one study at a time, and then combined all six studies into a single estimate.

Note

74 hours of video, 1,346 observers, 956 speakers, and 11,906 written guesses. Transcripts are IRB-restricted and stay in the lab; this page shows model summaries only.

What I found

Suggestions revealed the most. Suggestions (advice, recommendations, proposals) were, on average, the utterances closest to what speakers were actually thinking, closer than questions. Observers’ guesses pointed the same way, but the evidence there is weaker.

da = FileAttachment("data/dialogue_act_pooled.csv").csv({typed: true})
actOrder = ["suggestion", "question", "inform", "functional", "disagreement", "agreement", "acknowledgment"]
actLabel = new Map([["suggestion", "Suggestion"], ["question", "Question"], ["inform", "Statement"], ["functional", "Functional"], ["disagreement", "Disagreement"], ["agreement", "Agreement"], ["acknowledgment", "Acknowledgment"]])
sideLabel = new Map([["perceiver", "vs what observers guessed"], ["target", "vs what speakers were thinking"]])
// pool the two speakers (inverse-variance), then express each type as its distance from the average of the other six
daPooled = d3.groups(da, d => d.side, d => d.dialogue_act).flatMap(([side, acts]) => {
  const rows = acts.map(([act, g]) => {
    const w = g.map(d => 1 / Math.pow((d.upper - d.lower) / 3.92, 2));
    const m = d3.sum(g.map((d, i) => d.mean * w[i])) / d3.sum(w);
    return {side, act, m, se: Math.sqrt(1 / d3.sum(w))};
  });
  return rows.map(r => {
    const others = rows.filter(o => o.act !== r.act);
    const base = d3.mean(others, o => o.m);
    return {side: sideLabel.get(side), act: actLabel.get(r.act), dev: r.m - base, lo: r.m - base - 1.96 * r.se, hi: r.m - base + 1.96 * r.se, m: r.m, base};
  });
})
Plot.plot({
  width: 720, height: 300, marginLeft: 105, marginRight: 20,
  y: {label: null, domain: actOrder.map(a => actLabel.get(a))},
  x: {label: "Similarity, relative to the average of the other six utterance types →", domain: [-0.06, 0.08], grid: true, tickFormat: d => (d > 0 ? "+" : "") + d.toFixed(2)},
  color: {domain: ["vs what observers guessed", "vs what speakers were thinking"], range: ["#8a8f98", "#123F36"], legend: true},
  marks: [
    Plot.ruleX([0], {stroke: "#374151"}),
    ...["vs what observers guessed", "vs what speakers were thinking"].map((sd, k) => Plot.ruleY(daPooled.filter(d => d.side === sd), {y: "act", x1: "lo", x2: "hi", stroke: "side", strokeWidth: 2, dy: k === 0 ? -5 : 5})),
    ...["vs what observers guessed", "vs what speakers were thinking"].map((sd, k) => Plot.dot(daPooled.filter(d => d.side === sd), {y: "act", x: "dev", fill: "side", r: 4.5, dy: k === 0 ? -5 : 5, tip: true, title: d => `${d.act}, ${d.side}\n${d.dev >= 0 ? "+" : ""}${d.dev.toFixed(3)} relative to the other types (95% interval ${d.lo.toFixed(3)} to ${d.hi.toFixed(3)})\npooled mean ${d.m.toFixed(3)}, other types average ${d.base.toFixed(3)}`}))
  ]
})

How much each type of utterance stands out. Zero is the average of the other six types of utterances; points to the right matched more closely. Suggestions are the only type that clearly stands out for what speakers were thinking (green). For observers’ guesses (grey), the estimate is similar but less certain.

What the lines on each dot mean

Each line shows the range of values that fit the data. When the whole range sits to the right of zero, as it does for suggestions compared with speakers’ thoughts, we can be confident suggestions really stand out. On the observers’ side the range stretches back to zero. Observers may have picked up on suggestions, or they may not have; suggestions were rare enough in these recordings that a clearer inference couldn’t be made from the data.

The most recent words carried the most information. Similarity climbed toward each pause for both the guesses and the real thoughts (first figure). Because the climb shows up in the real thoughts too, observers who leaned on the last thing said were drawing on real information.

Who spoke made no difference. The partner’s words matched the speaker’s thoughts just as closely as the speaker’s own words did. While it may make intuitive sense to only focus on the target, the conversation partner’s speech was just as similar to their reported thoughts.

What this is good for

The same pipeline can be easily adapted to run on user interviews, support calls, or sales calls. For user research, the suggestions result is also a practical one. Analysis usually focuses on how people answer the questions we ask them. However, there are contexts when someone may volunteers a proposal (“it would be easier if…”) outside of an interview that may say more about what they are thinking.