Imagine you're generating training data for a mobile app support assistant. You ask a model for questions about uploads, account access, offline sync, and failed backups. It gives you a few thousand q&a pairs.
You read a few rows and the answers seem reasonable, until you notice how often the advice comes back to "Please restart the app and try again."
Changing the feature name gives you another row, but it doesn't necessarily give the model something new to learn. Before generating another batch, it helps to count how much language you're repeating.
Start with a small example
Suppose these three answers appear in your generated data.
None of them is an exact duplicate. A check that only compares whole answers would keep all three. But most of each answer is identical, and the troubleshooting advice hasn't changed.
A unique n-gram ratio can catch this. It measures how often short sequences of words repeat across the dataset, without needing another model to judge them.
What is an n-gram?
An n-gram is a sequence of n consecutive words. Lets consider this sentence
the app freezes during upload
Its bigrams, where n = 2, are:
the appapp freezesfreezes duringduring uploadIts trigrams, where n = 3, are:
the app freezesapp freezes duringfreezes during uploadTo calculate the ratio, count the unique n-grams and divide by the total number of n-gram occurrences:
Diversity ratio = unique n-grams / total n-grams
For the three support answers above, there are 21 trigram occurrences and 11 unique trigrams. The ratio is 11 / 21 = 0.524. Phrases such as "please restart the" appear in all three answers, so they add three to the total but only one to the unique count.
A score of 1.0 means every observed n-gram occurs exactly once. A lower score means more repetition. That makes this a useful check for templated language, though it can't tell you whether the advice is correct or whether you've covered enough different problems.
Calculate it in Python
This example uses only Python's standard library. It counts phrases within each record, so the end of one answer never gets joined to the beginning of the next.
from collections import Counter
def ngram_counts(texts, n=3):
counts = Counter()
for text in texts:
words = text.lower().split()
counts.update(
tuple(words[i:i + n])
for i in range(len(words) - n + 1)
)
return counts
def ngram_diversity(texts, n=3):
counts = ngram_counts(texts, n)
total = sum(counts.values())
return len(counts) / total if total else 0.0
results = [
{
"question": "Why is my upload stuck?",
"answer": "Please restart the app and try the upload again.",
},
{
"question": "Why is my sync stuck?",
"answer": "Please restart the app and try the sync again.",
},
{
"question": "Why is my backup stuck?",
"answer": "Please restart the app and try the backup again.",
},
]
questions = [row["question"] for row in results]
answers = [row["answer"] for row in results]
for n in (2, 3):
print(f"Question {n}-gram diversity: {ngram_diversity(questions, n):.3f}")
print(f"Answer {n}-gram diversity: {ngram_diversity(answers, n):.3f}")
print("Most common answer trigrams:")
for phrase, count in ngram_counts(answers, n=3).most_common(5):
print(count, " ".join(phrase))
Running it gives:
Question 2-gram diversity: 0.667
Answer 2-gram diversity: 0.500
Question 3-gram diversity: 0.778
Answer 3-gram diversity: 0.524
Most common answer trigrams:
3 please restart the
3 restart the app
3 the app and
3 app and try
3 and try the
The phrase counts make the score easier to investigate and you can see exactly which wording keeps appearing. Keeping questions and answers separate also helps, because varied questions can hide repetitive answers in a combined score.
Use positive integers for n. Texts shorter than n words contribute no n-grams. If the whole input produces none, this function returns 0.0 as a convenience and treat that as "nothing to measure" rather than evidence of repetition.
Read the numbers carefully
There isn't a universal passing score. Three short answers and ten thousand long answers aren't comparable just because both produce a ratio between zero and one. More text creates more opportunities to reuse phrases. If you want to know more about Length bias in Distinct metrics read this Rethinking and Refining the Distinct Metric.
For a useful comparison between generation runs, use the same tokenizer, similar sample sizes, and similar length distributions. Record the total n-gram counts too. If the batches differ greatly in size, compare repeated random samples of comparable size before interpreting a change.
Tokenization changes the result. The code lowercases text and splits on whitespace. It still treats "upload" and "upload." as different tokens. That's simple enough for this English example, but a real pipeline should choose consistent punctuation handling and tokenization appropriate to its languages. Preserve the distinctions that matter to your task, especially if the data contains code or error messages.
Bigrams and trigrams reveal different amounts of context. Bigrams expose repeated word pairs; trigrams make longer shared phrasing easier to inspect. Compare each metric with the same metric from another batch. A higher trigram score than bigram score isn't itself evidence of an improvement.
Different wording can still cover the same problem
Consider "How do I recover my password?" and "Where can I reset my login credentials?" They share little wording, but they ask for essentially the same help. A dataset full of these variations might score well while missing offline behavior, expired sessions, or storage errors entirely.
The reverse happens too. Good support answers may repeat a required instruction or a precise setting name. Removing every repeated phrase could make those answers less useful.
So I would pair this check with a simple coverage review. For our support assistant, that means counting examples by problem and condition like upload interrupted by a network change, sync attempted offline, backup blocked by full storage, account access failing after a session expires. Then read samples from each group to check that the answers actually address those conditions.
If "restart the app" dominates the phrase counts, inspect those rows first. Where the advice is too generic, revise the generation prompt with specific situations and expected troubleshooting steps. Generate a fresh batch, check its coverage, and compare the ratios under the same conditions.
Before keeping the new batch, read a few answers to those specific problems. Does the interrupted upload get useful recovery steps? Does the offline sync answer explain what happens when the connection returns? Those are the improvements to look for alongside a change in the score.