A semantic keyword clustering script in Python produces reliable page groups only when two things are right: a similarity cutoff derived from the corpus rather than hardcoded, and a linkage rule that cannot chain unrelated topics together. Fix those two, and the same command works on a keyword set it has never seen.
Tutorials usually fix neither. They embed keywords with a sentence-transformer, pick a cosine threshold of 0.75, group everything above it, and call it done. That version runs in twenty lines and falls apart on real data, because 0.75 means something completely different from one keyword set to the next. This guide builds the version that holds, then shows you how to act on its output.
What is Semantic Keyword Clustering?
Semantic keyword clustering is the process of grouping search keywords by meaning, using vector embeddings, so that each group maps to a single page. Two queries that mean the same thing, like “grey running shoes” and “gray shoes for running”, land in one cluster and get served by one URL. The technique replaces manual keyword grouping, which does not scale past a few hundred terms, with embeddings that place related queries close together in vector space and a clustering algorithm that reads those distances.
The output is a page map. Every cluster is one page’s target intent, and the head term of each cluster is its working title. Done well, semantic keyword clustering also exposes keyword cannibalization: any cluster that already ranks on more than one URL is a topic the site has split across competing pages.
Semantic Clustering vs SERP Clustering
The two ways to group keywords answer different questions, and knowing which you are running matters before you write code.
Semantic clustering groups keywords by embedding similarity. It needs only a local embedding model, runs offline, costs nothing per keyword, and is deterministic. Its weakness is that two queries can read alike yet return different search results, so pure meaning-matching occasionally groups intents Google keeps separate.
SERP clustering groups keywords whose top-ranking URLs overlap, for example any two queries that share three or more of the same top ten results. It reflects how Google actually clusters intent, which makes it the ground truth. Its weakness is cost: it needs a live SERP API call for every keyword, which is slow and metered.
| Semantic clustering | SERP clustering | |
|---|---|---|
| Groups by | Embedding similarity (meaning) | Overlap of top-ranking URLs |
| Data needed | An embedding model | A SERP API call per keyword |
| Cost per keyword | None | Metered API cost |
| Reflects Google intent | Approximately | Directly |
| Speed | Fast, offline | Slow, rate-limited |
For a Python workflow the practical answer is to run semantic clustering first, because it is free and fast, then SERP-validate only the borderline pairs where a wrong merge is expensive. The method below is semantic, and it adds a cannibalization report that reuses the ranking-URL column your keyword export already contains, so you get a SERP-like signal without paying for a single SERP call.
How to Cluster Keywords Semantically in Python?
You need three libraries: sentence-transformers for embeddings, scikit-learn for the clustering algorithm, and numpy for the vector math.
pip install sentence-transformers scikit-learn numpy
The pipeline is six steps: embed the keywords, measure the corpus, derive a cutoff from that measurement, cluster on average linkage, label each group, and score how tight each cluster is.
Step 1: Embed the keywords
Encode every keyword with a bi-encoder, L2 normalized so a dot product is a cosine. Cache the vectors to disk, because tuning means re-running, and re-encoding thousands of keywords on every run is the difference between a tool you tune and one you run once and abandon.
def embed(texts, model_name="BAAI/bge-small-en-v1.5", batch_size=64, cache=None):
if cache and Path(cache).exists():
blob = np.load(cache, allow_pickle=True).item()
if blob.get("texts") == texts and blob.get("model") == model_name:
return blob["vectors"]
from sentence_transformers import SentenceTransformer
model = SentenceTransformer(model_name)
vectors = model.encode(
texts, batch_size=batch_size, normalize_embeddings=True,
convert_to_numpy=True,
).astype(np.float32)
if cache:
np.save(cache, {"texts": texts, "model": model_name, "vectors": vectors})
return vectors
The cache key is the exact text list plus the model name, so changing either invalidates it automatically and you never cluster on stale embeddings.
Step 2: Measure the corpus before you cluster
Here is why a fixed cosine threshold fails. This is the pairwise cosine distribution measured on a real 2500 keyword corpus using BAAI/bge-small-en-v1.5:
mean similarity 0.616 (sd 0.088) p1 0.434 p50 0.607 p90 0.734 p95 0.782 p99 0.871 p99.9 0.953
Read the median. Two randomly chosen keywords from this corpus, sharing no content tokens, sit at 0.607 cosine. That is the floor of the embedding space for this data, not a relationship. A threshold of 0.75 lands near the 92nd percentile, so it admits roughly one random pair in twelve. On a narrower corpus, where every keyword shares a head noun, that same 0.75 lands somewhere else entirely and returns either one giant cluster or all singletons. Sample the distribution first:
def similarity_profile(vectors, sample=1500, seed=0):
n = vectors.shape[0]
rng = np.random.default_rng(seed)
idx = rng.choice(n, size=min(sample, n), replace=False)
sub = vectors[idx]
sim = sub @ sub.T
vals = sim[np.triu_indices(len(idx), k=1)]
return {"values": vals,
"percentiles": {p: float(np.percentile(vals, p)) for p in
[1, 5, 25, 50, 75, 90, 95, 99, 99.9]}}
Step 3: Derive the cutoff from the corpus, not from a blog post
Fix the percentile, not the cosine. The percentile is portable across keyword sets in a way the raw number never is.
def derive_cutoff(profile, percentile=99.0):
return float(np.percentile(profile["values"], percentile))
Now --percentile 99 means one concrete thing on any corpus: merge only pairs more similar than 99 percent of random pairs in this specific data. Point it at a broad keyword set or a narrow one, and the instruction holds. The setting travels. The cosine number never did.
Step 4: Cluster on average linkage
Feed the cutoff to agglomerative clustering. It works in distance space, so the threshold is one minus the similarity cutoff, and n_clusters stays None so the data decides how many groups the query set implies.
from sklearn.cluster import AgglomerativeClustering
model = AgglomerativeClustering(
n_clusters=None,
distance_threshold=1.0 - cutoff,
metric="cosine",
linkage="average",
)
labels = model.fit_predict(vectors)
Leaving n_clusters unset is the point. You are not guessing k. The corpus-derived cutoff determines how many pages the query set actually implies.
Step 5: Label each cluster with its head term
Every cluster needs a title, and the obvious choice is wrong. The highest-volume keyword in a group is usually the broadest one, which makes it a poor description and a bad page title. Take the member nearest the centroid, with volume only as a tiebreaker.
def pick_head(members, rows, vectors):
if len(members) == 1:
return members[0]
sub = vectors[members]
centroid = sub.mean(axis=0)
centroid /= np.linalg.norm(centroid) + 1e-12
scores = sub @ centroid
ranked = sorted(zip(members, scores),
key=lambda pair: (round(float(pair[1]), 4),
rows[pair[0]]["volume"]),
reverse=True)
return ranked[0][0]
The centroid term best represents the group’s meaning, which is exactly what belongs in the heading.
Step 6: Score how tight each cluster is
Cohesion is the mean pairwise cosine inside a cluster. It is the number that tells you whether a group is one topic or several stapled together. A large cluster with low cohesion is the signal that your cutoff is too loose.
def cohesion(members, vectors):
if len(members) < 2:
return 1.0
sub = vectors[members]
sim = sub @ sub.T
return float(sim[np.triu_indices(len(members), k=1)].mean())
Read cohesion alongside cluster size. A tight two-keyword cluster at 0.97 is trivially correct. A 143 keyword cluster that still holds 0.92 cohesion is a genuine single topic. A 300 keyword cluster at 0.30 is noise, and the fix is to raise the percentile and re-run.
The End-to-end Keyword Clustering Script
Here is the full flow assembled from the pieces above. It embeds, measures, derives the cutoff, clusters, and builds one record per keyword with its head term, cohesion, and the count of distinct ranking URLs in its cluster.
from collections import defaultdict
import numpy as np
def build_records(clusters, rows, vectors):
records = []
for cid, members in enumerate(clusters, start=1):
head_idx = pick_head(members, rows, vectors)
coh = cohesion(members, vectors)
head_vec = vectors[head_idx]
urls = {rows[i]["url"] for i in members if rows[i]["url"]}
total_vol = sum(rows[m]["volume"] for m in members)
for i in members:
records.append({
"cluster_id": cid,
"head_term": rows[head_idx]["keyword"],
"keyword": rows[i]["keyword"],
"volume": rows[i]["volume"],
"sim_to_head": round(float(vectors[i] @ head_vec), 4),
"cluster_size": len(members),
"cluster_volume": total_vol,
"cohesion": round(coh, 4),
"distinct_urls": len(urls),
"current_url": rows[i]["url"],
})
return records
def run(rows):
texts = [r["keyword"] for r in rows]
vectors = embed(texts, cache="embeddings.npy")
profile = similarity_profile(vectors)
cutoff = derive_cutoff(profile, percentile=99.0)
labels = AgglomerativeClustering(
n_clusters=None, distance_threshold=1.0 - cutoff,
metric="cosine", linkage="average",
).fit_predict(vectors)
grouped = defaultdict(list)
for i, lab in enumerate(labels):
grouped[int(lab)].append(i)
clusters = [sorted(v) for v in grouped.values()]
return build_records(clusters, rows, vectors)
rows is a list of dicts, each with keyword, volume, and url. Every row of the output tells you which page a keyword belongs on, how tight that page’s group is, and whether the group is already split across URLs.
Why Average Linkage Beats a Fixed Cosine Threshold?
The linkage rule is where keyword clustering scripts could potentialy fail, so it deserves its own decision. Three algorithms were tried on live keyword data, and the first two are the ones tutorials reach for.
Cosine threshold plus greedy seeding takes the keyword with the most neighbours above the threshold, claims them as a cluster, removes them, and repeats. It works when topics are already separated. When they are not, it carves arbitrary chunks out of a connected similarity graph, and the output looks structured while being noise. On the test corpus it produced one 345 keyword cluster with 0.03 internal cohesion, mixing bathroom, window, and siding queries.
Mutual k-nearest-neighbour connects two keywords only if each appears in the other’s top k neighbours. This fixes the hub problem, because a generic term adjacent to everything cannot absorb the corpus when specific keywords have closer neighbours that do not reciprocate. It does not fix chaining. “vinyl siding” links to “siding” links to “house siding” links to “house remodeling”, and connected components merges the whole chain. Result: 2348 of 2500 keywords in one component.
Agglomerative with average linkage prevents chaining by construction. Merging two clusters requires the mean similarity across every cross-pair to clear the cutoff, so a chain of individually close pairs cannot drag unrelated groups together. Single linkage chains. Complete linkage over-fragments on the worst pair. Average sits between them.
| approach | largest cluster | mean cohesion | clusters below 0.7 cohesion |
|---|---|---|---|
| threshold + greedy | 345 | 0.03 | most |
| mutual kNN | 345 | 0.15 | 21 |
| agglomerative, average linkage | 143 | 0.924 | 0 |
The 143 keyword cluster in the final run is grey and gray kitchen cabinet variants, which genuinely is one page.
Why Not K-means or DBSCAN?
Two other clustering algorithms come up constantly, and both are worse fits for keyword data than agglomerative clustering.
K-means requires you to specify the number of clusters in advance, which is the exact thing you do not know. The whole point of clustering a keyword export is to discover how many pages the query set implies. K-means also assumes clusters are roughly spherical and similar in size in Euclidean space, and keyword embeddings are neither.
DBSCAN does not need a cluster count and handles arbitrary shapes, which sounds ideal. The catch is that it needs a single density radius across the whole corpus, and keyword topics vary wildly in density. One radius either merges the dense areas into a blob or drops the sparse ones as noise. Agglomerative clustering with a distance threshold sets the cluster count from one interpretable knob, the cutoff, and average linkage gives you control over the chaining that density-based methods leave open.
Refuse to Cluster When There is Nothing to Cluster
Some corpora have no separable structure. When every keyword shares the same head noun, the embeddings occupy a narrow band and no cutoff produces meaningful groups. A tool that reports confident clusters on that input is worse than one that refuses, because the output still looks credible. The signal is the spread between the 99th and 50th percentiles.
def diagnose(profile, cutoff):
p = profile["percentiles"]
spread = p[99] - p[50]
if spread < 0.12:
return "degenerate", ["No separable structure. Split the corpus on its "
"shared head noun and cluster each part separately."]
if spread < 0.22:
return "weak", ["Cluster edges will be fuzzy. Treat output as a draft."]
return "ok", []
On a degenerate corpus the tool prints the diagnosis and exits instead of returning noise. That refusal is a feature, and it names the real next step: split the corpus on its shared noun and cluster each part on its own.
How to Read the Clustering Output?
Each row of the output CSV describes one keyword and the cluster it belongs to. These are the columns that drive decisions.
| column | meaning | how you use it |
|---|---|---|
cluster_id |
group identifier | the page this keyword maps to |
head_term |
keyword nearest the cluster centroid | the working page title |
sim_to_head |
cosine of this keyword to the head | low values flag a loose fit inside the group |
cluster_size |
keywords in the group | large plus low cohesion means split it |
cluster_volume |
summed search volume | rank your content plan by this |
cohesion |
mean pairwise cosine inside the group | below 0.55 on a big cluster means retune |
distinct_urls |
ranking URLs the group spans | above 1 is cannibalization |
Sort by cluster_volume to prioritize, then scan cohesion and distinct_urls to catch the two failure modes: a group that is too loose to be one page, and a group that is one page already split across many.
Read Ahrefs and Semrush exports without configuration
One practical note, because it is where real runs break before they start. SEO tool exports are hostile to naive CSV parsing. Ahrefs writes UTF-16 with a byte-order mark, tab-separated, inside a file named .csv. Semrush writes UTF-8. Column names vary by tool. Volume appears as 1200, 1,200, or 1.2K. A loader that auto-detects encoding and delimiter, maps column aliases across tools, and parses those volume formats is what lets one command run against any export. It is unglamorous, and it is the difference between a tool you use and one you fight.
Run the Keyword Clustering Script
python cluster.py keywords.csv --diagnose # inspect corpus structure first python cluster.py keywords.csv # cluster and report python cluster.py keywords.csv --percentile 99.5 # tighter groups python cluster.py keywords.csv --model BAAI/bge-base-en-v1.5 # higher-quality embeddings
Run --diagnose first, every time. It reports whether clustering is possible on this corpus and prints where your cutoff lands in the distribution. Degenerate means split the corpus. Weak means treat the groups as a draft. Clean means the cannibalization report is your consolidation roadmap.
Common Keyword Clustering Mistakes
The failures that produce confident-looking but useless clusters are consistent, and all of them are avoidable.
Hardcoding a cosine threshold is the first. A number that works on one corpus admits noise on a broad one and returns all singletons on a narrow one. Derive the cutoff from the corpus instead.
Choosing single linkage, or any algorithm that merges on the best pair, is the second. It chains unrelated topics through bridge keywords until half the corpus is one cluster. Average linkage requires the whole cross-group similarity to clear the cutoff.
Naming clusters by highest volume is the third. The biggest keyword in a group is usually the broadest, which makes a vague page title. Use the centroid term.
Trusting the output on a degenerate corpus is the fourth. If every keyword shares a head noun, no algorithm will separate them, and a tool that returns clusters anyway is lying to you. Check the similarity spread before you act.
Ignoring cohesion is the fifth. A cluster count looks tidy in a report, but a large group at 0.30 cohesion is several pages pretending to be one. Read cohesion and size together.
From Clusters to a Content Plan
Clusters are only useful when they become decisions. Work the output in this order.
- Rank every cluster by
cluster_volumeand treat the list as your production queue, highest demand first. Each cluster is one page, and itshead_termis the working title and primary target keyword. - For any cluster where
distinct_urlsis greater than one, you have cannibalization. Pick the strongest existing URL, fold the content from the competing pages into it, and redirect the losers to it. That consolidation usually moves rankings faster than new content, because you are concentrating signals you already earned. - For a large cluster with low cohesion, do not publish one bloated page. Raise the percentile, re-run, and let it break into the two or three real pages it contains.
- For high-volume singletons, decide case by case. A singleton with real demand is either a new page or a strong section on the nearest cluster’s page, depending on how much unique intent it carries.
Semantic Keyword Clustering FAQ
What Python libraries do I need for keyword clustering?
Three: sentence-transformers for the embeddings, scikit-learn for agglomerative clustering, and numpy for the vector math. Everything runs locally with no API keys, so there is no per-keyword cost.
Which embedding model should I use?
BAAI/bge-small-en-v1.5 is a strong default: fast, small, and accurate enough for keyword-length text. Swap it for a larger model like bge-base-en-v1.5 through the --model flag when you want higher separation quality and can accept slower encoding. Any sentence-transformers model works, so the choice is a speed-versus-quality trade you make per corpus.
Is semantic or SERP clustering better?
Start with semantic clustering. It is free, fast, offline, and deterministic, which makes it the right default for the first pass over a large keyword set. Add SERP clustering only to validate the borderline pairs where a wrong merge would cost you a page, since it needs a paid SERP call per keyword.
How do I choose the similarity threshold?
Do not choose a cosine number. Choose a percentile of the corpus’s own similarity distribution and let the code convert it to a cutoff. Start at the 99th percentile and raise it toward 99.5 if clusters are too loose, or lower it if you see too many singletons. The percentile is portable between keyword sets in a way a raw cosine is not.
How many keywords can it cluster at once?
The embedding step scales linearly, so it is not the bottleneck. Average-linkage agglomerative clustering builds a pairwise distance matrix, which grows with the square of the keyword count, so a single run is comfortable into the low tens of thousands of keywords on a laptop. Beyond that, split the corpus on a head noun and cluster each part separately, which the diagnosis step will tell you to do anyway.
What is keyword clustering in machine learning?
It is unsupervised learning applied to search terms. You convert each keyword to a vector with an embedding model, then group the vectors with a clustering algorithm, here agglomerative clustering with average linkage. There are no labels and no training step. The model reads the geometry of the vectors and the cutoff decides how many groups exist.
Why not use K-means for keyword clustering?
K-means makes you set the number of clusters up front, and discovering that number is the reason you are clustering in the first place. It also assumes even, spherical groups in Euclidean space, which keyword embeddings do not form. Agglomerative clustering with a distance threshold lets the data decide the count from one interpretable knob.
Is there a free keyword clustering tool?
Yes. The script in this guide is free and runs entirely on your machine with open-source libraries, so there is no subscription and no per-keyword charge. Commercial SaaS tools exist and are worth it when you need SERP-based clustering at scale, but for semantic grouping you own and control, a local Python script is the cheaper and more flexible option.
What is the best keyword clustering tool?
It depends on what you are optimizing. For control, zero per-keyword cost, and the ability to tune the method, a local semantic script like this one wins. For teams that need Google-accurate SERP clustering across very large keyword sets without writing code, a paid SERP-based tool is the better buy. Many practitioners use both: the script for the first pass, a SERP tool to validate the expensive merges.
Can I cluster keywords in Excel or Google Sheets?
Not semantically. Spreadsheets can only match on shared strings, so “shower remodel” and “bathroom shower renovation” never group even though they are one intent. Semantic clustering needs vector embeddings, which is why the work happens in Python rather than a formula.
How is this different from keyword grouping in Google Ads?
Google Ads grouping organizes keywords into ad groups by theme and match type to lift Quality Score and ad relevance, and a keyword can appear across several campaigns. SEO keyword clustering maps each query group to exactly one page to avoid two of your own URLs competing. The goals differ, so the groupings differ: tight ad groups can be far more granular than the page-level clusters you want for SEO.
Can it cluster non-English keywords?
Yes. Swap the default model for a multilingual sentence-transformers model through the --model flag, and the same pipeline clusters non-English or mixed-language keyword sets. The clustering logic does not change, only the embedding model.
How often should I re-run keyword clustering?
Re-run when your keyword export changes materially: after a fresh crawl, a new keyword research pull, or a batch of published pages. The embeddings cache on disk, keyed to the exact keyword list, so re-running after small changes is cheap. Quarterly is a reasonable cadence for a stable site, monthly for one publishing fast.
What separates a demo from a tool?
The twenty-line version is not wrong for using embeddings or cosine similarity. It is wrong because the number it hardcodes does not survive a new corpus and the linkage it picks chains unrelated topics. The percentile cutoff and average linkage remove both failures, so output quality holds on data the script has never seen. The diagnosis step refuses when there is no structure to find, and the cannibalization report turns clusters into a ranked list of pages to consolidate.
Point this at your own keyword export before your next content plan. Run --diagnose on your Ahrefs or Semrush CSV, read the cannibalization report, and consolidate the highest-volume split group first, because that is where the fastest ranking recovery sits. For the full cluster.py, the tool-agnostic loader, and the cannibalization report wired into one command, get the code and the walkthrough at lumkamishi.com.<di