How to Fix Keyword Cannibalization (and Find the Pages Most Tools Miss)

Keyword cannibalization is the single most common issue I keep seeing at larger sites, especially marketplaces. The standard advice is to pull your Ahrefs or Google Search Console query report, find pages ranking for overlapping queries, merging a couple of obvious duplicate pages and calling it done. That advice is not wrong as it works for the obvious cases. However, it is severely incomplete, and the incompleteness is the whole problem.

The pages you can see fighting in your reports are a fraction of the pages actually competing. On a large site, the standard query-overlap check resolves roughly 30% of the cannibalization present. The other 70%, the dangerous ones, are invisible to the reports you are looking at, which means you can fix every problem you can see and still leave most of the damage in place.

There’s no query data on these pages because they never rank into positions where impressions get counted. They just sit there, dozens or hundreds of them on a large site, each one silently preying on queries that should belong to a single canonical page.

First Things First: What is Keyword Cannibalization?

Keyword cannibalization is what happens when multiple pages on your site target the same search intent, so they compete against each other for the same positions instead of one strong page ranking on its own. My preferred term, however, is topical cannibalization.

Search engines pick one page per site to rank for a given intent. When two or three of your pages qualify, the engine has to choose, and it often splits ranking signals across the group, picks the weaker page, or discounts all of them because no single page is the clear owner of that topic.

Backlinks, internal links, and engagement that should have concentrated on one URL get scattered.

There is an AI-search dimension to this now too. Assistants like ChatGPT and Google’s AI Overviews cite a small number of pages per domain when they answer. If your coverage of a topic is spread thin across near-duplicate pages, none of them is the obvious source to cite, and you lose the citation to a competitor who consolidated.

What is not keyword cannibalization?

Before you fix anything, rule out the false alarms. Not every case of two pages ranking for one term is a problem.

Broad topics legitimately need multiple pages. A design tool ranking several pages for “ui/ux design” is covering different sub-intents (guides, product, templates), not cannibalizing. The test is intent.
Branded queries. It is normal and healthy to rank several pages for your own brand name.
Different points in the buyer journey. A how-to guide and a product page can share a keyword while serving completely different intents. Merging them would destroy value, not recover it.

The line is always search intent. Two pages that answer the same question for the same person are cannibalizing. Two pages that share vocabulary but answer different questions are not. This being said programmatic content, if done correctly, does not create cannibalization.

Why You Often Can’t Fix What You Can’t Find?

Query reports (Google Search Console, Semrush, Ahrefs) only show pages that already rank into impression-counting positions. A page that never breaks into those positions produces no query data at all. So the pages quietly suppressing each other, the ones that should rank but do not, are structurally invisible to the exact reports everyone uses to find cannibalization.

On a content-heavy site those hidden pages number in the dozens or hundreds. They are also the more damaging cases:

A visible split is bounded and diagnosable. You can see two pages trading positions, and you can fix it.
A topical overlap presents as nothing. Pages that should rank simply do not, and no signal tells you why. The whole cluster gets discounted because Google cannot find a clear owner for the topic, not just the duplicate pages.

Before you can fix keyword cannibalization properly, you need a way to surface the pages the reports never show you. The following three-layer method does that.

How to find every cannibalized page (3 layers)

The layered approach catches a class of problems the layer before could miss:

Layer 1: Query overlap (catches the obvious offenders)

Start with the standard checks. They are fast, free, and clear out the low-hanging fruit.

In Google Search Console

Open Performance, then Search results. Filter by a query and look at the Pages tab. If two or more URLs earn impressions and clicks for the same query, you have a candidate. That’s what that is, only a candidate.

In Semrush or Ahrefs

Use the position-tracking or organic-keywords view. Semrush has a dedicated Cannibalization report inside Position Tracking that groups this automatically. Flag three conditions:

Multiple pages ranking for the same query inside the top 20.
Multiple pages with heavy impression overlap across near-identical query sets.
One query where the ranking URL keeps swapping week to week, which signals Google cannot decide the owner.

With a manual site search

Run site:yourdomain.com "your keyword" in Google and read the results for pages serving the same intent. Document the problem URL pairs in a spreadsheet.

This layer finds the visible ranking splits. Do not skip it, but do not stop here. It is the 30%.

Layer 2: Topical overlap (catches the hidden 70%)

To find the invisible cases, you have to compare pages by what they are about, not by what already ranks. This runs in two stages, and neither works alone.

Stage 1, surface the candidates cheaply

Export every indexable URL with its title and meta description (a Screaming Frog crawl does this well). Turn each page’s title and description into an embedding (a numeric fingerprint of its meaning) and compute the similarity between every pair of pages. Pairs scoring above about 0.65 similarity become candidates. A free local model handles this fine; you do not need paid embeddings to surface candidates.

Here is the actual computation. Start with a CSV of three columns: URL, Text (the page title), and Description (the meta description). The embedding model turns each row into a vector, and cosine similarity measures the angle between two vectors, where 1.0 is identical and 0 is unrelated.

import pandas as pd
from sentence_transformers import SentenceTransformer, util

# CSV columns: URL, Text (page title), Description (meta description)
df = pd.read_csv("pages.csv").fillna("")

# Weight the title 2x by repeating it, then append the meta description.
# The title carries the intent; the description only adds context.
df["embed_input"] = df["Text"] + ". " + df["Text"] + ". " + df["Description"]

# all-MiniLM-L6-v2 is small, runs locally, and is free. Paid embeddings add
# no accuracy at the candidate-surfacing stage, so save the budget for Stage 2.
model = SentenceTransformer("all-MiniLM-L6-v2")
emb = model.encode(df["embed_input"].tolist(), normalize_embeddings=True)

# Pairwise cosine across every page. Normalized vectors mean the dot product
# IS the cosine, so one matrix multiply scores the whole site at once.
sim = util.cos_sim(emb, emb).numpy()

# Keep pairs at or above 0.65, skipping self-matches (i == j) and mirror
# duplicates (j starts at i + 1, so each pair is emitted once).
THRESHOLD = 0.65
pairs = []
for i in range(len(df)):
for j in range(i + 1, len(df)):
if sim[i, j] >= THRESHOLD:
pairs.append((round(float(sim[i, j]), 3), df.URL[i], df.URL[j]))

for score, a, b in sorted(pairs, reverse=True):
print(score, a, "<->", b)

Embed the title and description (or the first two H2 tags) only, not the full page body, because body text adds template boilerplate and calls to action that inflate similarity without signaling shared intent. And set the threshold at 0.65, not lower, because the band below that wastes review time and Stage 2 reclassifies the borderline pairs anyway. On a 2,000-page site this is a few seconds of compute and produces a ranked candidate list, highest similarity first.

all-MiniLM-L6-v2 is free and runs on your own machine, which is all you need to surface candidates. If you would rather call a hosted embedding API (stronger vectors, nothing to install), swap the model.encode() line for one of these and keep the rest of the code identical:

OpenAI embeddings (text-embedding-3-small or text-embedding-3-large)
Voyage AI, the provider Anthropic points to for embeddings since Claude has no first-party embedding model
Cohere Embed
Google Gemini embeddings

The math has a blind spot. It collapses pages that share words but split on intent, and it misses pages that share intent but not vocabulary. That is why there is a second stage.

Stage 2, verify intent with an LLM

Send each candidate pair to a language model and ask it to do what the math cannot: read both pages, extract the actual topic of each (not the keywords), and decide whether they target the same search intent. Then have it re-rate the risk:

Verdict	Condition
High	Same topic, same intent
Medium	Same topic, different angle
Low	Related topic, different intent

Here is that step in code. Feed it the candidate pairs from Stage 1, batch them so you are not paying for one request per pair, and ask the model to return a structured verdict rather than prose you have to parse.

import json
import anthropic

client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from the environment

# Each candidate is (title_a, url_a, title_b, url_b), carried over from Stage 1.
# A small, cheap model is the right call here: the work is a judgment about
# intent, not heavy reasoning, and batching keeps a 2,000-pair run near $0.30.
MODEL = "claude-haiku-4-5"
BATCH = 15

# Force a typed response so there is nothing to hand-parse and no format drift.
SCHEMA = {
"type": "object",
"properties": {
"results": {
"type": "array",
"items": {
"type": "object",
"properties": {
"index": {"type": "integer"},
"topic_a": {"type": "string"},
"topic_b": {"type": "string"},
"same_intent": {"type": "boolean"},
"risk": {"type": "string", "enum": ["High", "Medium", "Low"]},
},
"required": ["index", "topic_a", "topic_b", "same_intent", "risk"],
"additionalProperties": False,
},
}
},
"required": ["results"],
"additionalProperties": False,
}

def verify_batch(pairs):
listing = "\n".join(
f"{i}. A: {ta} ({ua})\n B: {tb} ({ub})"
for i, (ta, ua, tb, ub) in enumerate(pairs)
)
prompt = (
"For each numbered pair of pages, work out the real topic of each page "
"(what it is actually about, not the keywords in its title), decide "
"whether both pages target the SAME search intent, and rate the "
"cannibalization risk:\n"
"- High: same topic, same intent\n"
"- Medium: same topic, different angle\n"
"- Low: related topic, different intent\n\n"
f"Pairs:\n{listing}"
)
resp = client.messages.create(
model=MODEL,
max_tokens=4096,
output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
messages=[{"role": "user", "content": prompt}],
)
text = next(b.text for b in resp.content if b.type == "text")
return json.loads(text)["results"]

verdicts = []
for start in range(0, len(candidates), BATCH):
batch = candidates[start:start + BATCH]
for r in verify_batch(batch):
title_a, url_a, title_b, url_b = batch[r["index"]]
verdicts.append((r["risk"], url_a, url_b, r["same_intent"]))

# Sort High-risk pairs to the top for review.
verdicts.sort(key=lambda v: {"High": 0, "Medium": 1, "Low": 2}[v[0]])

Combined work of the two stages

Why both stages matter, from a real audit:

False positives caught. Of 45 pairs the math flagged as high risk, only 19 survived intent verification. A 33% false-positive rate at the top tier. Example: “Northwind Aeon trail running shoe” and “Northwind Aeon road running shoe” can score 0.86 similarity but cover different product lines. Merging them would have destroyed a legitimate comparison page.
False negatives recovered. 122 pairs (about 6% of the corpus) scored below the similarity threshold but were the same intent. Example: “Running shoes for flat feet” and “stability running shoes” score just 0.65, but both target the same buying decision, and neither one ranked. These pairs are invisible to the similarity math and invisible to the query reports at the same time. This is the hidden cannibalization that Layer 1 can never reach.

The math finds candidates, while the LLM makes the judgment call. Use one without the other and you either merge pages you should keep or miss pages you should merge.

Layer 3: SERP overlap (settles the close calls)

Google’s live results are the ground truth that settles a Stage 2 judgment call. Two pages whose target queries return the same results are competing for one intent, whatever the similarity score or the model said.

For any pair you are unsure about, search the core query of each page in an incognito window and compare the top 10 organic results:

Overlap of top 10	Verdict
90% or more	One intent. Cannibalization confirmed.
40% to 89%	Contested. Partial overlap, review by hand.
Below 40%	Different intent. Leave both pages alone.

Divergence has a recognizable shape. If one query returns mostly guides and the other returns mostly product pages, the two sit at different points in the buyer journey and are not in conflict, no matter how similar the titles look.

Run this on the borderline pairs at minimum. It is the cheapest possible tiebreaker, and it keeps you from merging pages that only looked like duplicates.

The full three-layer pipeline (embeddings, LLM intent verification, and SERP confirmation) is documented in more depth in the cannibalization detection method write-up. The point for this guide: by the time you reach the fixes below, you should have a list of confirmed cannibalized pairs, not a list of pages that merely share a keyword.

Automate the SERP pull, do not eyeball it

Nobody checking a real audit opens incognito windows one query at a time anymore, and eyeballing does not scale past a handful of pairs. Use a SERP API:

DataForSEO is the cost-effective option for batch work. Its standard queue runs a few minutes per request and comes in around $0.60 per 1,000 SERPs, which is the right shape for an audit, where you have hundreds of queries to check at once and no reason to need any single result back in under a second.
SerpApi, Serpstack, or Zenserp are simpler search APIs with the same idea: send a query, get back parsed organic results as JSON.

Whichever you use, the workflow is the same. Deduplicate your query list first, because a 2,000-pair audit collapses to a few hundred unique queries and you pay per query, not per pair. Pin every request to the same location and language, since comparing SERPs pulled under different geo settings measures the geo difference instead of the intent difference. Parse organic results only, strip ads and local packs, and compute the overlap of the two top-10 URL sets. At these volumes the whole SERP pass costs a dollar or two.

What “overlap” means here, and why it is not cosine?

This is the one place in the method where embeddings are the wrong tool. Layer 2 compared what two pages are about, so semantic vectors and cosine similarity were exactly right. Layer 3 on the other hand, compares which literal URLs Google returns, which is a set-membership question, not a meaning question.

You want to know whether the same pages show up for both queries, and “the same page” means an exact URL match, not a similar-sounding one. Running cosine over the SERPs would actively mislead you: two result sets can be semantically near-identical (all ten about running shoes) while sharing zero URLs, and that zero-overlap case is the signal that the two queries have different competitive sets and therefore different intents. Semantic fuzziness is the last thing you want here.

So the comparison is set overlap on normalized URLs. Reduce each result to domain plus path (drop the protocol, www, trailing slash, and query string), then count how many of the top 10 the two queries share. Two refinements earn their place: Jaccard, the formal set-similarity number, and a rank-weighted score so that eight shared URLs in the same order read as a tighter match than eight in scrambled order.

from urllib.parse import urlparse

def canon(url):
"""Reduce a result URL to domain + path, so only real page identity matters."""
p = urlparse(url if "://" in url else "https://" + url)
host = p.netloc.lower().removeprefix("www.")
return host + p.path.rstrip("/").lower()

def serp_overlap(list_a, list_b):
a = [canon(u) for u in list_a]
b = [canon(u) for u in list_b]
sa, sb = set(a), set(b)
shared = sa & sb

# Primary metric: fraction of the top 10 shared. This is what the decision
# table reads against (9 of 10 shared -> 0.9 -> "confirmed").
overlap = len(shared) / min(len(sa), len(sb))

# Jaccard: the formal set-similarity number. Reads lower than `overlap`
# because the union grows as the two sets diverge.
jaccard = len(shared) / len(sa | sb)

# Rank-weighted (Rank-Biased Overlap, persistence 0.9): rewards shared URLs
# that also sit at similar positions. Normalized so identical lists score 1.0.
p = 0.9
depth = min(len(a), len(b))
rbo = sum(len(set(a[:d]) & set(b[:d])) / d * p ** (d - 1) for d in range(1, depth + 1))
rbo = rbo * (1 - p) / (1 - p ** depth)

return {"overlap": round(overlap, 2), "jaccard": round(jaccard, 2), "rbo": round(rbo, 2)}

The overlap column is what maps to the decision table above. Jaccard and the rank-weighted score are there to break ties in the contested 40-to-89% band, where two SERPs share a moderate number of URLs and the ordering tells you whether they are really competing for one intent or just brushing against each other.

Manual incognito checking still works as a fallback for the last few genuinely borderline pairs, and it has one advantage: it surfaces AI Overviews, featured snippets, and people-also-ask boxes that some APIs drop. But it is the exception now, not the method.

How to Fix Keyword Cannibalization?

Now the part everyone came for. You have a confirmed list of cannibalized pages. The fix has two moves: decide which page survives, then apply the right technique to consolidate everything onto it.

Getting the second move right is easy. Getting the first move wrong can suppress the cluster for months, so start there.

Step 1: Choose the survivor

For each cannibalized group, pick the one page that becomes the canonical owner of that intent. Weigh:

Backlinks. The page with the strongest, most relevant referring domains is usually the survivor, because links are the hardest signal to move.
Rankings and traffic. The page already earning the most qualified traffic has momentum you do not want to throw away.
Content quality and fit. The page that most completely and accurately serves the intent, or the one easiest to expand into the definitive answer.
URL and internal-link equity. A page many internal links already point to is expensive to replace.

Then decide the fate of the others: merge, redirect, differentiate, canonical, noindex, or delete. The techniques below map each situation to the right move.

Step 2: Apply the right technique

Fix 1: Consolidate and 301 redirect

Best for: near-duplicate pages you do not need to keep separately. This is the highest-impact fix and the default for true duplicates.

Pick the preferred page (Step 1).
Fold the useful, unique content from the other pages into it, so the survivor becomes the more complete answer.
Add 301 permanent redirects from the retired URLs to the survivor.
Update internal links that pointed at the retired pages so they point at the survivor.
Remove the retired URLs from your XML sitemap.

A 301 consolidates ranking signals (including backlinks) from the retired pages into one location. This is what turns three weak pages into one strong one.

Fix 2: Canonical tags

Best for: identical or near-identical pages you must keep live (filtered e-commerce URLs, PPC landing pages, print or multi-path versions).

Add a canonical tag to the duplicates pointing at the preferred page:

<link rel="canonical" href="https://www.example.com/preferred-page/" />

Guidelines that keep canonicals from backfiring:

Use absolute URLs, not relative paths.
Point at an indexable page that returns a 200 status.
Never chain canonicals (A points to B points to C). Point every duplicate straight at the final survivor.

Canonicals consolidate signals without removing the duplicate from the site, which is why they suit cases where the duplicate has to stay reachable.

Fix 3: Differentiate and re-optimize

Best for: pages you want to keep and can meaningfully separate, where the overlap is an accident of optimization rather than genuine duplication. This is the right call for many Layer 2 “same topic, different angle” pairs.

Give each page a distinct primary intent, and rewrite so they stop overlapping. Cut the shared sections; keep each page to its own angle.
Re-optimize each page for its own target: title tag, H1, URL slug, headers, and meta description all pointing at the distinct keyword.
Add internal links between the two with descriptive anchor text so you tell Google which page owns which intent.

Done well, differentiation turns a cannibalizing pair into a small, coherent topic cluster where each page ranks for its own term.

Fix 4: Noindex

Best for: last resort. Thin pages with no backlinks and negligible traffic (some tag or filter pages) that you need reachable for users but not competing in search.

<meta name="robots" content="noindex" />

Understand the trade-off: noindex removes the page from the index, but it does not consolidate its ranking signals onto the survivor the way a 301 or canonical does. Use it only when there are no signals worth saving.

Fix 5: Delete

Best for: pages with no traffic, no links, and no user value that are only adding noise to the cluster. Redirect the URL if it has any history; otherwise let it 410. Fewer, stronger pages beat more, weaker ones.

Which fix to use: quick reference

Situation	Fix
Near-duplicate pages, don’t need both	Consolidate + 301 redirect
Must keep duplicate live (params, PPC, print)	Canonical tag
Want both, can separate the intents	Differentiate + re-optimize
Thin page, no signals, keep for users	Noindex
No traffic, no links, no value	Delete (or 410)

One rule cuts across all of them: do not split a topic across pages that share one intent. Every fix above is a way of telling Google which single page owns a given intent. Pick the technique that fits the page, but make the ownership unambiguous.

How to Confirm the Fix Worked?

The recovery pattern is the diagnostic in reverse, and it tells you whether you chose the right survivor.

Correct consolidation produces broad, sudden recovery rather than gradual gains. Pages across the whole topical neighborhood lift together, not just the URLs you merged. That cluster-wide lift is the proof that the cluster was being suppressed by cluster-wide noise, which is exactly what topical cannibalization does.

If instead you see a single page recover slightly while the cluster stays flat, you likely consolidated onto the wrong survivor, or you fixed the visible split while leaving the hidden 70% in place. Re-run Layer 2 on that cluster.

Give it a few weeks. Consolidation changes take time to recrawl and re-rank, so measure the trend over 4 to 8 weeks, not day to day.

How to Prevent Keyword Cannibalization?

Fixing is remediation. Prevention is cheaper. Once your site is clean, keep it clean:

Keep a keyword map. Maintain a spreadsheet that assigns one primary keyword (and its intent) to one canonical URL. Every new page has to claim an intent that no existing page owns.
Check before you publish. Before writing new content, search your own site and your map for existing coverage. If a page already targets that intent, improve it instead of creating a second one.
Expand, don’t duplicate. The instinct to publish a fresh post for every keyword variation is the root cause of most cannibalization. Ask whether the new angle deserves its own page or belongs as a section on an existing one.
Monitor continuously. Re-run the three-layer audit on a schedule (quarterly is reasonable for an active site). Layer 1 catches new visible splits; Layer 2 catches the topical drift that accumulates as your library grows.

Frequently Asked Questions

How do I know if keyword cannibalization is actually hurting me?

Look for pages that should rank but don’t, ranking positions that swap between URLs, or a whole topic cluster that underperforms its content quality. Confirm with the three-layer method: query overlap for the visible cases, topical similarity plus intent verification for the hidden ones, and SERP overlap to settle close calls.

Will fixing keyword cannibalization improve my rankings?

Usually yes, when you consolidate onto the right survivor. The signal is a broad, cluster-wide lift rather than a single page nudging up. If nothing moves after 6 to 8 weeks, you probably chose the wrong survivor or only fixed the visible portion.

Should I always merge cannibalizing pages?

No. Merge true duplicates. For pages that share a topic but serve different angles or different points in the buyer journey, differentiate and re-optimize them instead. Merging pages that target different intents destroys value.

Do canonical tags fix keyword cannibalization?

They help when you need to keep a duplicate live but want one page to get the ranking credit. They consolidate signals without removing the page. For duplicates you don’t need at all, a 301 redirect is stronger.

Why do most cannibalization tools miss so much?

Because they read query reports, and query reports only include pages already ranking into impression-counting positions. Pages that never break in produce no data, so they are invisible to those tools even while they suppress each other. Finding them requires comparing pages by topic and intent, not by existing rankings.

How long does it take to see results after fixing keyword cannibalization?

Typically 4 to 8 weeks. Google needs to recrawl the redirects or canonicals and re-evaluate the cluster. Treat it as a trend over weeks, not a day-to-day metric.