Search relevance is not a setting you switch on and forget. Catalogues change, language shifts, seasons turn, and what counted as a great result in spring looks wrong by autumn. This playbook is for the merchandiser or digital manager who owns on-site search and wants a repeatable way to measure relevance and improve it deliberately rather than by gut feel.
You do not need to be technical. You do need a method.
What “relevant” actually means
Relevance is often discussed as if it were a single quality, but it has at least three layers that you tune separately:
- Match relevance — does the result genuinely relate to what the shopper typed?
- Ranking relevance — among the matches, are the best ones at the top?
- Business relevance — do high-margin, in-stock, on-strategy products get a fair showing?
A result can be a perfect semantic match and still be the wrong thing to show first — out of stock, low margin, or off-brand. Good tuning balances shopper intent with commercial reality. We cover the commercial side in AI Merchandising Control.
Step 1: Establish a baseline you can measure
You cannot improve relevance you have not measured. Build a baseline before touching anything:
- Top-query report. Pull your most frequent searches over 60–90 days. The top 100–200 queries usually cover the majority of search traffic.
- Per-query metrics. For each, capture click-through rate, conversion rate, and zero-result rate.
- A judged sample. Have a person rate the top results for your most important queries on a simple scale — relevant, borderline, irrelevant. This human view catches what dashboards miss.
This combination of behavioural data and human judgement is your scorecard. Without it, every change is a guess.
Step 2: Find the queries worth fixing
Do not try to perfect every query. Prioritise by impact:
- High-volume, low-conversion queries — lots of traffic, poor outcome. Biggest wins.
- High-volume, zero-result queries — pure lost revenue. Fix urgently. See How to Fix Zero-Result Searches.
- High-value category queries — terms tied to your best margins or strategic ranges.
- Branded queries — shoppers searching a brand expect to see it first; failures here erode trust fast.
Rank queries by volume multiplied by the size of the problem. Work top-down.
Step 3: Diagnose before you change anything
For each priority query, ask why the results are weak:
- Is it a vocabulary problem — shopper words not matching catalogue words?
- Is it a ranking problem — right products present but buried?
- Is it a data problem — thin titles or missing attributes starving the engine?
- Is it a stock or merchandising problem — relevant but commercially wrong items on top?
Diagnosis determines the lever. Reaching for synonyms when the real issue is ranking just adds noise.
Step 4: Pull the right lever
Synonyms and query rewriting
Map shopper language to catalogue language. Keep the list curated and reviewed — an unmaintained synonym file becomes a source of bad results. For catalogues where the list is becoming unmanageable, consider a semantic layer that handles meaning automatically; see Semantic Search for eCommerce.
Ranking and boosting rules
Adjust the signals that order results: text-match strength, popularity, recency, conversion rate, margin, and stock. Boost in-stock and high-performing products; bury or hide out-of-stock items unless you have a reason to keep them visible.
Pinning and curation
For a handful of high-value queries, manually pin the products you want at the top. Use sparingly — pinning at scale is unmaintainable.
Data enrichment
Often the highest-leverage fix and the most overlooked. Better titles, attributes and descriptions improve relevance everywhere at once. Search quality is downstream of data quality, which we explore in eCommerce Data Foundations.
Step 5: Test changes, do not just ship them
A relevance change that helps one query can quietly harm ten others. Protect against this:
- A/B test material changes where traffic allows, comparing search conversion and revenue per search session. Our approach is in A/B Testing with AI.
- Re-run your judged sample after a change to confirm you have not introduced regressions on other queries.
- Watch the aggregate, not just the query you targeted. Tunnel vision on one term is how teams break the long tail.
Step 6: Make it a routine, not a project
The teams with consistently good search treat tuning as an operating rhythm:
- Weekly: scan zero-result and low-conversion queries; quick synonym and ranking fixes.
- Monthly: review top-query metrics against baseline; refresh the judged sample.
- Quarterly: revisit ranking weights, retire stale synonyms and pins, plan seasonal adjustments.
Assign a clear owner. Search relevance without an owner drifts.
Common pitfalls
- Over-pinning and over-synonyming. Manual rules accumulate into a mess no one understands. Periodically prune.
- Optimising clicks over revenue. A query can earn clicks and still fail to sell. Anchor on conversion and revenue per session.
- Ignoring the long tail. The head queries are visible; the thousands of rare queries collectively matter and benefit most from semantic approaches.
- Forgetting seasonality. “Boots” means something different in July and December. Plan seasonal boosts.
- Treating relevance as purely technical. Merchandising judgement is half the job; do not hand it entirely to an algorithm.
A relevance scorecard to track
Keep these few numbers in front of you every month:
- Search conversion rate and revenue per search session — the outcomes that matter.
- Zero-result rate — should trend down.
- Click-through depth — how far shoppers scroll before clicking, a proxy for ranking quality.
- Judged relevance on your top queries — the human check.
The bottom line
Relevance tuning is a continuous, measured discipline: establish a baseline, prioritise queries by impact, diagnose the real cause, pull the matching lever, test before shipping, and run it on a regular cadence with a named owner. The retailers who do this consistently turn search from a cost centre into one of their best-converting surfaces.
If you would like a structured relevance audit and a tuning routine your team can run, our AI Search and Recommendations team can set it up with you. Get in touch and we will start with your own top-query data.