Turn thumbs-up and thumbs-down into keyword-weight proposals by recording which terms contributed to each score, linking explicit feedback to that record, and aggregating signals only when there is enough evidence. Use a bounded update rule to suggest small changes—not to deploy them automatically. A person should review each proposal because feedback is not ground truth, and keyword weights interact with one another and with the score threshold.
What you need to record when a score is calculated
Feedback is useful for adjusting a term only if you can connect the reaction to the terms that actually influenced the result. At scoring time, persist an attribution record alongside the score. A practical record includes:
As an Amazon Associate I earn from qualifying purchases.
- Query or task context and the item identifier.
- Timestamp and the terms that matched.
- Each matched term’s contribution to the score.
- The weight version used and the resulting score.
If you retain only the final score, you cannot reliably reconstruct which terms caused it later. Keep the attribution record’s identifier available to the feedback mechanism so a reaction can be attached to the result that was shown.
How to collect feedback without treating it as truth
Record an explicit positive or negative reaction against the result and its attribution record. Preserve relevant context, such as a user or visitor identifier where appropriate, so you can distinguish user-specific signals from anonymous aggregate feedback. Amazon Kendra’s feedback API, for example, represents relevance as RELEVANT or NOT_RELEVANT and supports contextual association as well as anonymous aggregation. Its options and support vary by index type; check the current AWS documentation for the specific API and index you use.
#1 Best Overall
A thumbs-up means that a person judged a result useful in a particular context; it does not prove that every matching term deserves more weight. A thumbs-down has the same limitation in the opposite direction. As Craig Solomon puts it in the tutorial that frames this workflow, “Feedback is an opinion about relevance, and relevance is a business judgement.” Keep that distinction visible in the data model and review process.
How to turn feedback into a bounded proposal
Aggregate positive and negative signals by term, and by query segment when different kinds of searches should be treated separately. Set a minimum number of observations before a term becomes eligible for a proposal. This prevents a handful of reactions from triggering changes, but the threshold must be chosen and validated for your application; there is no universally established sample size for this workflow.
Rank #2
- Physical Condition: No Defects
- Great one for reading
- It's a great choice for a book person
One simple implementation heuristic is to calculate a positive share for eligible feedback:
positive_share = positive_count / (positive_count + negative_count)Compare that share with configured decision bands. If it is above the upper band, propose a small increase; if it is below the lower band, propose a small decrease; otherwise, make no proposal. Clamp the proposed value to configured minimum and maximum weights:
Rank #3
proposed_weight = clamp(current_weight + adjustment, minimum_weight, maximum_weight)The bands, minimum evidence, adjustment step, and weight limits are design parameters—not validated universal values. A small script can generate proposals, but this particular formula is only a transparent starting point. It does not account by itself for every interaction among terms, query types, score thresholds, or the circumstances in which results were displayed.
What a reviewer should see before approving a change
Keep proposals separate from active weights. For each proposal, show the term, old and proposed weights, positive and negative counts, the evidence window, and the weight version used to collect the observations. Where possible, include the expected effect on score distributions or representative results. Let a knowledgeable reviewer approve, reject, or defer the proposal, and record the reviewer, decision, and version history.
Rank #4
This approval step matters because changing one term can alter which results cross a threshold, while changing several terms can compound those effects. Relevance also depends on business judgment. Deploy approved changes as a versioned update so you can evaluate the result and restore the previous weights if needed.
Recommended Free Tools
How to evaluate approved changes
Before relying on a change, compare the updated scorer with the prior version on a held-out or curated set of relevance judgments. Include a range of query types rather than judging a change from the examples that generated it. If you evaluate online, use an evaluation design suited to your system and account for how results are exposed; do not infer success from a few favorable reactions.
Best Value
Click behavior deserves particular care: position and presentation affect what people click, so treating clicks as relevance labels can create biased training data. Research by Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel discusses position and presentation bias and propensity weighting as one correction; its findings are specific to the methods and experimental setting in the 2016 paper. Explicit thumbs feedback avoids some ambiguity in clicks, but it can still reflect the user and context rather than the intrinsic relevance of a result.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.When fixed keyword weights stop being enough
Manual proposals are relatively easy to inspect: a reviewer can see which term’s weight would change and why. They work best when terms have clear contributions and attributable feedback is available. If ranking depends on many interacting query-document features, a learning-to-rank (LTR) system may be a better fit, but it brings additional requirements for feature extraction, relevance judgments or suitable behavioral data, training, and deployment.
| Approach | What it uses | Review and operational considerations |
|---|---|---|
| Manual keyword-weight proposals | Term-level score attribution and positive/negative feedback tied to scored results. | Individual changes are straightforward to inspect; a bounded proposal script can be small, but evidence and human approval still matter. |
| Learning to rank | Query-document examples, features, and relevance labels or suitable behavioral judgments. | A model combines multiple features and requires feature, training, and deployment tooling. Click-derived labels may need bias correction. |
OpenSearch documents a Learning to Rank plugin for feature and model workflows in its LTR documentation. Elastic documents judgment lists, feature extraction, training, and inference, and recommends balancing examples across query types and including positive and negative judgments in its LTR guide. These systems are options for a broader ranking pipeline, not necessary components of a modest manual proposal loop.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
