Before you change a single ts_rank weight, freeze your corpus and score the current ranking against a set of graded judgments. Otherwise you are comparing two moving targets — new weights and new documents — and you will never know which one moved the needle. A relevance regression harness turns "search feels worse since Tuesday" into a number you can diff, the same way a test suite turns "the code feels broken" into a red check.
I wrote earlier about load-testing a Postgres full-text index and pinning down relevance. A commenter on that post sharpened the key point: if you tune weights against a live table, you are comparing two systems that are both changing at once. This post is the practical build-out of the fix — how to actually collect judgments, freeze the corpus, and compute the scores.
Why can't I just eyeball the search results?
Because search quality has no single-row assertion. A unit test says assert total == 42. Relevance says "for the query postgres connection pool, the pgbouncer guide should rank above the unrelated changelog entry, and the deep tuning post should sit somewhere in the top five." That is a judgment about an ordering, and orderings degrade quietly. You bump a title weight to fix one complaint, three other queries get subtly worse, and nobody notices until a user does.
