Early studies of information retrieval systems featured quantitative approaches characteristic of the physical sciences. Mathematical formulas for precision and recall created an aura of objectivity for the nascent field of information science. And yet behind every formula lurked a variable that resisted isolation. Today we call this infuriating variable "the user" and we recognize that research must integrate rather than isolate the goals, behaviors, and idiosyncrasies of the people who use the systems.
Upon admitting the people problem, relevance was the first casualty, for measures of relevance are highly subjective. Ask individuals to evaluate the relevance of search results, and their responses will vary according to what they already know and what they want to know. Even the same individual may evaluate the same results differently as her knowledge and interest changes over time. Now this doesn't mean we should dismiss the metrics of precision and recall. For defined audiences and contexts (e.g., engineers using the HP intranet), sufficient agreement among users exists to make relevance measures meaningful. But we should proceed with an understanding that relevance is subjective, situational, and dynamic. Like beauty, relevance exists in the eye of the beholder.
Perhaps the most important thing we know about users is that they vigorously embrace what our friend George Kingsley Zipf called the Principle of Least Effort :
Each individual will adopt a course ...