Third edition of Artificial Intelligence: foundations of computational agents, Cambridge University Press, 2023 is now available (including the full text).
7.2.3.1 Probabilities from Experts
The use of pseudocounts also gives us a way to combine expert opinion and data. Often a single agent does not have good data but may have access to multiple experts who have varying levels of expertise and who give different probabilities.
There are a number of problems with obtaining probabilities from experts:
- experts' reluctance to give an exact probability value that cannot be refined,
- representing the uncertainty of a probability estimate,
- combining the numbers from multiple experts, and
- combining expert opinion with actual data.
Rather than expecting experts to give probabilities, the experts can provide counts. Instead of giving a real number such as 0.667 for the probability of A, an expert can give a pair of numbers as ⟨n,m⟩ that can be interpreted as though the expert had observed n A's out of m trials. Essentially, the experts provide not only a probability but also an estimate of the size of the data set on which their opinion is based.
The counts from different experts can be combined together by adding the components to give the pseudocounts for the system. Whereas the ratio reflects the probability, different levels of confidence can be reflected in the absolute values: ⟨2,3⟩ reflects extremely low confidence that would quickly be dominated by data or other experts' estimates. The pair ⟨20,30⟩ reflects more confidence - a few examples would not change it much, but tens of examples would. Even hundreds of examples would have little effect on the prior counts of the pair ⟨2000,3000⟩. However, with millions of data points, even these prior counts would have little impact on the resulting probability.
