Hammitt, James K. and Zhang, Yifan (2012) Combining Experts’ Judgments: Comparison of Algorithmic Methods using Synthetic Data. TSE Working Paper, n. 12-293

There is a more recent version of this item available.

Preview

Text Download (616kB) | Preview

Abstract

Expert judgment (or expert elicitation) is a formal process for eliciting judgments from subject-matter experts about the value of a decision-relevant quantity. Judgments in the form of subjective probability distributions are obtained from several experts, raising the question how best to combine information from multiple experts. A number of algorithmic approaches have been proposed, of which the most commonly employed is the equal-weight combination (the average of the experts’ distributions). We evaluate the properties of five combination methods (equal-weight, best-expert, performance, frequentist, and copula) using simulated expert-judgment data for which we know the process generating the experts’ distributions. We examine cases in which two well-calibrated experts are of equal or unequal quality and their judgments are independent, positively or negatively dependent. In this setting, the copula, frequentist, and best-expert approaches perform better and the equal-weight combination method performs worse than the alternative approaches.

Available Versions of this Item

• Combining Experts’ Judgments: Comparison of Algorithmic Methods using Synthetic Data. (deposited 09 Jul 2014 17:24) [Currently Displayed]
  • Combining Experts' Judgments: Comparison of Algorithmic Methods. (deposited 09 Jul 2014 17:34)