In my view,
one of the merits of a Bayesian analysis is the opportunity to develop a prior
probability model using expert elicitation to express scientific knowledge. Expert
elicitation involves a carefully-crafted interview process
with a subject-area expert to translate an expert’s knowledge into a prior probability
for a Bayesian analysis. For the elicitation, a specific strategy is recommended
that involves: motivating, structuring, conditioning, encoding, and verifying:
· Motivating (establishing rapport) : This involves making sure that
the expert has a comfortable understanding of the process. This includes explaining the nature of
problem and analysis, giving the expert context on how his or her judgments
will be used, discussing the general methodology of a probabilistic assessment,
explaining heuristics expert can use, and identifying any potential
motivational biases.
· Structuring (defining uncertain quantities) : Once the expert is oriented as
to the general what, how, and why, the next step is to clearly define the
specific questions about which the expert will be providing judgment. During this step, it is important to define
variables of interest unambiguously and identify variable units as well as
possible ranges of values. Variables can be disaggregated into more elementary
variables, if necessary, or combined into summary variables, as appropriate.
· Conditioning (thinking about all evidence) : After the specific values to be
elicited are chosen, the expert should then be prompted to think about all of
his or her relevant expert knowledge concerning the variables and relationships
of interest. This knowledge could
include data, theoretical models, analogies with similar systems, or other
sources of understanding. The expert
should be encouraged to think from different perspectives and draw on as much
information as possible in order to overcome potential biases related to
consideration of limited scope. For
example, the elicitor can ask the expert to invent scenarios for extreme
outcomes and ask the expert to explain how these different outcomes could
occur.
· Encoding (quantifying expert judgment) : After proper preparation in
previous steps, this step comprises the actual elicitation. Probabilistic information can be elicited
according to many different proposed protocols, for example, the elicitor can
fix the probability and directly elicit the variable value or conduct an
indirect reference lottery.
· Verifying (checking the answer) : Finally, after all desired
probabilities are encoded, the elicitor should test the expert answers given to
see if they correctly capture the expert’s opinion. This can be done by rephrasing an expert’s
answer in another way to see if the expert still agrees with the
assessment. If the expert does not
confirm the answer, the elicitor may need to repeat conditioning and encoding
steps.
As an example, Reckhow (1988) used expert
judgment to improve a model of fish population response to acid deposition in
lakes when the knowledge of an expert is elicited and formally incorporated
into the model using Bayes Theorem. In Reckhow’s study, an expert (Dr. Joan Baker) in fish response to acidification was
interviewed to elicit a prior probability for the model parameters. The model was a logistic regression model with
the form
Since with a statistical model, scientific
experts are more likely to think in terms of the variables (pH, calcium, and
species presence/absence) rather than in terms of the model parameters, a
predictive distribution elicitation approach was used to determine the prior
probabilities. For this procedure, the expert was given a set of predictor
variables and then asked to give her estimate of the median response. A
frequency perspective was thought to facilitate response; thus a typical
question was: “Given 100 lakes that have supported brook trout populations in
the past, and if all 100 lakes have pH = 5.6 and calcium concentration = 130
ueq/L, what number do you now expect continue to support the brook trout
population?” This question was repeated 20 times with a variety of pH-calcium
pairs to yield 20 predicted responses. Twenty was chosen to provide some redundancy
to improve characterization of the prior yet not burden the expert with
time-consuming questions. The pH-calcium pairs were not randomly selected but
rather were chosen to resemble the sample data matrix.
The
expected response provided by Dr. Baker does not provide a crucial measure of
error. Thus, it was assumed that the errors in the conditional response were
approximately normally distributed, and additional questions were posed to Dr.
Baker to determine fractiles of the predictive distribution, conditional on pH
and calcium. A typical question was: “For pH = 5.1 and calcium = 90 ueq/L, you
estimated that 55 lakes supported brook trout populations. If the odds are 3:1
that the number of lakes (of 100 total lakes) supporting brook trout is greater
than a particular value, what is that value?” This question yields the 25th%,
and other similar questions provide other percentiles. These fractiles were
assessed for six conditional y [p(Presence)] distributions, producing
six estimates for standard error that are conditional on an assumed known underlying
variance (estimated from the data). A thorough description of this probability
elicitation and the complete Bayesian analysis can be found in Reckhow (1988).
Kashuba, Roxolana, McMahon, Gerard, Cuffney, T.F., Qian,
Song, Reckhow, Kenneth, Gerritsen, Jeroen, and Davies, Susan, 2012, Linking
urbanization to the Biological Condition Gradient (BCG) for stream ecosystems
in the Northeastern United States using a Bayesian network approach: U.S.
Geological Survey Scientific Investigations Report 2012–5030, 48 p. (http://pubs.usgs.gov/sir/2012/5030/.)
Reckhow, K.H. 1988. A
Comparison of Robust Bayes and Classical Estimators for Regional Lake Models of
Fish Response to Acidification. Water
Resources Research. 24:1061-1068.
No comments:
Post a Comment