7 Expert judgement

Obtaining expert opinion in a structured way — expert elicitation (EE) — is a common approach if environmental data are poor or unattainable. EE is a powerful tool but its efficacy depends on careful consideration of the process and factors that influence judgement elicitation.

CEBRA combined different elicitation strategies into a single comprehensive procedure and provided training to a range of professionals.86 Results from this project were incorporated into a book on expert judgement87 and the tool is being adopted by a range of disciplines internationally. The department used the EE procedure to estimate inputs to the Risk-Return Resource Allocation model. It has also been used as part of other CEBRA projects. For example, in a novel application of the method, for the risk assessment of pest incursion through vessel biofouling, CEBRA asked experts to rate the relative risk of different scenarios leading to biofouling of a vessel’s hull. The method proved suitable for ranking relative risks and presents an advance in an area where the use of process-based models has not been feasible. It is an expert-based model for evaluating biosecurity risk (likelihood of marine pest establishment) based on vessel biofouling management, history of recent hull survey vessel movement history and the characteristics of Australian ports. Results from this project have been incorporated into departmental protocols.88

CEBRA reviewed and evaluated existing methods for extreme risk analysis used in a range of disciplines, including elicitation of expert opinion. It concluded that evaluations of extreme risks should be supported by quantitative analysis, even in data poor environments, but that the most effective strategies give preference to qualitative reasoning in final conclusions.89 Experts disagree about various scientific matters. In this context, CEBRA reviewed formal consensus methods which aim to settle disagreements among experts. The project applied the formal methods to a plant pest management problem in a case study. Project outputs categorised issues with formal group aggregation methods to allow decision-makers to choose the appropriate method for a particular context. Also, as part of the same broader project, CEBRA investigated approaches for representing uncertainty, which is a part of risk analysis. CEBRA provided theoretical reasoning for the argument that non-statistical uncertainties are credible sources of uncertainty and for the use of non-probabilistic methods to represent uncertainty instead of relying on probability theory to do that.90,91,92

The import of products into Australia can pose a biosecurity risk. CEBRA developed a new method that supports the iterative process of modelling of incursion risk and illustrated the approach in an import risk assessment case study. The method includes the use of EE and allows experts to re-evaluate their opinion after seeing the likelihoods generated by the model, which used their opinion as an input.93 CEBRA also developed an approach, based on expert judgements, that overcomes difficulties around the construction of scoring systems for risk analysis.94

Birds Australia adopted CEBRA’s expert question procedure for estimating threat. The procedures are applicable to a wide range of circumstances.


86. Speirs-Bridge, A. (2010). Eliciting expert judgements: Process Manual. Australian Centre of Excellence for Risk Analysis, report 0611.

87. Burgman, M. A. (Cambridge University Press, 2015). Trusting Judgements. doi:10.1017/cbo9781316282472

88. Barry, S. et al. (2015). Development of a marine spatial analysis model for improved biofouling risk assessment. Centre of Excellence for Biosecurity Risk Analysis, report 1402A.

89. Franklin, J. (2007). Assessment of strategies for evaluating extreme risks. Australian Centre of Excellence for Risk Analysis, report 0602.

90. Colyvan, M., Burgman, M., Lyon, A., Regan, H. & Steele, K. (2009). Evaluation and development of formal consensus methods. Australian Centre of Excellence for Risk Analysis, report 0607 ID3.

91. Colyvan, M. (2008). Is probability the only coherent approach to uncertainty? Australian Centre of Excellence for Risk Analysis, report 0607 ID1.

92. Steele, K., Colyvan, M., Regan, H. & Burgman, M. (2008). Survey of group ’consensus’ methods. Australian Centre of Excellence for Risk Analysis, report 0607 ID2.

93. Kuhnert, P. & Barry, S. (2011). Bayesian Learning and Synthesis through the Elicitation of Risk: BLASTER. Australian Centre of Excellence for Risk Analysis, report 0705D.

94. Barry, S. & Lin, X. (2011). Point of truth calibration: Putting science into scoring systems. Australian Centre of Excellence for Risk Analysis, report 0705E.