RESEARCH POSTER PRESENTATION DESIGN © 2015
www.PosterPresentations.com
Traditional reliability methods are increasingly
challenged in developmental testing (DT):
increasingly complex systems and limited test time.
Bayesian methods can combine data consistently
across test segments and include additional
information beyond the test.
Given their promise, Bayesian reliability methods
could be more widely applied.
Study goal: provide a set of recommendations to
assist the practical use of Bayesian reliability
methods in DT.
Motivation
Bayesian Reliability Analysis
Subject Matter Expert (SME) Interviews Best Practices Summary
Selected References
Contact
Institute for Defense Analyses
Paul Fanto
David Spalding
Best Practices for Using Bayesian Reliability Analysis in Developmental Testing
Bayesian reliability analysis combines prior
information about the system with test data to
obtain a probability distribution for a reliability
metric, e.g., a system failure rate.
E.g., Bayesian analysis provides a more precise
estimate of failure rate in notional system with
limited test data.
The study consisted of interviews with 12 SMEs at
IDA and in Service test organizations.
Key Lessons Learned
Reliability analysis should be focused on reducing
program risk.
o It is important to make reliability more
than a “box-checking exercise.”
o Reliability assessments should highlight
the impact of reliability shortfalls on
mission success.
Reliability analysts should attempt to be involved in
the program as early as possible
o Early involvement provides analysts with
necessary background.
o Early involvement enables analysts to
influence test planning and data storage
strategy, which will benefit future
analysis.
Useful analytic tools are straightforward to
understand and communicate.
o Any complex method, e.g., Bayesian
analysis, should be used only if it provides
a clear benefit over more standard
methods.
The experience and knowledge of the service test
organizations is valuable.
o Army: U.S. Army Combat Capabilities
Command, Data Analysis Center; and the
Army Test and Evaluation Command (ATEC)
o Navy: Operational Test and Evaluation Force
(OPTEVFOR)
o Air Force: Air Force Operational Test and
Evaluation Center (AFOTEC)
o Marines: Marine Corps Operational Test and
Evaluation Activity (MCOTEA)
o OSD/DTE&A: STAT Center of Excellence
(STAT COE)
o Caveat: these organizations are often
focused on OT rather than DT.
Overarching perspective: the goal of reliability
analysis is to help DT programs make more
informed decisions within a complex and
constrained test environment.
Many of these best practices are common sense to
experienced analysts.
Working with the Decision-Making Environment
1. Explain how reliability impacts program goals,
e.g., mission, cost.
2. Get early visibility into programs.
Determining Analytic Goals
1. Follow a risk reduction approach.
2. Access government expertise, e.g., OTAs.
Choosing a Bayesian Method
1. Use Bayesian method only if it provides a clearly
identifiable benefit over standard methods.
2. Access government expertise, e.g., OTAs.
Applying the Model
1. Exploit existing software tools.
2. Focus on credibility of the prior distribution.
3. Communicate results in relatable terms.
This study identified best practices to guide
the analyst in the application of Bayesian
reliability analysis in DT programs.
The study combined SME interviews with a
review of Bayesian models in the literature
to identify best practices.
The practices focus on enabling the analyst
to inform relevant decisions in a complex
and constrained DT test environment.
1. M. Ambroso, A. Kelley, and A. Wilson. Reliability Basics: Key
Reliability Concepts for DT&E. IDA Paper NS-P-4925 (2013).
2. M. Wayne and M. Modarres. A Bayesian Model for Complex
System Reliability Growth Under Arbitrary Corrective
Actions. IEEE Transactions on Reliability 64, 206 (2015).
3. M. Wayne. Modeling Uncertainty in Reliability Growth
Plans. 2018 Annual Reliability and Maintainability
Symposium (RAMS).
4. R. M. Dickinson, L. J. Freeman, B. A. Simpson, and A. G.
Wilson. Statistical Methods for Combining Information:
Stryker Family of Vehicles Reliability Case Study. Journal of
Quality Technology 47, 400 (2015).
Dr. Paul Fanto, pfanto@ida.org
Dr. David Spalding, dspaldin@ida.org
CLEARED
For Open Publication
Department of Defense
OFFICE OF PREPUBLICATION AND SECURITY REVIEW
