DAVID SCHULKER, JOSHUA WILLIAMS, CHERYL K. MONTEMAYOR, LI ANG ZHANG, MATTHEW WALSH
The Personnel Records
Scoring System
Volume 3, A Methodology for Designing
Tools to Support Air Force Human
Resources Decisionmaking
T
he U.S. Department of the Air Force (DAF) has begun to develop and field artificial intel-
ligence (AI) and machine learning (ML) systems for myriad mission areas and support
functions, including human resource management (HRM). This report describes an ML
decision-support tool to summarize the information in officer performance reports (OPRs)
and other narrative-style documents to help the HRM system make personnel decisions more effec-
tively, more efficiently, and in better alignment with the DAF’s strategic goals.
Background
The latest data from the McKinsey
Global Survey on Artificial Intelligence
show that private-sector companies
have continued the march toward
greater adoption of AI, especially
for optimizing services or enhanc-
ing product offerings (Chui et al.,
2021). However, the same survey
shows that AI adoption in the field
of human resources (HR) is still rela-
tively rare. Further, the percentage of
companies that use AI to optimize
talent management processes, such
as those associated with recruiting or
retention, declined from 10 percent
to 8 percent between the 2020 and
KEY FINDINGS
■ Department of the Air Force analysts can rapidly develop simple
models relating key text in officer records to past decisions. The
most-accessible approaches break the text into individual terms,
index the records according to which terms they contain, fit a
predictive model of the past decisions, and then create decision
inputs from the models. We demonstrate these steps through our
development process for PReSS.
■ The constrained language used in officer performance
reports makes them amenable to natural language processing
approaches, as shown by the fact that simple models with mini-
mal preprocessing and tuning achieved high levels of accuracy.
■ As compared with state-of-the-art machine learning approaches
(i.e., deep learning), simple linear models based on the presence
or absence of key terms achieve similar levels of predictive perfor-
mance but have the advantage of being inherently interpretable.
■ Key words and phrases that models base predictions on coincide
with statements recognizable to expert raters.
Research Report
