NSIA Scholarship in Action: Faculty鈥揂lumni Research Supported by ODNI
NSIA Scholarship in Action: Faculty鈥揂lumni Research Supported by ODNI
Dr. John Borek is a faculty lecturer in the , and Seth Marcotte is a recent NSIA graduate. Together, they were published in the International Journal of Intelligence and CounterIntelligence.
Supported by the Office of the Director of National Intelligence (ODNI), 鈥淎n Evaluation of Large Language Models Using AnalyticTradecraft Standards,鈥 by John Borek and Seth Marcotte reflects the strength of the National Security Intelligence Analysis program at the 糖心视频. It also demonstrates the sustained connection between faculty, students, and graduates.
, the authors evaluate the output of Google鈥檚 Gemini and Microsoft鈥檚 Copilot against ODNI analytic tradecraft standards the ODNI Rating Scale for Evaluating Analytic Tradecraft Standards, and compare those results to human analysts.
Their evaluation found large language models struggled with tasks requiring disaggregation of information, particularly describing source quality and credibility, expressing uncertainties, and distinguishing between facts, assumptions, and judgments. The models achieved highest ratings on communication-related tasks including analysis of alternatives, relevance and implications, and clear and logical argumentation.