Can Artificial Intelligence Assess Reproducibility? A Proof-of-Concept Study in Ocean Science
编号:1618 访问权限:仅限参会人 更新:2026-09-02 16:51:28 浏览:0次 口头报告

报告开始:暂无开始时间(Asia/Shanghai)

报告时间:暂无持续时间

所在会场:[暂无会议] [暂无会议段]

暂无文件

摘要

Reproducibility is fundamental to scientific integrity, yet low levels of reproducibility have been demonstrated in many field of study. In ocean science, reproducibility is complicated by the use of large observational datasets, numerical models, remote sensing products, and complex computational workflows. Evaluating reproducibility at scale requires substantial time and expertise, creating a need for more efficient assessment methods. Recent advances in artificial intelligence (AI), particularly large language models (LLMs), offer new opportunities to automate aspects of reproducibility assessment. This proof-of-concept study investigates whether LLMs can be used to evaluate the computational reproducibility of ocean science publications. We developed a framework in which LLMs analyze published articles and associated reproducibility materials and assign a score using the 10-level Accelerating Computational Reproducibility (ACRe) framework. The ACRe framework evaluates the availability and completeness of data, code, documentation, and other materials required to reproduce published results. A sample of ocean science articles spanning multiple sub-disciplines, including physical oceanography, marine ecology, biogeochemistry, and remote sensing, is assessed. AI-generated scores are compared with human evaluations to determine the extent to which LLMs can reliably identify reproducibility evidence and apply the ACRe scoring framework. Particular attention is given to distinguishing between objective indicators, such as the presence of data repositories and source code, and more nuanced judgments regarding the completeness and usability of reproducibility materials. We hypothesize that LLMs will perform well when identifying explicit reproducibility indicators but less well when assessing whether available materials are sufficient to reproduce published findings. If successful, AI-assisted assessment could provide a scalable approach for screening large numbers of publications. Beyond research assessment, such tools could support the peer-review and publication process by assisting reviewers and editors in evaluating reproducibility, screening submitted manuscripts for compliance with journal requirements, and communicating reproducibility expectations to authors.

关键词
暂无
报告人
Anders Knudby
Professor University of Ottawa

稿件作者
Anders Knudby University of Ottawa
发表评论
验证码 看不清楚,更换一张
全部评论
重要日期
  • 会议日期

    01月12日

    2027

    01月15日

    2027

  • 07月21日 2026

    初稿截稿日期

  • 01月15日 2027

    注册截止日期

主办单位
State Key Laboratory of Marine Environmental Science, Xiamen University (MEL)
Department of Earth Sciences, National Natural Science Foundation of China (NSFC)
联系方式
历届会议
移动端
在手机上打开
小程序
打开微信小程序
客服
扫码或点此咨询