报告开始:暂无开始时间(Asia/Shanghai)
报告时间:暂无持续时间
所在会场:[暂无会议] [暂无会议段]
暂无文件
Reproducibility is fundamental to scientific integrity, yet low levels of reproducibility have been demonstrated in many field of study. In ocean science, reproducibility is complicated by the use of large observational datasets, numerical models, remote sensing products, and complex computational workflows. Evaluating reproducibility at scale requires substantial time and expertise, creating a need for more efficient assessment methods. Recent advances in artificial intelligence (AI), particularly large language models (LLMs), offer new opportunities to automate aspects of reproducibility assessment. This proof-of-concept study investigates whether LLMs can be used to evaluate the computational reproducibility of ocean science publications. We developed a framework in which LLMs analyze published articles and associated reproducibility materials and assign a score using the 10-level Accelerating Computational Reproducibility (ACRe) framework. The ACRe framework evaluates the availability and completeness of data, code, documentation, and other materials required to reproduce published results. A sample of ocean science articles spanning multiple sub-disciplines, including physical oceanography, marine ecology, biogeochemistry, and remote sensing, is assessed. AI-generated scores are compared with human evaluations to determine the extent to which LLMs can reliably identify reproducibility evidence and apply the ACRe scoring framework. Particular attention is given to distinguishing between objective indicators, such as the presence of data repositories and source code, and more nuanced judgments regarding the completeness and usability of reproducibility materials. We hypothesize that LLMs will perform well when identifying explicit reproducibility indicators but less well when assessing whether available materials are sufficient to reproduce published findings. If successful, AI-assisted assessment could provide a scalable approach for screening large numbers of publications. Beyond research assessment, such tools could support the peer-review and publication process by assisting reviewers and editors in evaluating reproducibility, screening submitted manuscripts for compliance with journal requirements, and communicating reproducibility expectations to authors.
01月12日
2027
01月15日
2027
初稿截稿日期
注册截止日期
2024年12月11日 中国
第七届厦门海洋环境开放科学大会(XMAS 2025)2023年01月09日 中国 Xiamen
第六届厦门海洋环境科学开放大会
发表评论