[PROPL'24] The programming challenges of climate data analysis

Ezequiel Cimadevilla

The ongoing evolution of the climate system and assessment of climate change requires sophisticated tools and methodologies for the analysis of vast and complex datasets generated by climate models, emerging records of satellite data and observational datasets. The pipelines and workflows involved in this complex task are performed in several phases that include multi-year and international planning, acquisition of data from heterogeneous data sources, complex infrastructures supporting the distribution of information and global collaborations for evaluation and archival. This work provides an update on concrete state-of-the-art methods currently used in data analysis workflows of climate data generated by Global Climate Models, focusing on the activities of the Climate Model Intercomparison Project (CMIP) and its underlying data infrastructure, the Earth System Grid Federation (ESGF). The presentation will dive into different topics of interest for practitioners and software developers of climate data analysis tools, including an overview of the software libraries from the Python ecosystem used for multidimensional climate data analysis and storage, a formal definition of climate data in the scope of the relational data model and an overview and description of requirements of different storage systems (HPC and cloud) used by the climate community.