Data Compression Hackathon explores new approaches to reducing the size of weather and climate datasets
From 15 to 17 September, the ESiWACE3 Data Compression Hackathon brought together researchers and experts in Bonn, Germany, and online to explore how advanced compression techniques can help manage the growing volumes of data generated by weather and climate simulations.
The hackathon gathered around 40 participants, with approximately 20 attending in person and another 20 joining online. Participants came from Germany, Finland, Spain, the UK, the USA, Denmark, Iran and Slovenia,** bringing together expertise from across the weather, climate and high-performance computing communities.**
A key focus of the event was understanding the requirements and acceptable error bounds for lossy compression, with particular attention to the ERA5 dataset.
Discussions explored how much data can be compressed while still retaining the information and accuracy required for scientific applications.
One of the highlights was a dedicated session with ECMWF domain experts, which was very well received by participants. The session provided an opportunity to discuss the scientific requirements of weather and climate applications directly with domain specialists and led to a number of interesting discussions around compression quality and usability.
The participants then put different compression methods, compressors and configurations to the test. Their goal was to achieve the highest possible compression ratios while remaining within the required error bounds.
The results were very encouraging. Several of the compression ratios achieved during the hackathon were higher than initially expected, demonstrating the potential of advanced compression techniques to significantly reduce the storage and data-transfer requirements of large weather and climate datasets.
The hackathon provided a valuable opportunity to bring together the data compression, HPC and weather and climate communities, exchange expertise and explore practical approaches to one of the growing challenges of data-intensive Earth system modelling.
