published Jan 15, 2025

15.01.2025

As part of the ESiWACE3 Centre of Excellence, our researchers have developed the second public version of the Online Laboratory for Data Compression tailored to climate science and meteorology. This initiative is complemented by the release of specialised datasets designed to test various compression methods on outputs from both weather and climate models.

The laboratory includes a collection of Jupyter notebooks that allow users to:

  • Explore datasets for medium-range weather predictions and climate simulations.
  • Apply both lossless and lossy compression techniques through an intuitive API.
  • Perform diagnostics to evaluate potential degradations in the information content of data resulting from lossy compression.

Additionally, the platform provides the more broadly-applicable Online Laboratory as a Service, enabling users to access and utilize the compression tools or run arbitrary Python code without the need for local installation or setup, as popular earth science Python packages come pre-installed. Users can seamlessly work with datasets generated from prominent models, including the IFS, OpenIFS, and ICON models, which are available for exploration within the laboratory or for direct download via an S3 bucket.

In the coming two years, the team will actively engage the community through outreach initiatives, including a dedicated hackathon to explore compression techniques in depth. The project will also focus on documenting trade-offs among key factors, such as the reduction in storage requirements, the computational cost of compression and decompression or the impact of compression errors on subsequent diagnostics.

You can find the Online Laboratory here: