Cylc ("silk") is a Python based workflow engine and meta-scheduler – that is, Cylc will manage a set of dependant tasks that need to run in a given order and deal with exceptions. It specialises in continuous workflows of cycling tasks such as those used in weather and climate forecasting and research (i.e. tasks can repeat at particular time intervals and can be triggered off the wall clock time as well as other tasks). Cylc is also easy to use with non-cycling systems. Cylc was created by Hilary Oliver at NIWA. Its core team now includes Hilary as well as members in the Modelling Infrastructure Support Systems Team at the Met Office. Cylc is free software under the GNU GPL v3 license.
Cylc was developed as a generic tool to help with the increased complexity of workflows used in the weather and climate communities to replace the increasingly complex script-based solutions typically used within the community. Cylc is currently being used for a very wide range of requirements from research to real-time operations including ensemble prediction systems. It is used for a wide variety of workflows such as generation of input data, assimilation of observational data, modelling post processing and commercial product generation. Cylc’s portability has been proven by its deployment on Linux/Unix platforms at a number of institutions.
Cylc was a major topic for discussion at the IS-ENES2 Workshop on Workflow Solutions in Earth System Modelling with a number of different centres already using, evaluating or planning to evaluate Cylc. As a result, the Met Office has received funding via ESiWACE (Centre of Excellence in Simulation of Weather and Climate in Europe) for the next 3 years to continue development of Cylc to better meet the needs of an expanding user base and to support users evaluating or adopting Cylc.
A further IS-ENES2 Workshop on Workflow Solutions is planned for September. Anyone interested in attending should watch out for an announcement on this web site.
IS-ENES2 Workshop on Workflow Solutions : https://is.enes.org/events/isenes2-workshop-on-workflows
ESiWACE Cylc support and development plan: https://www.esiwace.eu/results/milestones/ms-1-report/view