Volume rendering of cloud liquid content in a simulation of shallow cumulus clouds (BOMEX)

Large Eddy Simulations with ClimateMachine.jl

Climate models rely on parameterizations of physical processes whose direct numerical simulation (DNS) is infeasible because of its enormous computational cost. The accurate representation of unresolved processes below the grid scale of global climate models (GCMs), such as atmospheric turbulence and convection, is important for our ability to predict and understand climate. Large-eddy simulation (LES) frameworks are designed to allow studies of processes that are subgrid-scale in GCMs by examining smaller sections of the globe in greater detail. Processes whose relevant length-scales are smaller than the grid-scales in GCMs (typically tens to hundreds of kilometers) can be examined in greater detail in LES with grid resolutions of meters to tens of meters (Figure 1). With the goal of developing an accessible, performant high-resolution model of atmospheric phenomena, the CliMA team has developed a new large-eddy simulation (LES) framework called the ClimateMachine.

Volume rendering of cloud liquid content in a simulation of shallow cumulus clouds (BOMEX)
Figure 1: Instantaneous volume rendering of cloud liquid in shallow cumulus clouds (BOMEX), simulated using ClimateMachine LES. The domain has dimensions of 30 km x 30 km x 3.6 km, and is horizontally doubly periodic with grid resolutions of 50 m in the horizontal direction, and 20 m in the vertical direction. Variables are represented within each element with fourth-order polynomials.

ClimateMachine is an open-source, Julia language package that facilitates model prototyping and research with the same underlying code on both CPU and GPU hardware. One of the core components of this framework is a partial differential equation solver using the discontinuous Galerkin (DG) method for the governing conservation equations for mass, momentum, energy, and total water content of air. Essentially, any limited-area on the globe is sub-divided into a series of elements, with flow variables in each such element represented by a polynomial approximation to the true fluid system. The exchange of these quantities across elements is then modeled through numerical fluxes. Properties of the DG discretization facilitate parallel computation with relative ease, as overlap between computation and communication loads are naturally achieved by splitting the physical terms into those that can be computed locally within elements (volume integrals), and those that require information exchange across neighboring elements (surface integrals).

Figure 2: Horizontally averaged statistics of key flow parameters, with comparisons using two different subgrid-scale turbulence models (Vreman and Smagorinsky-Lilly (SL)) in ClimateMachine, against simulations performed with PyCLES, a Python based LES framework used by the Climate Dynamics Group at Caltech. Variables, from left to right, include liquid potential temperature, specific total moisture, cloud fraction, cloud liquid, and vertical velocity variance as measures of thermodynamic and turbulent flow quantities.

In our GMD pre-print, we demonstrate the utility and fidelity of the ClimateMachine LES through a series of benchmark problems, including that of atmospheric turbulence and shallow cumulus convection. By design, ClimateMachine supports modules that permit physical models to be added or replaced with relative ease, facilitating comparative studies of flow physics. For example, we demonstrate the use of Julia’s multiple-dispatch paradigm by applying different models to represent the energy cascade in small scale flow turbulence (Figure 2). The framework supports multiple options for the numerical time integration. We also examine the scaling characteristics of this LES framework, noting excellent weak scaling on up to 16 GPUs on the Google Cloud Platform (Figure 3), with greater than 95% weak scaling for the BOMEX shallow-cumulus convection test problem. We also note a factor 19.7 speedup when considering CPU strong scaling over a 32 rank simulation (Figure 4).

Figure 3: Weak scaling of ClimateMachine LES (shallow-cumulus BOMEX test problem). Both semi-implicit (red) and explicit (blue) timestepping methods are examined, and we show that this LES framework is suitable for parallelized solutions of problem setups with larger spatial extents while maintaining comparable grid resolutions for the corresponding single GPU runs.
Figure 4: Strong scaling of ClimateMachine LES (rising thermal bubble test problem). We examine results in both dry (blue) and moist (red) atmospheric configurations to examine the effects of additional equations in the PDE system. In the moist system, an iterative solution (“saturation adjustment”) is required to determine the phase partitioning of the total moisture content in the physical system.

The ClimateMachine LES allows us to generate high-resolution datasets with which parameterizations in GCMs can be improved and trained using data assimilation and machine learning methods. Reduced uncertainties in parameterizations are critical for our goal of improved climate predictions, to provide accurate and actionable information for climate change mitigation and adaptation strategies. The ClimateMachine LES framework is a step toward our longer-term goal of providing seamless modeling framework that can go from coarser-resolution GCMs to higher-resolution LES, can exploit modern computing hardware including accelerators, and uses data assimilation and machine learning methods to inform a unified “virtual-earth” model. The team’s current efforts include the development of a new Julia dynamical core, ClimaCore, and several physics modules to capture phenomena critical to our understanding of current and future climate states.