Add use of atlas float fields #79
Conversation
* Make reading of nemo data more robust to netcdf type * Add new option to set the atlas field type independently of nemo data * NOTE: due to limitations in `atlas::interpolation::NonLinear`, all fields must be the same type, and the configuration of missing values adjustment to the weights must include the fortran label of the type after the name (e.g "-real32"). * An atlas change is required to allow one run, one interpolator, with two fields of different type.
|
I have run a few experiments with this change, and I find that ocean profile runs will now run until running out of wallclock time instead of memory. This is an improvement but not yet conclusive evidence in support of this change.
For comparison, this is the memory profile for an orca025 run using floats:
|
|
I have made sure now that the MPI buffer is the same type as the netcdf data and atlas fields where possible. I am hoping this improves the performance a little. |
Great thanks! I am not sure that everything is working correctly yet, and I am also not sure if I am getting any performance improvements, but I wanted you to see what I was up to. We would also need to coordinate any change here with a change in jjdocs. |
|
I added an atlas issue based on this change, in case there is a better way of solving this problem, now or in the future with further atlas changes: ecmwf/atlas#173 |
|
For further info, I ran this change using a checkerboard distribution of the orca12 grid on 16 processors. It is clear that most of the memory usage now comes from data in the obspace which ought to scale much better (the stepped part at the beginning third is allocating the fields, with the later spike happening once obs-filtering begins and the various filters/variables are constructed). The runtime was also down to around half an hour (the graph is misleading as I am not logging anything towards the end of the run). However, this run is not accurate as the orca grid doesn't yet include halos, which are required for interpolation over the checkerboard distribution.
|
Description
Memory limitations on the HPC mean that it is difficult to fit the larger models with lots of variables into memory.
This change adds a configuration option to specify either float or double for the atlas fields. The default is also updated to float, which ought to approximately half memory use by orca-jedi. I have included additional logging to the trace output stream to track the memory usage over run time.
This change also includes making the reading of nemo data more robust to the underlying type of the netcdf data. There is probably a better way of managing this in future changes. Either using the type in the nemo file to determine the type of the atlas field, or avoiding reading all data in from the file -> std::vector buffer to mpi broadcast data to other processes -> filling the atlas field with specified type.
However, due to limitations in
atlas::interpolation::NonLinear, all fields must be the same type, and the configuration of missing values adjustment to the weights must include the fortran label of the type after the name (e.g "-real32"). An atlas change is required to allow one run, one interpolator, with two fields of different type.For now, in this change, I am keeping separate the concepts of the netcdf type of the nemo file variable and the atlas field datatype. Changes to the types used inside a run are controlled for now in two places:
Issues
Fixes #78
Dependencies
Mutually dependent on merge of MetOffice/jjdocs/pull/128
Checklist
Testing
Tested using adjusted apps from jjdocs/feature/atlas-field-float-precision and the MPI sith branch
Tests using the jjdocs branch and the develop branch of sith: