Read volume vars from file

[twsearle]

Description

We resolved #70 by reducing the memory usage of each process. Unfortunately, we found that the program spent most of its time in IO. This change reads the entirety of a volume variable in one go to reduce the number of file reads, rather than reading and broadcasting one horizontal level at a time.

Dependencies

Impact

The second cycle of the global ocean runs in MetOffice/sith/pull/271 should hopefully complete.

Checklist

• I have updated the unit tests to cover the change
• New functions are documented briefly via Doxygen comments in the code
• I have linted my code using cpplint
• I have run the unit tests
• I have run mo-bundle to check integration with the rest of JEDI and run the unit tests under all environments
• ostia configuration on spice
• ocnd configuration on spice
• gl_ocn configuration on spice
• ostia configuration on xc
• ocnd configuration on xc
• gl_ocn configuration on xc

[twsearle]

So far in my limited testing it appears this change might actually be making JOPA slower! I might withdraw this PR unless you can see anything wrong with my results. It is puzzling though, as I was pretty sure a change like this had delivered improvements in the past.

spice gl_ocn:

• spice gl_ocn on this ticket takes 955 secs with 4 MPI tasks
• spice gl_ocn on develop takes 835 secs with 4 MPI tasks

xc ocnd:

• xc ocnd on this ticket takes 2178 secs with 16 MPI tasks
• xc ocnd on develop takes 1497 secs on 16 MPI tasks

xc ocnd orca12:

• xc ocnd orca12 on this ticket takes 1502 secs on 12 MPI tasks
• xc ocnd orca12 on develop takes 1492 secs on 12 MPI tasks

There can be a lot of variability in runtimes on xc depending on the load of the file system I think, but still it seems that this change has either neutral impact or makes things worse.

[s-good]

It does seem like this change should speed things up!

This is a complete guess, but I'm wondering about the changes in src/orca-jedi/nemo_io/ReadServer.cc, where it is doing the distribution of data in memory by level previously and now all at once. Is it possible that there is some optimisation happening that makes the old version go faster? I.e. could it be fastest to read all data from file in one go and then distribute in memory by level?

[twsearle]

Closing this for now as we couldn't demonstrate any impact.