Facilitate access to S3 buckets

[jfigui]

Reminder that at some point we would like to be able to work directly with data stored in S3 buckets

[jfigui]

Hi @wolfidan,

I have pushed a first version on commit b4187d8 in dev.

Please have a look at it and let me know what you think.

[jfigui]

Hi @wolfidan ,

I pushed several enhancements in dev.

You can now work with data stored in s3 buckets for data types "ODIM", "ODIMBIRDS", "CFRADIAL", "CFRADIAL2", "CF1", "NEXRADII", "GAMIC", "ODIMGRID", "KNMIH5GRID" and "SKYECHO".

I had to do a significant revamping of read_data_radar.py but I think that with what I have done we are on path to simplify our data reading functions.

Please have a close look to what I have done and let me know if you find any issues. It would be good to be sure that I did not break anything before pushing to the master and I had a limited time for testing.

[wolfidan]

Hi @jfigui ,

Looks great, thanks a lot for your work, I didn't get the time today, but I will do it tomorrow for sure,

[wolfidan]

At the moment, the tests fail because compatibility is broken in https://github.com/MeteoSwiss/pyrad/blob/dev/src/pyrad_proc/pyrad/io/io_aux.py get_file_list
The issue is that the indexing cfg["datapath"][ind_rad] is not done correctly for some cases, for example at l.2984 we have datapath = (
f'{cfg["datapath"]}{dayinfo}/{basename}/')
since cfg["datapath"] is a list this ends up in a wrong file pattern.

Another issue is that you try to use cfg["DataTypeIDInFiles"] in the nexrad reader. I don't think this works. I would suggest removing it.

[wolfidan]

Please also think about adding the necessary info in https://meteoswiss.github.io/pyrad/overview/main.html by editing https://github.com/MeteoSwiss/pyrad/blob/master/doc/source/overview/main.rst

[jfigui]

Thanks for the feedback.

I corrected some bugs and it passes the ubuntu tests now but not the windows one. It fails when setting up miniconda. I do not see how this can be related to the changes in the code. Can you have a look?

[wolfidan]

Hi @jfigui , thanks, sorry I am taking some vacation, but I will try to look into it asap

[wolfidan]

Managed to make the tests run

[jfigui]

Great! Thanks a lot! Let's keep this issue open until we can use s3 buckets with any file.

[wolfidan]

Closing as taken care of by 4f6d353 and (4f6d353)