Proposed changes to the grey list

[apswong]

I propose to make two changes to Argo RTQC Test 15 - the grey list test.

1. AST-25 action: AST asks ADMT to rename the grey list to remove any reference to a color

I suggest changing the name of the test to "Supplementary Sensor Failure Test". Does anyone else have other suggestions?

2. Withhold the grey list from external users

Making the grey list available to external users is a legacy from the old TESAC days. We are now in the BUFR era, when QC flags are sent to the GTS, and when we are actively encouraging users to use our QC flags. It is time to clean up this legacy issue, withhold the grey list from external users, and keep the grey list for internal use within Argo Data Management only. What are your thoughts?

[DebWest]

The name describes it well, but it is a mouthful. Can we come up with a nickname so it is easier to say? Officially it can be the 'Supplementary Sensor Failure Test', but with a short name of 'FS list' or 'failed sensor list'.

[tlmaurer]

1. I like Deb's suggestion. "Supplementary Sensor Failure Test" and the associated list would be referred to as "failed sensor (FS) list".
2. This makes sense to me, in the context of our system anyway. We have an internal list registry for failed BGC sensors, and we have not been concurrently updating (what's currently referred to as) the Argo 'grey list'. It is somewhat burdensome to add to two lists. The one way I could see a shared registry being useful is to track failed sensor statistics across the array, but that is currently done (in an ad-hoc sense) by assessing flags (and this approach is maybe more comprehensive anyway?).

[hristinagh]

Delayed mode-operators and PIs are supposed to update the greylist as needed. As someone that got involved in DMQC only recently I find the guidelines on how to do that and how often vague and unclear (or I am not aware of the right ressource). Clear instructions would simplify the process for all parties involved.

[coatanoan]

A clear process is to simply pass on to the relevant DAC manager the information to put such and such a float on the 'greylist' and provide the information as described in the Argo Quality Control Manual for CTD and Trajectory Data (https://doi.org/10.13155/33951), test 15 page 15.

[fionacarse]

Hi, I've checked back through my emails about grey list matters. The operational ocean forecasting system here at Met Office no longer uses the grey list (email dated August 2023). The producers of the EN4 product do use it. But I can easily simply tell them it has changed its name. Would it be helpful at this time if I found out exactly how they access it? Cheers
Fiona

[mscanderbeg]

Thanks for checking @fionacarse. I think the proposal is to actually not make the new list public, so EN4 would need to adjust their workflow if we did this. It would be very helpful if you could find out more about how they use the greylist. thank you.

[fionacarse]

Thanks Megan. I have asked Rachel Killick and will let you know what she says (she's on leave this week).

[apswong]

@fionacarse @mscanderbeg I am in communication with the EN4 team (Rachel Killick) and a few other big analysis centers. I will get feedback directly from them on this proposal. I do not want this proposal to be misrepresented, and it's best if we do not duplicate our communications. Thanks.

[cgourcuf]

I agree with both propositions. 2 comments:

1. The new name for the greylist is a bit long but it has the merit to be precise, and if we keep the list internal it will have limited impact. I like the nickname idea, but would suggest 'SSF', to be consistent with the official 'Supplementary Sensor Failure' name.
2. We need to ensure a clear communication with potential external users still using the greylist.

[apswong]

@cgourcuf Thanks for your comments. As I replied earlier to @fionacarse and @mscanderbeg , I will get feedback directly with several external users on this proposal. If this proposal goes ahead and becomes an action at ADMT, we can then discuss how to communicate this change.

[fionacarse]

Hi @apswong , sorry for duplication! I will let Rachel know you will be in touch with her directly.

[coatanoan]

For the next ADMT, I also have an action related to a new list of doubtful floats that are not grey-listed “Make available the list of doubtful floats from the min/max test that were not grey-listed. Advertise this as an additional list to be used with all QC test results in Argo data files.”
So I have started work and can propose an index file ar_minmax_fs.txt to stick with the same idea as the new fs name for failure sensors (see https://data-argo.ifremer.fr/etc/ https://data-argo.ifremer.fr/etc/ar_minmax_fs.txt). The format is very similar to that of the 'ar_greylist.txt' file. The floats inside are those that have not been added to the grey list and are still with an uncorrected QC in the Argo data flow. I propose to update this list each month at the same time as I provide the monthly report, which means that I will remove floats that have been added to the ‘grey list test’. However, if a new float is quickly detected as drifting, I will automatically add it in real time.
@apswong , since you are in communication with the teams that use the data for their models, I will let you talk with them to see what is the best way for them to obtain this information.

[apswong]

After the ADMT-25 meeting in Trieste, and discussions with @coatanoan and @tcarval, here are the conclusions.

1. Argo RTQC Test 15: "Grey List Test", will be renamed as "Supplemental Sensor Exclusion List Test". This is in response to a request from the AST to remove any mention of color in the name. The short name is the "exclusion list", as suggested by Dirk. Please note that this is just a name change; the workings of RTQC Test 15 stay the same.

Actions:

• Annie will change the name of RTQC Test 15 in the next update of the QC manual.
• The name entry in R11 for Test 15 should also be changed.

2. The aggregated grey list file, "ar_greylist.txt", which is made by the GDACs and is available to the general public, will be removed from the GDACs from public access. This is a legacy from the TESAC era and should be cleaned up. The proposed removal date is sometime in June 2025 to facilitate the transition of several super users.

Actions:

• Thierry & Mike (GDAC managers) and Annie to fix a removal date in June 2025, and communicate this to all Argo data users with ADMT co-chairs.
• Thierry & Mike (GDAC managers) to proceed with the removal of "ar_greylist.txt" from the GDACs on the agreed date in June 2025.

3. After the removal date in June 2025, to facilitate internal users within the ADMT and OceanOPS who use "ar_greylist.txt", Coriolis will make an aggregated "exclusion list", with warning in the header that this is not a comprehensive list of failed sensors, and should only be used by experts with full knowledge of its limitations. @tcarval and @coatanoan will decide where in Coriolis to put this aggregated "exclusion list".

Actions:

• After the "ar_greylist.txt" removal date in June 2025, DACs will continue to send their internal "exclusion list" to Coriolis. Coriolis will make an aggregated "exclusion list". Thierry and Christine will put this in an area in Coriolis (to be determined), with warnings.