Can we rely on Kafka file-writer messages for the live data reduction?

[jokasimr]

Me and @YooSunYoung talked with @g5t (Greg) about live data reduction and kafka messages.
He mentioned that it's likely that he as a user might want to run the instrument and view the live data reduction while not producing a file that is stored in Scicat.
I don't know enough to say if this is even possible, or if it even should be possible (I think it should not ;)). But since our live reduction approach will likely use file writing kafka messages such as pl72_run_start.fbs "File writing, run start message for file writer" to initiate the live data reduction, it seems like something we should investigate.

In short, are the kafka messages that we depend on to do the live data reduction always emitted regardless of if the run is written to a file or not?

[jokasimr]

Slightly unrelated, but from talking with Greg we also got the impression that the nexus-json-templates have some optional / empty fields that might or might not be populated depending on the mode of the run. Odin was mentioned as one instrument that has several modes and where the file will have fields that will be used only in some modes. In that case I guess the user will have to specify the mode that they are running the instrument in before they start the live data reduction view.

[SimonHeybrock]

Me and @YooSunYoung talked with @g5t (Greg) about live data reduction and kafka messages. He mentioned that it's likely that he as a user might want to run the instrument and view the live data reduction while not producing a file that is stored in Scicat. I don't know enough to say if this is even possible, or if it even should be possible (I think it should not ;)).

That will be ECDC's decision. I don't think it is.

But since our live reduction approach will likely use file writing kafka messages such as pl72_run_start.fbs "File writing, run start message for file writer" to initiate the live data reduction, it seems like something we should investigate.

In short, are the kafka messages that we depend on to do the live data reduction always emitted regardless of if the run is written to a file or not?

Beamlime will subscribe to the Kafka topic, so it should get messages. Not sure I understand the question.

[jokasimr]

Beamlime will get the messages if they're emitted. Are they? Or are they only emitted when the user decides to write a file? Do we want users to be able to use beamlime also when they're not writing a file?

[SimonHeybrock]

File writing is completely unrelated to Beamlime. The file writer will subscribe to the same Kafka topic, that's it.

[jokasimr]

From yesterdays talk I got the impression that they're not totally unrelated. For example, that the run_start message is only emitted when a file is to be written.

[g5t]

If I understand correctly, during the intended normal-operation mode NICOS will emit a pl72_run_start, likely using file_writer_control.WriteJob or similar, at the start of any data acquisition period. At end of the counting/scanning/acquiring period it will use, e.g., file_writer_control.JobHandler, to update the write-job end time by sending a 6s4t_run_stop message.

What I mentioned yesterday is that these messages are not required to coincide with the start and stop of the acquisition period. If a user decides, after the fact, e.g., that the wrong NeXus Structure was used to write a file; a new file with the 'correct' data can be written by sending a new pl72_run_start message (likely with both start and end times specified).

If you accept every pl72_run_start message on the Kafka topic, you may reconfigure beamlime for streaming the wrong data.
To avoid this you could handle only messages that are produced by NICOS, but only if it isn't used to write 'historical' data.

[g5t]

He mentioned that it's likely that he as a user might want to run the instrument and view the live data reduction while not producing a file that is stored in Scicat.
— @jokasimr

I still think that a user might want this, but your suggestion that it may not be possible is both valid and probably right.
In such a case, I would hope that a user can set a metadata flag before or during a data-acquisition that indicates, e.g., 'this is an alignment scan, hide it in SciCat'; but that is a separate issue.

[YooSunYoung]

Conclusion

We had a brief meeting with ECDC on the 08-03-2024 (Fri).

In the current status, starting instrument will always trigger file-writing anyways but this behaviour may be changed in the future.
Currently, there is no other way to obtain the information that run_start_message carries, e.g., nexus_structure, without triggering filewriter.
So users(especially super-users) will have to trigger the filewriter in order to use the live-data-reduction.
Therefore we asked ECDC to provide an indicator in the file or in the Scicat metadata so that users(including superusers) can filter the files that were not meant to be stored.