Prometheus Metric Indicating the Time of Returning Experiment Verdict

[chirangaalwis]

Description:

As per evaluation of the LitmusChaos Exporter, it was noticed that the returned Chaos Result object Prometheus metric only has a property chaos_injection_time indicating the starting time of the chaos experiment.

This causes us to miss some experiments for which we may intend to get alerts based on the final verdict.

For example, if we create a check that periodically observes if there are any alerts (e.g. every 30mins) and if we use the chaos_injection_time for filtering the ChaosResults during the past 30mins, there maybe some experiments which start at one 30min block and end in the next. Since we are alerting every 30mins and the specific experiment which failed started in the previous 30min block, this will be filtered out.

IMO, there should be a metric that indicates the time of returning the experiment verdict rather than the starting time of chaos injection.

In an ideal solution, IMO the ChaosResult instance for a given ChaosEngine instance should be created at the point of returning the verdict rather than at the point of creating the ChaosEngine.