Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retire the #dev-ops-notifications channel #347

Open
jchristgit opened this issue Jun 6, 2024 · 1 comment
Open

Retire the #dev-ops-notifications channel #347

jchristgit opened this issue Jun 6, 2024 · 1 comment
Labels
component: monitoring An issue relating to a monitoring component (e.g. Prometheus, Grafana) group: kubernetes Issues and pull requests related to the Kubernetes setup

Comments

@jchristgit
Copy link
Member

Our #dev-ops-notifications channel is not being reviewed actively, and
presumably has not been for a long time. We should remove it.

The reasoning is relatively clear. When looking into the channel, we find a
large amount of almost completely unactionable problems with absolutely no
cross-referencing or other useful information. An issue was made to improve
this situation in Olli in August 2021, but has received no traction, see
python-discord/olli#12.

To further drive the point home, over the past few days, I have added a webhook
that periodically sends defamatory statements whilst impersonating Joe, and
nobody has complained yet. I therefore ask we remove the channel.

@jchristgit jchristgit added component: monitoring An issue relating to a monitoring component (e.g. Prometheus, Grafana) group: kubernetes Issues and pull requests related to the Kubernetes setup labels Jun 6, 2024
@jchristgit jchristgit moved this from Up next to Backlog in Infrastructure Jul 24, 2024
@MarkKoz
Copy link
Member

MarkKoz commented Aug 17, 2024

Let's open a discussion about what this channel was trying to achieve. Evidently it has failed, but were we trying to resolve real problems at the time? Have these problems since been solved by other means?

I am 100% on board with only showing actionable items when it comes to operational alerts. However, there is a grey area there. For example, let's say we have a hard dependency on a third party service. If that service experiences downtime, it is likely out of our control and there is nothing immediately actionable. But we still want visibility into it, so that we can assess impact and notify users downstream. And in the long term, such things can spark discussions like "Why do we have this dependency? Is there an alternative?" This is just one example from my experience that shows how it can be tricky to set the bar for alerting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: monitoring An issue relating to a monitoring component (e.g. Prometheus, Grafana) group: kubernetes Issues and pull requests related to the Kubernetes setup
Projects
Status: Backlog
Development

No branches or pull requests

2 participants