metrics for the standard library

[jba]

The runtime/metrics package exposes wealth of detailed information about the Go runtime. Would other packages in the standard library benefit from their own metrics? What would be useful to know about the internal state, of, say, the net or net/http packages?

[niiad]

for the net or net/http packages, it will be useful to know about the number of incoming requests, request rates, response times, maybe insight about the connection and memory usages, then error metrics as well.

[deefdragon]

Some of those could be broken down by response code, and having access to that info might be helpful as well. (ie 500's p99 etc.)

[golightlyb]

A lot of these metrics could be captured at a HTTP middleware layer. If the net/http modules themself provide metrics I think it should probably be ones that must be done there (e.g. memory, fine grained timings) and others implemented in a separate module as middleware.

[LinuxSploit]

Connection Statistics: For the net package, it would be beneficial to have metrics related to the number of open connections, connection states (idle, active, closed), connection errors, etc. These metrics could aid in diagnosing issues related to network connections and potentially optimize connection management.

HTTP Performance Metrics: For the net/http package, metrics on request/response times, the number of active connections, and error rates could be insightful. This information could be used to analyze the overall health and performance of HTTP servers, identify slow endpoints, or detect potential performance regressions.

Resource Utilization: Metrics that provide insights into resource utilization, such as memory and CPU consumption for specific HTTP handlers or net connections, could be valuable for optimizing the efficiency of servers and identifying resource-heavy components.

Error Metrics: Knowing the frequency and types of errors encountered in the net and net/http packages could help pinpoint common issues and improve error handling strategies.

TLS Handshakes: For secure connections, having metrics related to TLS handshakes (e.g., handshake duration, handshake failures) could aid in understanding the security overhead and optimizing TLS configuration.

[seankhliao]

http server:

• distribution metrics (by path and status code)
  • processing latency
  • request / response size
• counters
  • panics recovered
  • errors/warnings (everything that triggers net/http.Server.ErrorLog)
  • rejected malformed requests

http client:

• distribution metrics (by path and status code)
  • processing latency
  • request / response size

something about the connection pools

sql

• distributions
  • query latency
  • response size

something about the connection pools

---

for distributions, it would be great if they could be exponential rather than fixed bucket.

---

opentelemetry have some experimental conventions

https://opentelemetry.io/docs/specs/otel/metrics/semantic_conventions/http-metrics/
https://opentelemetry.io/docs/specs/otel/metrics/semantic_conventions/database-metrics/

[jhg03a]

For sql, I'd also include information about the sql connection pool and connection state transitions to help appropriately tune better similar to pgBouncer.

[seankhliao]

for http paths, typically I've seen it masked by (templated) handler route, either within the instrumentation sdk, or later in post processing (eg using opentelemetry collector)

[swi-jared]

for http paths...

Notably, the http.target OpenTelemetry attribute was removed from ServerRequest in the Otel Go SDK, despite being required by the Otel specification (as of semconv v1.17). This was due to high-cardinality problems in resulting metrics. However, for traces the path is still quite relevant. So if there is any way to provide the original as well as more coarse-grained version (/user/123, and /user/{id} for example), that would be beneficial.

(Note that Otel semantic conventions are going through a massive change from v1.21 and beyond, so http.target is not the correct attribute going forth. However, as of this writing, the Otel Go SDK supports v1.17 internally)

[hherman1]

Ya, the metric should key on the pattern which was matched, not the actual HTTP Path

[treuherz]

being able to get the pattern back out to do this kind of instrumentation is exactly what I had in mind when proposing #61551. I think having aggregation follow routing is the only way to aggregate requests in a way that's semantically meaningful

[jba]

@seankhliao, it seems like many of those can be done already, without adding anything to the standard library. I'm not sure if you can measure panics recovered or rejected malformed requests, and I know the connection pool stuff would need internal support. But that's it—everything else you mention can be done outside. Am I wrong about that?

[seankhliao]

It certainly can be done outside std, but it would be nice not to need to import https://pkg.go.dev/github.com/felixge/httpsnoop every time for the server side, not sure about client side libraries. Plus there are ordering issues which you have to be very careful with when chaining middleware together.

If #61410 were to be accepted, it would be a shame if you can't easily get back the parameterized pattern for use as the cleaned path attribute for the metric.

More importantly, I'd like it to be observable by default. Having standard metrics that instrumentation libraries can hook into mean I can trust the numbers mean the same thing, regardless of which way they were reported into the monitoring backend.

[seankhliao]

If you look at https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/instrumentation , instrumentation needs be (re)written for every http router people may use, with varying levels of feature completeness. Having it in the server just after it's parsed but before it's routed would instrumentation libraries only have to be written once.

[niiad]

You've got a point. Also agree to it being observable by default.

[jba]

Same question for @niiadoteibrown and @LinuxSploit. The ones you can't already get seem to be about the underlying connections, TLS handshakes, and handler-specific or connection-specific resource usage. What am I missing?

[niiad]

Seems about right, though insight about the underlying connections and connection-specific resource usage will be very useful. Error metrics as pointed out by @LinuxSploit can really improve error handling strategies. Having the net or net/http emit their own detailed metrics on the stated areas, like you said, the ones we can't already get, can go a long way to ease development process for example.

[lrweck]

Is it possible to track connection reuse for incoming/outgoing requests?

[bwplotka]

👋🏽 That's an epic idea, especially for net and net/http packages. I would love to help!

I'm part of the Prometheus team and I maintain https://github.com/prometheus/client_golang. It gives us quite an overview of what's used and how often, as the package is imported by over 38k packages in OSS. Related to net and net/http, we maintain set of so called "middlewares" (or in gRPC - interceptors) that allow users, right now, to have set of golden signal metrics (total and current count, errors, latencies, sizes, .etc) for their HTTP servers and clients, which got a lot of traction e.g.:

• HTTP: https://pkg.go.dev/github.com/prometheus/[email protected]/prometheus/promhttp
• gRPC: https://pkg.go.dev/github.com/grpc-ecosystem/go-grpc-middleware/providers/prometheus with https://github.com/grpc-ecosystem/go-grpc-middleware

There is also https://github.com/mwitkow/go-conntrack for connection tracking (asked by @lrweck)

That's only for the HTTP part, we could walk through other packages and identify what others use and built in OSS for those.

Providing them natively by Go std packages might definitely improve accessibility, consistent semantics and convenience. However we should discuss how to also allow certain flexibility and to what degree, to make it useful. For example, middlewares allow you to change dimensions (e.g. code/method) per certain HTTP paths, or skip instrumentation for some paths/methods completely.

[rhysh]

I wonder if #49881 is a useful precedent for how to decide which metrics are useful and appropriate here. That added a sum of sync.{,RW}Mutex wait time to the runtime/metrics package.

That same data is available by enabling mutex profiles with runtime.SetMutexProfileFraction, using runtime/pprof to obtain a profile, and summing the entries. But the /sync/mutex/wait/total:seconds metric described in that issue is a quick summary to indicate whether there's a problem worth further investigation.

Several of the metrics mentioned in this discussion already could be collected at the application layer by user-provided instrumentation on a net.Dialer, http.RoundTripper, httptrace.ClientTrace, etc. That instrumentation can give details, after the application owner invests the effort to build/integrate/deploy it. But as an earlier step, I wonder if app owners should be able to get simple counters that show them whether or not there'd be anything to find if they dug deeper.

[glenjamin]

I've arrived here after posting #63494

I think request lifecycle stuff is reasonably well-serviced by httptrace or general interface wrapping approaches

I would say the highest priority areas to expand metrics on should be areas that are currently invisible, by which I mostly mean connection pools.

The interface exposed by sql.DB for Stats() works very well in my experience, and is quite easy to hook up to various forms of external telemetry.

[kalin-sa]

Note that httptrace is servicing only client-side metrics, and is not relevant for http server instrumentaion.

[digi0pz]

I would love to see metrics from the std library especially in the connection pooling area.
Is this proposal accepted?

[ianlancetaylor]

This is an open discussion, not yet a proposal.