[Ratelimit Processor] Instrument the ratelimiter service with telemetry metrics

[gizas]

This is the initial implementation for https://github.com/elastic/hosted-otel-collector/issues/513

The code adds a newMetrics reader in the ratelimitprocessor.

At the moment we have implemented only ratelimitRequests metric.Int64Counter
Also the newMetricsReader makes use of the global otel processor that we assume that is initialized upon processor usage.

To be reviewed:

• if err != nil: WithReason = "request_error" and WithDecision = "accepted"
• if n := len(responses); n != 1: WithReason ="request_error" and WithDecision = "accepted"
• if n := len(responses)==1 and resp.GetError() != "": WithReason = "limit_error" and WithDecision ="accepted"
• if gubernator.Status_OVER_LIMIT: WithReason = "over_limit" and WithDecision ="throttled"
• Default: "WithReason" = "under_limit" and "WithDecision" ="accepted"

Sample document

After running make smoketest from the hosted-otel-collector repo, I can see this document in Kibana:

{
  "_index": ".ds-metrics-apm.app.hosted_otel_collector-default-2025.04.30-000001",
  "_id": "4sKIsJYBgg67ctLl7qDw",
  "_version": 1,
  "_source": {
    "@timestamp": "2025-05-08T15:34:45.244Z",
    "agent": {
      "name": "otlp",
      "version": "unknown"
    },
    "data_stream": {
      "dataset": "apm.app.hosted_otel_collector",
      "namespace": "default",
      "type": "metrics"
    },
    "labels": {
      "orchestrator_cluster_name": "default",
      "orchestrator_deploymentslice": "",
      "orchestrator_environment": "default",
      "processor_id": "ratelimit",
      "ratelimit_decision": "accepted",
      "reason": "under_limit",
      "x-elastic-project-id": "local"
    },
    "metricset": {
      "name": "app"
    },
    "numeric_labels": {
      "limit_threshold": 0
    },
    "observer": {
      "hostname": "apm-server-apm-server-56d7f679df-fcqdg",
      "type": "apm-server",
      "version": "8.17.0"
    },
    "otelcol_ratelimit": {
      "requests": 6
    },
    "service": {
      "framework": {
        "name": "github.com/elastic/opentelemetry-collector-components/processor/ratelimitprocessor"
      },
      "language": {
        "name": "unknown"
      },
      "name": "hosted-otel-collector",
      "node": {
        "name": "ed945a71-f853-42d9-a324-14092609edd9"
      },
      "version": "git"
    }
  }
}

[vigneshshanmugam]

The approach looks good to me. Some thoughts on additional metrics that we would need to introduce

• Request duration
• Can we measure the no of requests that are over the limits at a given time? I believe we can by using the decision dimension, just confirm if thats is the case.
• Concurrent requests

@lahsivjar Could you think of additional metrics that were useful in MIS so we can add them here? Thanks.

[vigneshshanmugam]

It would be good to add this also to the local rate limiter, to make sure we track all the incoming requests to this component. I would move this out.

[kaiyan-sheng]

@vigneshshanmugam Could you explain a bit more here please? Sorry I'm lost 😅

[vigneshshanmugam]

That was definitely without much context, apologeis. What I meant was to add it to

opentelemetry-collector-components/processor/ratelimitprocessor/local.go

Line 53 in 81b3c24

func (r *localRateLimiter) RateLimit(ctx context.Context, hits int) error {

as well as there is way to disable gubernator and use local in-memory ratelimier instead.

[lahsivjar]

Could you think of additional metrics that were useful in MIS so we can add them here? Thanks.

Currently in MIS we collect only 2 metrics from the rate limiter (ref):

1. A metric to track the status of the ingest requests. This metric has dimensions with bounded cardinality to record the decision and the reason for the decision by the rate limiter service.
2. A metric to track the update requests sent between gubernator instances to update them. This metric has just one dimension of status.

While I like the idea of collecting request duration, however, I am not sure how useful concurrent requests would be. We should also be wary of adding too many metrics in hot path, as it does add some overhead.

[vigneshshanmugam]

Please see this comment #562 (comment), We dont want to expose the project_id etc details, instead keep this to attributes being derived from metadata_keys. Not sure if thats something possible with the metadata schema, but we can come back to later as well.

[kaiyan-sheng]

Ahh sorry I missed it. Will remove project_id. Yeah I dont think the current mdatagen supports dynamic attribute keys. Can look into it more later.
How about processor_id then? ProcessorID is from telemetry.WithProcessorID(r.set.ID.String()).

[kaiyan-sheng]

Todo list in followup PR:

• Look into map type attribute for metadata_keys
• Add more telemetry metrics like request duration and etc