Limiter extension API interfaces and implementation helpers (**draft 5**)

[jmacd]

Description

Draft demonstrating the following advances on the previous draft:

1. Base limiter is the MustDeny function as a stand-alone, like memory limiter extension/processor would use.
2. Rate/Resource limiters have a single non-blocking interface
3. Helper functions wrap the non-blocking interfaces with blocking forms
4. Limiter middleware helpers for HTTP and gRPC revived from earlier draft
5. Basic rate limiter based on golang.org/x/time/rate
6. Basic resource limiter based on collector-contrib/internal/otelarrow/admission2
7. More-consistent functional style
8. Suggested configmiddleware integration
9. memorylimiterextension

Follows drafts
1: #12558
2: #12633
3: #12700
4: #12953

Link to tracking issue

Part of #9591.
Part of #12603.

Testing

NONE: This is a demonstration, smaller PRs will be broken apart and tested individually if reviewers like the look of this, the full assembly.

Documentation

Updated.

[bogdandrutu]

In general LGTM extensionlimiter package, for the "limiterhelper" only looked very high level and I will do a more deeper review after we go over the first round of comments related to the main extensionlimiter package.

[bogdandrutu]

nit: Rename to BasicLimiter? or Limiter?

Reasoning: For me "base" means that it will be embedded into something else in order to use it, and cannot be used as a standalone. Please correct me if I am wrong.

[jmacd]

Thanks. I'm open to either of these, and I agree with your reasoning that "base" isn't great. @axw has suggested that the type and the method name should be related, so if we have Limiter then the API method should be Limit(), if we have Checker it should be Check(), however I find it difficult to apply this pattern to MustDeny. Should this be something else completely, like SaturationChecker with a method named CheckSaturation as @axw proposed?

[axw]

To explain my thought process:

I would typically name interfaces after a capability, hence naming the interface after a method. This is also suggested in https://go.dev/doc/effective_go#interfaces_and_types:

Interfaces with only one or two methods are common in Go code, and are usually given a name derived from the method, such as io.Writer for something that implements Write.

So to me, RateLimiter and ResourceLimiter feel natural, but BaseLimiter does not -- you're not limiting a base. Also, something like a "base" sounds like a mixin type, which would be relevant for implementation (e.g. a struct) rather than the capability/behaviour (interface).

It seems to me that the ability to check saturation is the capability here, hence SaturationChecker/CheckSaturated.

[axw]

It's also possible that we don't really need to expose this interface at all, but just have a saturation check method on each of RateLimiter and ResourceLimiter. See also my other questions about BaseLimiter.

[jmacd]

I agree and have responded to your other questions, let's take this discussion over to:
https://github.com/open-telemetry/opentelemetry-collector/pull/13051/files#r2131844080
https://github.com/open-telemetry/opentelemetry-collector/pull/13051/files#r2131827330

[bogdandrutu]

I would like to include the idea of "as early as possible" into this explanation.

[jmacd]

Will do.

[bogdandrutu]

I commented in the BaseToResourceLimiterProvider for some issues that I can imagine, but in general I am not sure the path is to make all look the same vs have helpers for the 3 types independent.

[jmacd]

The approach I've taken is to provide every possible conversion function, which is a fairly small set. BaseToRate, BaseToResource, or RateToResource providers are for situations where LimiterWrapper can't be used (these are "independent", in the sense you mean it, I think), in which case circumstances vary. Two examples follow for Rate and Resource limits.

As pointed at here, middleware is often in a place where only a Rate limit can be applied -- typically for compressed bytes -- and in this case BaseToRate applies (e.g., to adapt memorylimiterextension for middleware).

In scenarios where LimiterWrapper does not apply -- having to do with flow control (e.g. for stream backpressure) -- but where the Resource limiter can be applied because the application is able to release after the request finishes. Here, a Resource limiter interface should be preferred because its interface can be adapted from all the others. In these cases you will use RateToResource and BaseToResource.

[bogdandrutu]

One of the top section should be for "end users" how to configure limiters in some of the most common cases.

[jmacd]

I will work on this -- although I haven't actually produced anything for end users that is new here, only the memorylimiterextension could be documented (and it is already linked below).

I would agree to update this with links to the implementations. I would like to request input from @axw on this topic, because while I have created helper implementations of the rate-limiter (from golang.org/x/time/rate) and the resource-limiter. Both of these have two numeric parameters, and we still have to discuss how a real limiter will be configured, which will lead back to our open questions (e.g., how to configure the limiter based on client metadata, how to configure the limiter based on the component name or the signal type?).

[bogdandrutu]

nit: Called this basic at the beginning of the document, please be consistent.

[jmacd]

Yes, I've been inconsistent. See https://github.com/open-telemetry/opentelemetry-collector/pull/13051/files/ba136deb9590d6fc5999c1af548f59965cbd0599#r2106643694.

[bogdandrutu]

Please bear with me on this:

I am trying to evaluate if for end users (not how we implement it, which we can do with same structs) they would like to see:

extensions:
  ratelimiter/limit_for_grpc:
    # rate limiter settings for gRPC
  ratelimiter/limit_for_grpc:
    # rate limiter settings for HTTP

receivers:
  otlp:
    protocols:
      grpc:
        middlewares:
        - ratelimiter/limit_for_grpc
      http:
        middlewares:
        - ratelimiter/limit_for_http

OR

extensions:
  ratelimiter/limit_for_grpc:
    # rate limiter settings for gRPC
  ratelimiter/limit_for_grpc:
    # rate limiter settings for HTTP

receivers:
  otlp:
    protocols:
      grpc:
        limiters:
        - ratelimiter/limit_for_grpc
      http:
        limiters:
        - ratelimiter/limit_for_http

Suggestion being in confighttp/configgrpc to add a Limiters []configmiddleware.Config mapstructure:"limiters,omitempty" similar with https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/configgrpc.go#L111 but just call it Limiters.

[jmacd]

This is a side-note, but a relevant one.

Keep in mind that our present definition for configmiddleware.Config is

// Middleware defines the extension ID for a middleware component.
type Config struct {
	// ID specifies the name of the extension to use.
	ID component.ID `mapstructure:"id,omitempty"`
	// prevent unkeyed literal initialization
	_ struct{}
}

which means the config reads with an id like:

      grpc:
        middlewares:
        - id: ratelimiter/limit_for_grpc

Does squash support what you've written? Otherwise the way it reads, I think you want something like

type Config = component.ID

instead of a struct for middleware config.

[jmacd]

As for your question about separating limiters and middleware, I think we should focus on the HTTP case, where it is pretty typical to interleave limiters and other kinds of middleware. There is a potential to refactor confighttp so that some of the existing setup would move into middleware components, for example:

• compression middleware
• opentelemetry instrumentation

at this point, more questions are raised, because we have discussed limiting by both compressed and uncompressed weights. gRPC gives you compressed and uncompressed in the same stats handler call, but HTTP does compression in middleware, so the network-bytes limit has to be applied before middleware.

In my presentation here, I've implemented "memory_size" -- you suggested request_bytes, sure -- in the receiver. If we want to implement request-size earlier, in the middleware, then we need to configure HTTP limiters before and after the compression middleware.

For this reason, I think limiters should remain in the list of middleware.

For a receiver that otherwise does not use middleware, I think it would be appropriate to follow your example above, that is to have a Limiters []configmiddleware.Config field where we expect to find only limiters.

If you agree, I would file separate tracking issues for the process of moving compression and/or opentelemetry instrumentation into middleware components. This would allow building the collector with alternative compression libraries and/or OTel SDKs, too 😀.

[mx-psi]

Jmacd's comment makes me feel sympathetic to just putting everything in a middlewares list

[axw]

In my ideal world, users should be able to configure component-specific rate limiters independently of transport. Maybe they use middleware that uses the same limiter extension, maybe not -- the user should be able to choose.

I would also want to get rid of the "StandardNotMiddlewareKeys" function and leave it to users to choose which limits they want to apply. This can be done by having multiple, specific limiter configurations. e.g.:

extensions:
  ratelimiter/limit_for_grpc:
    # rate limiter settings for gRPC
  ratelimiter/limit_for_http:
    # rate limiter settings for HTTP
  ratelimiter/limit_for_otlp:
    # rate limiter settings for OTLP, independent of transport

receivers:
  otlp:
    # limiters holds OTLP-specific limiters. For transport-level limiters, configure middleware.
    limiters:
      # requests defines a limiter for processing OTLP requests (batches), independent of signal.
      requests: ratelimiter/limit_for_otlp
      # items defines a limiter for processing OTLP items: log records, spans, data points, profile samples
      items: ratelimiter/limit_for_otlp
      # bytes defines a limiter for processing OTLP-encoded bytes.
      bytes: ratelimiter/limit_for_otlp
    protocols:
      grpc:
        middlewares:
        - ratelimiter/limit_for_grpc
      http:
        middlewares:
        - ratelimiter/limit_for_http

If you agree, I would file separate tracking issues for the process of moving compression and/or opentelemetry instrumentation into middleware components.

Yes please. FWIW, we just internally found a need to do the same for auth, IMO that would be good to have too.

[jmacd]

@axw I like your proposal.

I will remove all the "Standard" keys. I had been trying to avoid applying the same limit more than once, but your proposal avoids the problem in a straightforward way. Users would be able to limit something more than once on purpose.

Moreover, this simplifies the detail you raised about some rate limiters not supporting cancellation, because there would be no way to configure multiple simultaneous limiters. No need to cancel a reservation because some but not all limiters granted a request.

Moreover, this will simplify most receivers because we can automatically implement limiters at the receiver helper level.

I will follow up. See also #13228 and #13229.

[bogdandrutu]

I know you will tell me "I told you", but how can we configure limiters for non grpc/http scrapers like hostmetrics/filebased?

[jmacd]

... 😂 I mean yes, the point of this section was to answer your question. We could go with either of the options you presented, also I made the mistake about id: here too: we should think about whether we want to eliminate the id: prefix, see https://github.com/open-telemetry/opentelemetry-collector/pull/13051/files/ba136deb9590d6fc5999c1af548f59965cbd0599#r2106696658.

[jmacd]

Note to incorporate feedback in #12953 (review)

[MovieStoreGuy]

I would strongly advise against anon struct definitions here, please move the definition to be a Nop type.

[jmacd]

I have been trying to follow a particular style that is already present in the repository. Since the Nop instance is usually reserved for the extension-test library, I wasn't sure about what you proposed. If we create a Nop instance here then should we do that everywhere? See this comment, #13051 (comment), which led me to believe I should not create a Nop type as well. In some ways, I wrote it this way to elicit a conversation about style when it comes to extension interfaces with >1 function.

[axw]

Sorry for the delay. Overall I think it's looking good.

I left a few related questions about BaseLimiter - it feels like it could be removed, and left to extensions to present either a RateLimiter or ResourceLimiter that does something specific to that extension where there's no natural rate or resource.

I left a suggestion for how we could structure the component-specific limits vs. transport-specific limits, which I think also provides an answer to @bogdandrutu's question about scraper receivers.

[axw]

To explain my thought process:

I would typically name interfaces after a capability, hence naming the interface after a method. This is also suggested in https://go.dev/doc/effective_go#interfaces_and_types:

Interfaces with only one or two methods are common in Go code, and are usually given a name derived from the method, such as io.Writer for something that implements Write.

So to me, RateLimiter and ResourceLimiter feel natural, but BaseLimiter does not -- you're not limiting a base. Also, something like a "base" sounds like a mixin type, which would be relevant for implementation (e.g. a struct) rather than the capability/behaviour (interface).

It seems to me that the ability to check saturation is the capability here, hence SaturationChecker/CheckSaturated.

[axw]

I would caution against overindexing on that package. I quite like its API, but reservation is easy for local rate limiters, not so easy for distributed ones.

We wouldn't be able to adapt our rate limiter to this API, since Gubernator just does not support reservation. Similarly, it wouldn't be possible to use https://pkg.go.dev/github.com/juju/ratelimit (though I don't think we would be able to use it here anyway -- it's LGPL)

IIUC, your goal is to enable non-blocking behaviour, and not specifically to enable reservation/cancellation -- is that correct? If so, it is possible to make the rate limiters non-blocking without supporting reservations. For example, you could return a duration that the caller is expected to wait before proceeding, like:

type RateLimiter interface {
	// RateLimit reserves n tokens, either immediately or in the future,
	// returning the duration that the caller must wait before those tokens
	// are available.
	RateLimit(ctx context.Context, n int) (time.Duration, error)
}

Internally, a golang.org/x/time/rate-based implementation would create a reservation to implement this interface.

The key difference here is that once you've requested the limit, there's no going back.

[jmacd]

There's some misunderstanding--I will try to address the source of confusion.

IIUC the API you showed above is identical to RateReservation. (In this PR) the RateLimit call returns an error (to fail fast) or a reservation object--the reservation object has the time.Duration (maybe 0 for immediate admission) and a way to cancel in case the caller gives up before the time elapses. This appears to be typical for local rate-limiters, because it's a feature of the Leaky- and Token-bucket algorithms that you can admit some traffic ahead of time, e.g., in Juju. Note that Juju doesn't have a way to cancel and return an unfilled limit request, but this can just be documented: i.e., this limiter ignores request cancellation, limits cannot be "undone" with this implementation.

As for a distributed limiter, I agree it is not feasible to cancel a reservation and I see that the Gubernator API does not return such a time duration. In this case, I would simply document it: the limiter which calls Gubernator will always either fail fast or grant immediate admission. I believe this works in the APIs I have proposed, just that the time.Duration will always be zero and the cancel function does nothing.

Given this evidence, I'm on the fence about something: we could simplify matters in this code by having a single unified reservation object modeled on the ResourceReservation. Rate limiters would be forced to return a new channel whenever they have to delay, and they would be responsible for setting a timer to close the channel themselves. This would amount, I suspect, to marginal overhead for rate limiters while simplifying the API structure quite a bit. Reviewers, would you prefer to see two reservation APIs, one for rate and one for resource, or would you prefer to see one reservation API that is slightly less optimized.

[axw]

IIUC the API you showed above is identical to RateReservation.

Not quite identical - the Cancel method is issue

... but this can just be documented: i.e., this limiter ignores request cancellation, limits cannot be "undone" with this implementation.

I guess that would be OK, and may be a little better for local rate limiters that can cancel.

As for a distributed limiter, I agree it is not feasible to cancel a reservation and I see that the Gubernator API does not return such a time duration.

Gubernator returns the time at which the rate limit resets (https://github.com/gubernator-io/gubernator/blob/427194a6c593e6d9da7283fbaea49939052c8676/gubernator.proto#L204-L205). So you could retry after that time, but it's not guaranteed, and thus won't be fair. (Seems in our implementation we're misusing the reset time, and taking it to mean the time at which the request will succeed.)

[jmacd]

Thanks. I'm starting to think cancellation is a non-issue and should be removed from the interface to reduce surface area. For one, the proposal you made in #13051 (comment) means that there are no "multi"-limiters needed anywhere, which makes cancellation less important.

[axw]

👍 Sounds good. As I mentioned above, while I was looking at our rate limiter implementation again I realised that we're misusing the reset time (elastic/opentelemetry-collector-components#619). Given that, I don't know that returning a duration is going to work either.

[axw]

This feels a bit too prescriptive. The wrapper API seems like it should be orthogonal to whether the limiter is blocking or non-blocking. Then for example, users could configure their receivers to either return immediately with 429 or block. Either way I would expect this limiter provider to be used, it would just be a matter of whether it internally blocks or not.

[jmacd]

I'm having trouble following your exact meaning. In this location, we are constructing a blocking interface object, i.e., a wrapper that is able to automatically block the caller because of its call semantics. (The same can't be said for the lower-level Rate- and ResourceLimiter APIs, they aren't able to block the caller they must return a reservation. Using a blocking limit is exactly how the wrapper implements its contract.)

users could configure their receivers to either return immediately with 429 or block

I agree with this aspect. The blocking resource limiter helper, as applied here, will only block the caller when it is invited to do so by the limiter. Users can configure the limiter to return an error or to return instructions for blocking. To be concrete, in my proposal, here's how this works say for ResourceLimiter.

If the limiter extension is configured (or coded) to fail fast, then it will return an error instead of a Reservation object. Blocking only happens for valid reservations. When not returning a reservation, an error will be translated to HTTP 429 or gRPC RESOURCE_EXHAUSTED. Note that we haven't exactly looked at how these errors are synthesized: I would say it depends on #13042.

For the code in this PR, I left a couple of TODOs about how gRPC and HTTP will actually do this. Reviewers, we should scrutinize the TODO mentioned in grpc's limiter. The gRPC stats handler interface has no way to inject an error, so if we need the network-bytes rate limiter to be able to fail the connection, we have to pass something through context--this implies new memory allocations, so I thought we should be clear before doing something about it.

[axw]

Thanks for explaining. I had in my mind that the limiter would not return an error, but the caller would decide to error or block depending on its configuration if the limiter were to return a non-zero wait duration.

[jmacd]

Thanks. I would like your help to decide where this configuration belongs. For middleware components, the extensions themselves are adapted into middleware via helpers, so the only thing to configure is the extension. In the two basic implementations, each has a parameter for how much blocking is permissible, or that's how I understand it. The rate limiter's "burst" controls how much waiting will be prescribed, and the resource limiter's "waiting_mib" controls how much waiting will be prescribed. I believe this is the role of "throttle_behavior" in the elastic ratelimiter component.

Limiters have the context argument, which means they can derive the deadline of the request and use it to make their decision. There is a related specification for OTLP retries, which gives us a way to return an error asking for a retry after a certain period, but I'm skeptical of this approach. I would be interested in a second configurable parameter, a threshold used to decide when to return a retry-error (e.g., if greater than X duration or 75% of deadline) and when to just wait as a matter of efficiency.

I have a related question about limiting network bytes in middleware. Is there some place we should put a buffer size, or a minimum request size, so that relatively large requests can be made to Gubernator while relatively small blocks are read in the RPC library. Have you considered a minimum-request size and a buffering mechanism for rate limiters?

[axw]

I think adding configuration to the limiter makes sense, I just hadn't though it through properly before :)
Putting it anywhere else would suggest multiple users of the same rate limiter with different behaviour, which seems unnecessary.

I have a related question about limiting network bytes in middleware. Is there some place we should put a buffer size, or a minimum request size, so that relatively large requests can be made to Gubernator while relatively small blocks are read in the RPC library. Have you considered a minimum-request size and a buffering mechanism for rate limiters?

I haven't thought about this tbqh. Sounds a bit like https://github.com/gubernator-io/gubernator/blob/master/docs/architecture.md#global-behavior

[axw]

Would anything ever need just a BaseLimiter? I'm wondering if the BaseLimiterProvider interface is actually needed, or if clients can request either a RateLimiterProvider or a ResourceLimiterProvider, given an extension.

[jmacd]

I have the same question. If there are existing integrations with the memorylimiter extension, I expect them to continue on this path, however they could just as easily request either a Rate or Resource limiter and then only use its CheckSaturation method (a.k.a. MustDeny). It is difficult to imagine a scenario where a component would want to check saturation but be incapable of applying quantifiable limits.

See also https://github.com/open-telemetry/opentelemetry-collector/pull/13051/files#r2131844080 cc/ @dmitryax @bogdandrutu

[axw]

I think this name needs to be a bit more descriptive about the behaviour. A key aspect is that if the limit is not currently saturated, any number of concurrent operations may will be allowed through: the operation will not directly affect the saturation. (Maybe indirectly, e.g. by increasing memory usage)

I also wonder if there really can be a one-size-fits-all for this? Perhaps it should be left to each extension to choose how to adapt? Do you foresee anything other than memorylimiter implementing only BaseLimiter and neither RateLimiter nor ResourceLimiter?

[jmacd]

Do you foresee anything other than memorylimiter implementing only BaseLimiter and neither RateLimiter nor ResourceLimiter?

Great question. I do not see immediately any other implementations of the saturation-checker.

In hindsight, "BaseLimiter" was a terrible choice of names. Thanks to your input, I prefer SaturationChecker with method CheckSaturation, however I would like to hear from others. There has been a suggestion for this to be named MustDeny following the memorylimiter extension's example. I'm not sure if anyone is using the memorylimiter extension today -- if so they are retrieving the component and expecting it to have a MustDeny method.

@dmitryax would you offer an opinion on these topics?

[axw]

It's also possible that we don't really need to expose this interface at all, but just have a saturation check method on each of RateLimiter and ResourceLimiter. See also my other questions about BaseLimiter.

[axw]

In my ideal world, users should be able to configure component-specific rate limiters independently of transport. Maybe they use middleware that uses the same limiter extension, maybe not -- the user should be able to choose.

I would also want to get rid of the "StandardNotMiddlewareKeys" function and leave it to users to choose which limits they want to apply. This can be done by having multiple, specific limiter configurations. e.g.:

extensions:
  ratelimiter/limit_for_grpc:
    # rate limiter settings for gRPC
  ratelimiter/limit_for_http:
    # rate limiter settings for HTTP
  ratelimiter/limit_for_otlp:
    # rate limiter settings for OTLP, independent of transport

receivers:
  otlp:
    # limiters holds OTLP-specific limiters. For transport-level limiters, configure middleware.
    limiters:
      # requests defines a limiter for processing OTLP requests (batches), independent of signal.
      requests: ratelimiter/limit_for_otlp
      # items defines a limiter for processing OTLP items: log records, spans, data points, profile samples
      items: ratelimiter/limit_for_otlp
      # bytes defines a limiter for processing OTLP-encoded bytes.
      bytes: ratelimiter/limit_for_otlp
    protocols:
      grpc:
        middlewares:
        - ratelimiter/limit_for_grpc
      http:
        middlewares:
        - ratelimiter/limit_for_http

If you agree, I would file separate tracking issues for the process of moving compression and/or opentelemetry instrumentation into middleware components.

Yes please. FWIW, we just internally found a need to do the same for auth, IMO that would be good to have too.

[jmacd]

Status: I am freezing this PR in its current state, to allow the open discussions to continue, and I will make revisions in a new branch. I will follow the proposal in #13051 (comment) by @axw.