Just an observation: the user experience might be tricky here. Because now we might have the potential scenarios where disabled stats appear in the summary.

I know that isn't a new limitation introduced here, instead it should highlight the need to track/untrack specific metrics as described in #1321, where we should extend the same concept to stats as well.

I'm sorry @codebien, but I don't fully get your point here. As far as I can see, #1321 is all about metrics and sub-metrics, but here we only care about different "calculations" (percentiles, basically") of already existing metrics/sub-metrics, so I cannot see how it related 🤔 Conceptually I see those two as different, unrelated things.

@joanlopez, my interpretation of the issue is that the user wants to add a threshold for a specific metric at a particular percentile without adding that percentile to all the other metrics.

If that's not the case, then I guess we could simply suggest using summaryTrendStats: [/* other stats */..., 'p(99.99)'] or on our side, skip visualizing the non-computed stats to avoid undefined values.

However, neither of these solutions addresses the user's specific requirement.

Instead, I think 1321 might address it, if we integrate the proposed API with an option for defining the stats. Something similar to the example below:

export const options = { 
  summaryTrendStats: ['max', 'p(95)'], 
};

track("http_req_duration", {percentile: 99.9})

Where without the track line, the threshold will never fail because the stats for the metric doesn't exist. Adding this feature then will lead to a breaking change, but that should be acceptable since I don't expect this being real before v2.

@joanlopez, my interpretation of the issue is that the user wants to add a threshold for a specific metric at a particular percentile without adding that percentile to all the other metrics.

Let me slightly correct you, because I think it's key here:

"...at a particular percentile without displaying that percentile in summary to all the other metrics."

What I mean here is basically two things (in my humble opinion):

1. The summaryTrendStats option is, as its name hints, an option to configure the "end-of-test summary", that let users choose what's displayed there (which percentiles values are displayed in the summary for trend metrics). NOT an option to configure what metrics are tracked/untracked.
2. Percentiles aren't a form of "(sub)metrics tracked", but just a calculation you do over a metric, or more correctly over a sink. It's like when we refer to "min" or "max" for a given "gauge sink". What I think could make sense in the future, as you mentioned, is a mechanism to let the user choose what (sub)metrics are tracked, but less likely what calculations.

If that's not the case, then I guess we could simply suggest using summaryTrendStats: [/* other stats */..., 'p(99.99)'] or on our side, skip visualizing the non-computed stats to avoid undefined values.

I disagree with both, because:

• I still see value of having the opportunity to define a threshold for a concrete percentile, even if you don't want to see the value of that percentile for ALL the metrics displayed in the end-of-test summary. In other words, why do I need to add to the summary the value of p(99.99) for all the trend metrics, if I only care about the p(99.99) of this particular metric X?
• I think it's better to display the value in the thresholds section that hiding it because:
  • It keeps that behavior consistent with the whole "thresholds" section, where the value of the aggregation the threshold is defined on is always displayed. It also keeps it consistent with the rest of the summary,as the value won't be displayed in the list below, where all the metrics are displayed, according to summaryTrendStats.
  • Accompanying thresholds results with the value of the particular aggregator used by the threshold was precisely one of the main ideas of the revamped end-of-test summary, because it really makes it very easy for the user to understand why that threshold succeed/failed.

However, neither of these solutions addresses the user's specific requirement.

Instead, I think 1321 might address it, if we integrate the proposed API with an option for defining the stats. Something similar to the example below:

export const options = { 
  summaryTrendStats: ['max', 'p(95)'], 
};

track("http_req_duration", {percentile: 99.9})

Where without the track line, the threshold will never fail because the stats for the metric doesn't exist. Adding this feature then will lead to a breaking change, but that should be acceptable since I don't expect this being real before v2.

That's definitely out of the scope of this PR, so I'd postpone that discussion for whenever we design that feature, but I'm not super convinced that having to specify what calculations do we want to track is really a good idea, because calculations aren't that associated with noise/high memory consumption, which I think are the underneath goal of that feature, and doing so would make both the design and implementation of such feature way more complex.

cc/ @oleiade any take here?

@joanlopez However, the CI seems to be red, can you check that is not related, please?

@joanlopez However, the CI seems to be red, can you check that is not related, please?

It isn't, it's just one of the flaky Browser E2E tests that define a thresholds that isn't always satisfied.