Expand OONI’s testing model to support richer testing input

[bassosimone]

This epic issue is about expanding OONI's testing model to support richer testing input. The general idea is that using an URL as input has served us well but now it has become a bottleneck to implement specific kind of experiments. For example, currently we cannot express that we want to measure a URL using HTTP/3. Likewise, we cannot include good IP addresses for the URL's domain.

Here's a breakdown of the milestones based on a presentation I delivered in the 2023 @ooni/ooni-team meeting, with subsequent integrations as we continued to evolve our planning.

stateDiagram-v2
  state "M0 (@bassosimone)" as M0
  state "M1 (@bassosimone)" as M1
  state "M2 (@bassosimone)" as M2
  state "M3 (@bassosimone)" as M3
  state "M4.1 (@bassosimone)" as M4_1
  state "M4.2 (@aanorbel)" as M4_2
  state "M5 (@hellais)" as M5
  state "M6 (@majakomel)" as M6
  M0 --> M1
  M1 --> M2
  M1 --> M3
  M1 --> M4_1
  M4_1 --> M4_2
  M2 --> M5
  M5 --> M6

Loading

Here are the issues for each milestone:

M0. preconditions for implementing richer input

• probeservices: make sure we can get a compressed check-in response probe#2379

M1. richer-input microservices

• richer input: support dnscheck backend#754
• richer input: support signal backend#756
• richer input: support stunreachability backend#755
• richer input: support torsf backend#758
• check-in v1: richer input for riseupvpn: discuss protoype backend#757
• check-in v1: richer input for urlgetter: discuss prototype backend#759

M2. miniooni uses richer input

• miniooni: invoke check-in for every experiment probe#2603
• engine: introduce executor for richer input probe#2607
• miniooni: refactor to use the richer input executor probe#2609
• engine: deliver richer input to dnscheck probe#2606
• engine: deliver richer input to stunreachability probe#2598
• engine: deliver richer input to riseupvpn/openvpn probe#2600
• engine: deliver richer input to torsf probe#2601
• engine: deliver richer input to urlgetter probe#2602
• engine: deliver richer input to signal probe#2599

M3. ooniprobe uses richer input

• ooniprobe: refactor to use the richer input executor probe#2610
• ooniprobe: invoke check-in for every run invocation probe#2604

M4.1 oonimkall uses richer input

• ???

M4.2 ooni/probe-mobile uses richer input

• ???

M5. ooni/data powered analysis

• ooni/data: support for processing signal results: no action needed, we're already good;
• ooni/data: support for processing stun_reachability results: no action needed, we're already good;
• ooni/data: support for processing dnscheck: need to support HTTP/3 measurements;
• Create list_observations API endpoint data#35
• Create aggregate_observations endpoints data#36

M6. ooni/explorer work

• Support richer input - explorer explorer#870

Because this is an epic issue, we will create child issues as needed.

[agrabeli]

For more context, this is what is written in our DRL proposal:

We aim to expand OONI’s testing model to support richer testing input and, by extension, enable all OONI Probe app users worldwide to more easily run novel experiments, automatically process these measurements and present findings on OONI Explorer as open data. This involves making improvements to the communication layer between the measurement coordination infrastructure and probes to improve our ability to provide a richer set of configuration parameters to network experiments that goes beyond just providing them with URLs to test.

Specifically, this involves making all the components of the OONI ecosystem (the backend API, OONI Probe clients, the data processing pipeline, the database model, and OONI Explorer) aware that the input for a measurement is now an object containing a URL, as well as other configuration parameters (for example, the backend would aggregate measurements based on a richer definition of input to score HTTP/2 measurements differently than HTTP/3 measurements).

To this end, this may entail:

• Changes to the backend API to serve richer testing input to probes;
• Changes to OONI Probe’s API to pass the rich input object to experiments;
• Changes to the data processing pipeline to use the richer input model to group and score measurements;
• Changes to the database model to use the richer definition of input for the indexing of measurements;
• Adding support to OONI Explorer for querying measurements with richer input and displaying the full input (instead of just measured URLs).

This work is a requirement for expanding our censorship measurement capabilities, and will enable us to ship a new methodology for measuring throttling. For example, if we suspect that access to https://twitter.com is being throttled in Russia, we could serve richer input (including a boolean flag) to OONI Probe clients in Russia to perform additional measurements that are useful for detecting throttling. In such a case, for example, the probe would fetch the whole page (as opposed to fetching a small part of the webpage) and would collect and submit download speed measurements to OONI’s backend.

[agrabeli]

I have set a tentative timeline on zenhub which probably needs to be adjusted.

[bassosimone]

As explained in ooni/2023-05-richer-input@7871d3b, the https://github.com/ooni/2023-05-richer-input contains a prototype that explores the richer input domain and redesigns ooniprobe around richer input.