feat: improve transfer, publish, resolve examples #3296

Frando · 2025-05-09T09:26:22Z

Description

This improves the transfer, publish and resolve examples, which we often use for dogfooding new releases.

`transfer` example:

Set prod or staging environments with a single arg, while still optionally allowing to override relay, pkarr, and DNS servers.
Add a dev environment that uses locally running iroh-relay and iroh-dns-server instances, for easily using the example with a locally running dev setup of all services.
Optionally enable mDNS discovery
Print connection type changes
Allow reusing a node id by setting IROH_SECRET environment variable
Improve help and print output
Improve code to not duplicate endpoint setup and arguments between fetch and provide side

Here's the new help text and output:

Help text for provide

$ cargo run --release --example transfer --all-features -- provide --help
Provide data

Usage: transfer provide [OPTIONS]

Options:
      --size <SIZE>
          [default: 100M]

  -e, --env <ENV>
          Set the environment for relay, pkarr, and DNS servers.
          
          If other options are set, those will override the environment defaults.
          
          [default: staging]

          Possible values:
          - prod:    Use the production servers hosted by number0
          - staging: Use the staging servers hosted by number0
          - dev:     Use localhost servers

      --relay-url <RELAY_URL>
          Set one or more relay servers to use

      --no-relay
          Disable relays completely

      --relay-only
          If set no direct connections will be established

      --pkarr-relay-url <PKARR_RELAY_URL>
          Use a custom pkarr server

      --no-pkarr-publish
          Disable publishing node info to pkarr

      --dns-origin-domain <DNS_ORIGIN_DOMAIN>
          Use a custom domain when resolving node info via DNS

      --dns-server <DNS_SERVER>
          Use a custom DNS server for resolving relay and node info domains

      --no-dns-resolve
          Do not resolve node info via DNS

      --mdns
          Enable mDNS discovery

  -h, --help
          Print help (see a summary with '-h')

Help text for fetch

$ cargo run -q --release --example transfer --all-features -- fetch --help
Fetch data

Usage: transfer fetch [OPTIONS] <TICKET>

Arguments:
  <TICKET>
          

Options:
  -e, --env <ENV>
          Set the environment for relay, pkarr, and DNS servers.
          
          If other options are set, those will override the environment defaults.
          
          [default: staging]

          Possible values:
          - prod:    Use the production servers hosted by number0
          - staging: Use the staging servers hosted by number0
          - dev:     Use localhost servers

      --relay-url <RELAY_URL>
          Set one or more relay servers to use

      --no-relay
          Disable relays completely

      --relay-only
          If set no direct connections will be established

      --pkarr-relay-url <PKARR_RELAY_URL>
          Use a custom pkarr server

      --no-pkarr-publish
          Disable publishing node info to pkarr

      --dns-origin-domain <DNS_ORIGIN_DOMAIN>
          Use a custom domain when resolving node info via DNS

      --dns-server <DNS_SERVER>
          Use a custom DNS server for resolving relay and node info domains

      --no-dns-resolve
          Do not resolve node info via DNS

      --mdns
          Enable mDNS discovery

  -h, --help
          Print help (see a summary with '-h')

Output on provide side

$ cargo -q run --release --example transfer --all-features -- provide -e staging
Our node id:
        fdeb67bd2995cbdf1844e1170b01848af5e6b13beb4d754ec86dd38b4ecdae3a
Our direct addresses:
        78.xx.xx.xx:44377 (type: Stun)
        192.168.xxx.xxx:44377 (type: Local)
Our home relay server:
        https://staging-euw1-1.relay.iroh.network./

Ticket with our home relay and direct addresses:
nodead66wz55[redacted]

Ticket with our home relay but no direct addresses:
nodead66wz55fgk4xxyyitqrocybqsfplzvrhpvu25kozbw5hc2ozwxduajlnb2hi4dthixs643umftws3thfvsxk5zrfuys44tfnrqxsltjojxwqltomv2ho33snmxc6aa

Ticket with only our node id:
nodead66wz55fgk4xxyyitqrocybqsfplzvrhpvu25kozbw5hc2ozwxduaaa

[7b1a7ac46f] Connected
[7b1a7ac46f] Received: "7b1a7ac46f is saying hello!"
[7b1a7ac46f] Connection type changed to: direct(192.168.xxx.xxx:48801)
[7b1a7ac46f] Transferred 100.00 MiB in 0.6929s, 144.31 MiB/s
[7b1a7ac46f] Disconnected

Output on fetch side

$ cargo -q run --release --example transfer --all-features -- fetch -e staging nodead66wz55fgk4xxyyitqrocybqsfplzvrhpvu25kozbw5hc2ozwxduaaa
Generated a new node secret. To reuse, set
        IROH_SECRET=a924573310dc0c0763b64ea82c307895965883d1046f4555954689dd8a937d8c
Our node id:
        7b1a7ac46f528bc5182ea214326fe2a348d2a6f3a3a63d505947c3f0980e8f43
Our direct addresses:
        78.xxx.xxx.xxx:48801 (type: Stun)
        192.168.xxx.xxx:48801 (type: Local)
Our home relay server:
        https://staging-euw1-1.relay.iroh.network./

Connected to fdeb67bd2995cbdf1844e1170b01848af5e6b13beb4d754ec86dd38b4ecdae3a
Sent: "7b1a7ac46f is saying hello!"
[fdeb67bd29] Connection type changed to: direct(192.168.xxx.xxx:44555)
Received 100.00 MiB in 1.0968s (91.17 MiB/s, time to first byte 0.002208444s, 3518 chunks)

`publish` and `resolve` examples

Set a default relay URL in the publish example. Otherwise an empty node info would be published if not setting any further arguments, which was a major UX footgun. A new argument no-relay-url allows to publish without relay URL.
Cleanup constant names to match those used in the transfer example
Allow to set a custom origin domain in the resolve example

Breaking Changes

Notes & open questions

Change checklist

Self-review.
Documentation updates following the style guide, if relevant.

github-actions · 2025-05-09T09:29:05Z

Documentation for this PR has been generated and is available at: https://n0-computer.github.io/iroh/pr/3296/docs/iroh/

Last updated: 2025-05-12T07:31:16Z

…rvers

flub

Very nice!

flub · 2025-05-12T08:00:57Z

iroh/examples/transfer.rs

@@ -404,3 +472,16 @@ fn parse_byte_size(s: &str) -> Result<u64> {
    let cfg = parse_size::Config::new().with_binary();
    cfg.parse_size(s).map_err(|e| anyhow::anyhow!(e))
 }
+
+fn watch_conn_type(endpoint: &Endpoint, node_id: NodeId) -> AbortOnDropHandle<()> {
+    let mut stream = endpoint.conn_type(node_id).unwrap().stream();


Not a PR review comment, rather out API here isn't very nice. It seems this Result that is unwrapped only exists because the NodeId might not be known.

At the very least this should be an Option, not a result.

Why not make it return pending forever? Or rather until that node becomes known. This could be either a footgun or a nice feature that you can start watching before the NodeId is known to the magic socket. And because a NodeId can be "forgotten" by the magic socket, and that doesn't suddenly result in an error at this point, it already behaves like this.

Turning this into an issue: #3300

Yeah, this is very true, it should just return a stream that waits for the node id to appear.
Because currently our API does not allow to match all conn type changes, we regularily miss the first change: I tried to create the conn_type stream before calling connect but then the unwrap hits. And if calling after connect (as we do atm) then the first conn type change (from none to relay or mixed) never appears.

Frando · 2025-05-12T08:32:01Z

Note: This needs updates to chuck because the transfer example is used for netsim. Tracking this here: n0-computer/chuck#75

github-actions · 2025-05-12T11:08:21Z

Netsim report & logs for this PR have been generated and is available at: LOGS
This report will remain available for 3 days.

Last updated for commit: 4854b4d

github-actions · 2025-05-12T11:08:21Z

Frando/improve-transfer-example.2d7c528
Perf report:

test	case	throughput_gbps	throughput_transfer
iroh_cust_10gb	1_to_1	2.82	3.13
iroh_cust_10gb	1_to_3	6.92	7.53
iroh_cust_10gb	1_to_5	12.12	13.25
iroh_cust_10gb	1_to_10	16.81	17.86
iroh_cust_10gb	2_to_2	5.43	6.00
iroh_cust_10gb	2_to_4	9.08	9.86
iroh_cust_10gb	2_to_6	11.75	12.62
iroh_cust_10gb	2_to_10	19.21	20.60
iroh_latency_20ms	1_to_1	1.40	2.77
iroh_latency_20ms	1_to_3	4.40	9.17
iroh_latency_20ms	1_to_5	7.32	15.03
iroh_latency_20ms	1_to_10	12.25	21.60
iroh_latency_20ms	2_to_2	2.68	5.14
iroh_latency_20ms	2_to_4	6.00	12.67
iroh_latency_20ms	2_to_6	8.39	16.62
iroh_latency_20ms	2_to_10	14.27	28.60
iroh	1_to_1	1.44	2.93
iroh	1_to_3	4.68	10.33
iroh	1_to_5	7.14	14.52
iroh	1_to_10	12.51	22.43
iroh	2_to_2	2.83	5.61
iroh	2_to_4	5.92	12.30
iroh	2_to_6	8.71	17.85
iroh	2_to_10	14.16	28.37
iroh_latency_200ms	1_to_1	1.43	2.87
iroh_latency_200ms	1_to_3	4.65	10.21
iroh_latency_200ms	1_to_5	7.16	14.47
iroh_latency_200ms	1_to_10	12.17	21.28
iroh_latency_200ms	2_to_2	2.83	5.64
iroh_latency_200ms	2_to_4	5.91	12.25
iroh_latency_200ms	2_to_6	8.33	16.39
iroh_latency_200ms	2_to_10	13.83	27.16
iroh_relay_only	1_to_1	2.80	2.82
iroh_relay_only	1_to_3	4.51	4.52
iroh_relay_only	1_to_5	4.53	4.53
iroh_relay_only	1_to_10	4.32	4.32
iroh_relay_only	2_to_2	3.80	3.82
iroh_relay_only	2_to_4	6.05	6.06
iroh_relay_only	2_to_6	6.58	6.60
iroh_relay_only	2_to_10	8.08	8.10

feat: improve transfer example

f1035b4

chore: clippy

638e0ed

n0bot bot added this to iroh May 9, 2025

github-project-automation bot moved this to 🏗 In progress in iroh May 9, 2025

Frando added 4 commits May 10, 2025 00:03

improve publish and resolve examples

e22819a

feat(transfer example): add dev env arg to use local dns and relay se…

66f2bfc

…rvers

transfer example: cleanups

1cb3dd9

transfer example: further cleanups

2d7c528

Frando changed the title ~~feat: improve transfer example~~ feat: improve transfer, publish, resolve examples May 12, 2025

flub approved these changes May 12, 2025

View reviewed changes

Frando mentioned this pull request May 12, 2025

fix: updates for changed transfer example n0-computer/chuck#75

Merged

flub mentioned this pull request May 12, 2025

Endpoint::conn_type API improvements #3300

Open

Frando enabled auto-merge May 12, 2025 11:34

Frando added this pull request to the merge queue May 12, 2025

Merged via the queue into main with commit 30577d3 May 12, 2025
35 of 37 checks passed

github-project-automation bot moved this from 🏗 In progress to ✅ Done in iroh May 12, 2025

ramfox deleted the Frando/improve-transfer-example branch June 25, 2025 17:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: improve transfer, publish, resolve examples #3296

feat: improve transfer, publish, resolve examples #3296

Uh oh!

Frando commented May 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 9, 2025 •

edited

Loading

Uh oh!

flub left a comment

Uh oh!

flub May 12, 2025

Uh oh!

flub May 12, 2025

Uh oh!

Frando May 12, 2025

Uh oh!

Frando commented May 12, 2025

Uh oh!

github-actions bot commented May 12, 2025 •

edited

Loading

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

Uh oh!

Uh oh!

feat: improve transfer, publish, resolve examples #3296

feat: improve transfer, publish, resolve examples #3296

Uh oh!

Conversation

Frando commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

transfer example:

publish and resolve examples

Breaking Changes

Notes & open questions

Change checklist

Uh oh!

github-actions bot commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flub left a comment

Choose a reason for hiding this comment

Uh oh!

flub May 12, 2025

Choose a reason for hiding this comment

Uh oh!

flub May 12, 2025

Choose a reason for hiding this comment

Uh oh!

Frando May 12, 2025

Choose a reason for hiding this comment

Uh oh!

Frando commented May 12, 2025

Uh oh!

github-actions bot commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented May 12, 2025

Uh oh!

Uh oh!

Uh oh!

Frando commented May 9, 2025 •

edited

Loading

`transfer` example:

`publish` and `resolve` examples

github-actions bot commented May 9, 2025 •

edited

Loading

github-actions bot commented May 12, 2025 •

edited

Loading