Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/dictionary.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,5 @@ byteranges
dups
zcash
unavail
bagqbeaawn
bagqbeaa
1 change: 0 additions & 1 deletion packages/interop/src/websites.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ describe('@helia/verified-fetch - websites', () => {
let verifiedFetch: VerifiedFetch

before(async () => {
// 2024-01-22 CID for _dnslink.helia-identify.on.fleek.co
verifiedFetch = await createVerifiedFetch({
gateways: ['http://127.0.0.1:8180'],
routers: ['http://127.0.0.1:8180'],
Expand Down
81 changes: 73 additions & 8 deletions packages/verified-fetch/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -693,16 +693,10 @@ Plugins are executed in a chain (a **plugin pipeline**):

2. **Pipeline Execution:**

- The pipeline repeatedly checks, up to a maximum number of passes (default = 3), which plugins
are currently able to handle the request by calling each plugin’s `canHandle()` method.
- The pipeline checks which plugins can handle the request by calling each plugin’s `canHandle()` method.
- Plugins that have not yet been called in the current run and return `true` for `canHandle()`
are invoked in sequence.
- If a plugin returns a final `Response` or throws a `PluginFatalError`, the pipeline immediately
stops and that response is returned.
- If a plugin returns `null`, it may have updated the context (for example, by
performing path walking), other plugins that said they `canHandle` will run.
- If no plugin modifies the context (i.e. no change to `context.modified`) and no final response is
produced after iterating through all plugins, the pipeline exits and a default “Not Supported”
- If no plugin can handle the request, the pipeline exits and a “Not Supported”
response is returned.

**Diagram of the Plugin Pipeline:**
Expand Down Expand Up @@ -811,6 +805,77 @@ To add your own plugin:

For a detailed explanation of the pipeline, please refer to the discussion in [Issue #167](https://github.com/ipfs/helia-verified-fetch/issues/167).

### Server-Timing

Detailed timing is found in the [Server-Timing](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Server-Timing)
HTTP header that is returned with every response when a resource is requested
with the `withServerTiming` init option set to `true`.

To prevent the header value growing too large, PeerIDs/CIDs are truncated to
their first 10 characters and common strings are abbreviated.

The values you may expect to see are described in the following table. Note
that not all of them may be present in a given response.

Router, block broker and transport abbreviations used in the `desc` fields
follow.

| Timing metric | Elaboration | Detail | Example |
| ------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| d | DNSLink.resolve | Resolving a DNSLink to an IPFS path or IPNS name | `d;dur=0.200` |
| i | IPFS.resolve | Resolving a CID + path to a CID | `i;dur=0.200` |
| n | IPNS.resolve | Resolving an IPNS name to an IPFS path | `n;dur=0.200` |
| p | Provider | A provider was found. The `desc` field contains the routing system that found the provider and the first 10 characters of the PeerId | `p;dur=0.000;desc="h,bagqbeaawn"` |
| f | Find Providers | The total duration of the routing systems Find Providers operation. The `desc` field contains the routing system and how many providers were found | `f;dur=2.000;desc="h,4"` |
| c | Connect | How long it took to connect to a provider. The `desc` field contains the type of provider, the first 10 characters of their PeerId and the transport used | `b;dur=0.000;desc="t,bagqbeaa7n,bafybeigoc,t"` |
| b | Block | How long it took to retrieve a block from the provider once connected. The `desc` field contains the type of provider, the first 10 characters of their PeerId and the first 10 characters of the CID | `b;dur=0.000;desc="t,bagqbeaa7n,bafybeigoc"` |

A full header might look like:

```
i;dur=0.000,p;dur=0.000;desc="h,bagqbeaawn",p;dur=0.000;desc="h,bagqbeaawn",p;dur=1.000;desc="h,bagqbeaa7n",p;dur=1.000;desc="h,bagqbeaa7n",f;dur=1.000;desc="h,4",f;dur=1.000;desc="h,4",f;dur=144.000;desc="l,0",f;dur=144.000;desc="l,0",c;dur=206.000;desc="t,bagqbeaa7n,h",b;dur=1.000;desc="t,bagqbeaa7n,bafybeigoc"
```

Here resolving a CID to a CID+path took less than a millisecond (e.g. a bare
CID was requested).

Two HTTP Gateway providers were found in the routing (`bagqbeaawn` and
`bagqbeaa7n`). They are found twice because two block brokers are configured
(bitswap and trustless-gateway) which both make a routing request (results
are cached internally so the duration to find them the second time differs).

It took 206ms to connect to `bagqbeaa7n` over HTTP, and 1s to retrieve the
block for the CID `bafybeigo` from the Trustless Gateway `bagqbeaa7n`.

All PeerIDs and CIDs above are truncated to 10 characters.

#### Router abbreviations

| Router | Elaboration |
| ------ | --------------------- |
| h | HTTP Gateway |
| l | Libp2p (e.g. Kad-DHT) |

#### Block broker abbreviations

| Block Broker | Elaboration |
| ------------ | ----------------- |
| t | Trustless Gateway |
| b | Bitswap |

#### Transport abbreviations

| Transport | Elaboration |
| --------- | ------------- |
| t | TCP |
| h | HTTP |
| w | WebSockets |
| r | WebRTC |
| d | WebRTC-Direct |
| q | QUIC |
| b | WebTransport |
| u | Unknown |

# Install

```console
Expand Down
28 changes: 15 additions & 13 deletions packages/verified-fetch/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -164,14 +164,14 @@
"release": "aegir release"
},
"dependencies": {
"@helia/block-brokers": "^5.1.0",
"@helia/car": "^5.3.7",
"@helia/block-brokers": "^5.2.1",
"@helia/car": "^5.4.0",
"@helia/delegated-routing-v1-http-api-client": "^6.0.1",
"@helia/dnslink": "^1.1.4",
"@helia/interface": "^6.1.0",
"@helia/ipns": "^9.1.8",
"@helia/routers": "^5.0.2",
"@helia/unixfs": "^7.0.3",
"@helia/dnslink": "^1.2.0",
"@helia/interface": "^6.2.0",
"@helia/ipns": "^9.2.0",
"@helia/routers": "^5.1.0",
"@helia/unixfs": "^7.2.0",
"@ipld/dag-cbor": "^9.2.3",
"@ipld/dag-json": "^10.2.4",
"@ipld/dag-pb": "^4.1.5",
Expand All @@ -182,8 +182,10 @@
"@libp2p/webrtc": "^6.0.8",
"@libp2p/websockets": "^10.1.0",
"@multiformats/dns": "^1.0.6",
"@multiformats/multiaddr": "^13.0.1",
"@multiformats/multiaddr-matcher": "^3.0.2",
"file-type": "^21.1.1",
"helia": "^6.0.18",
"helia": "^6.1.1",
"interface-blockstore": "^6.0.1",
"ipfs-unixfs-exporter": "^15.0.2",
"ipns": "^10.1.3",
Expand All @@ -202,10 +204,10 @@
"uint8arrays": "^5.1.0"
},
"devDependencies": {
"@helia/dag-cbor": "^5.0.0",
"@helia/dag-json": "^5.0.0",
"@helia/http": "^3.0.5",
"@helia/json": "^5.0.0",
"@helia/dag-cbor": "^5.1.0",
"@helia/dag-json": "^5.1.0",
"@helia/http": "^3.1.1",
"@helia/json": "^5.1.0",
"@ipld/car": "^5.4.2",
"@libp2p/crypto": "^5.1.13",
"@libp2p/logger": "^6.2.0",
Expand All @@ -214,7 +216,7 @@
"aegir": "^47.0.24",
"blockstore-core": "^6.1.1",
"browser-readablestream-to-it": "^2.0.9",
"cborg": "^4.2.11",
"cborg": "^5.1.0",
"ipfs-unixfs-importer": "^16.0.1",
"it-all": "^3.0.8",
"magic-bytes.js": "^1.12.1",
Expand Down
73 changes: 72 additions & 1 deletion packages/verified-fetch/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -772,6 +772,77 @@
* - Any thrown error immediately stops the pipeline and returns the error response.
*
* For a detailed explanation of the pipeline, please refer to the discussion in [Issue #167](https://github.com/ipfs/helia-verified-fetch/issues/167).
*
* ### Server-Timing
*
* Detailed timing is found in the [Server-Timing](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Server-Timing)
* HTTP header that is returned with every response when a resource is requested
* with the `withServerTiming` init option set to `true`.
*
* To prevent the header value growing too large, PeerIDs/CIDs are truncated to
* their first 10 characters and common strings are abbreviated.
*
* The values you may expect to see are described in the following table. Note
* that not all of them may be present in a given response.
*
* Router, block broker and transport abbreviations used in the `desc` fields
* follow.
*
* | Timing metric | Elaboration | Detail | Example |
* | ------------------ | --------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
* | d | DNSLink.resolve | Resolving a DNSLink to an IPFS path or IPNS name | `d;dur=0.200` |
* | i | IPFS.resolve | Resolving a CID + path to a CID | `i;dur=0.200` |
* | n | IPNS.resolve | Resolving an IPNS name to an IPFS path | `n;dur=0.200` |
* | p | Provider | A provider was found. The `desc` field contains the routing system that found the provider and the first 10 characters of the PeerId | `p;dur=0.000;desc="h,bagqbeaawn"` |
* | f | Find Providers | The total duration of the routing systems Find Providers operation. The `desc` field contains the routing system and how many providers were found | `f;dur=2.000;desc="h,4"` |
* | c | Connect | How long it took to connect to a provider. The `desc` field contains the type of provider, the first 10 characters of their PeerId and the transport used | `b;dur=0.000;desc="t,bagqbeaa7n,bafybeigoc,t"` |
* | b | Block | How long it took to retrieve a block from the provider once connected. The `desc` field contains the type of provider, the first 10 characters of their PeerId and the first 10 characters of the CID | `b;dur=0.000;desc="t,bagqbeaa7n,bafybeigoc"` |
*
* A full header might look like:
*
* ```
* i;dur=0.000,p;dur=0.000;desc="h,bagqbeaawn",p;dur=0.000;desc="h,bagqbeaawn",p;dur=1.000;desc="h,bagqbeaa7n",p;dur=1.000;desc="h,bagqbeaa7n",f;dur=1.000;desc="h,4",f;dur=1.000;desc="h,4",f;dur=144.000;desc="l,0",f;dur=144.000;desc="l,0",c;dur=206.000;desc="t,bagqbeaa7n,h",b;dur=1.000;desc="t,bagqbeaa7n,bafybeigoc"
* ```
*
* Here resolving a CID to a CID+path took less than a millisecond (e.g. a bare
* CID was requested).
*
* Two HTTP Gateway providers were found in the routing (`bagqbeaawn` and
* `bagqbeaa7n`). They are found twice because two block brokers are configured
* (bitswap and trustless-gateway) which both make a routing request (results
* are cached internally so the duration to find them the second time differs).
*
* It took 206ms to connect to `bagqbeaa7n` over HTTP, and 1s to retrieve the
* block for the CID `bafybeigo` from the Trustless Gateway `bagqbeaa7n`.
*
* All PeerIDs and CIDs above are truncated to 10 characters.
*
* #### Router abbreviations
*
* | Router | Elaboration |
* | ------ | --------------------- |
* | h | HTTP Gateway |
* | l | Libp2p (e.g. Kad-DHT) |
*
* #### Block broker abbreviations
*
* | Block Broker | Elaboration |
* | ------------ | ----------------- |
* | t | Trustless Gateway |
* | b | Bitswap |
*
* #### Transport abbreviations
*
* | Transport | Elaboration |
* | --------- | ------------- |
* | t | TCP |
* | h | HTTP |
* | w | WebSockets |
* | r | WebRTC |
* | d | WebRTC-Direct |
* | q | QUIC |
* | b | WebTransport |
* | u | Unknown |
*/

import { bitswap, trustlessGateway } from '@helia/block-brokers'
Expand Down Expand Up @@ -903,7 +974,7 @@ export interface PluginContext extends ResolveURLResult, Omit<VerifiedFetchInit,
/**
* A callback that receives progress events
*/
onProgress?(evt: ProgressEvent): void
onProgress?(evt: VerifiedFetchProgressEvents): void

/**
* Any async operations should be invoked using server timings to allow
Expand Down
7 changes: 4 additions & 3 deletions packages/verified-fetch/src/url-resolver.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import { CID } from 'multiformats/cid'
import QuickLRU from 'quick-lru'
import { toString as uint8ArrayToString } from 'uint8arrays/to-string'
import { CODEC_LIBP2P_KEY, SESSION_CACHE_MAX_SIZE, SESSION_CACHE_TTL_MS } from './constants.ts'
import { abbreviate } from './utils/abbreviate.ts'
import { applyRedirects } from './utils/apply-redirect.ts'
import { ServerTiming } from './utils/server-timing.ts'
import type { ResolveURLOptions, ResolveURLResult, URLResolver as URLResolverInterface } from './index.ts'
Expand Down Expand Up @@ -124,7 +125,7 @@ export class URLResolver implements URLResolverInterface {
}

private async resolveDNSLink (url: URL, serverTiming: ServerTiming, options?: ResolveURLOptions): Promise<ResolveURLResult | Response> {
const results = await serverTiming.time('dnsLink.resolve', `Resolve DNSLink ${url.hostname}`, this.components.dnsLink.resolve(url.hostname, options))
const results = await serverTiming.time(abbreviate('dnsLink.resolve'), '', this.components.dnsLink.resolve(url.hostname, options))
const result = results?.[0]

if (result == null) {
Expand Down Expand Up @@ -161,7 +162,7 @@ export class URLResolver implements URLResolverInterface {

private async resolveIPNSName (url: URL, serverTiming: ServerTiming, options?: ResolveURLOptions): Promise<ResolveURLResult | Response> {
const peerId = peerIdFromString(url.hostname)
const result = await serverTiming.time('ipns.resolve', `Resolve IPNS name ${peerId}`, this.components.ipnsResolver.resolve(peerId, options))
const result = await serverTiming.time(abbreviate('ipns.resolve'), '', this.components.ipnsResolver.resolve(peerId, options))
const path = normalizePath(`${result.path ?? ''}/${url.pathname}`)

const ipfsUrl = new URL(`ipfs://${result.cid}${path}`)
Expand All @@ -180,7 +181,7 @@ export class URLResolver implements URLResolverInterface {
}

private async resolveIPFSPath (url: URL, serverTiming: ServerTiming, options?: ResolveURLOptions): Promise<ResolveURLResult | Response> {
const walkPathResult = await serverTiming.time('ipfs.resolve', '', this.walkPath(url, options))
const walkPathResult = await serverTiming.time(abbreviate('ipfs.resolve'), '', this.walkPath(url, options))

if (walkPathResult instanceof Response) {
return walkPathResult
Expand Down
57 changes: 57 additions & 0 deletions packages/verified-fetch/src/utils/abbreviate.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
import { HTTP, HTTPS, QUIC, QUIC_V1, TCP, WebRTC, WebRTCDirect, WebSockets, WebSocketsSecure, WebTransport } from '@multiformats/multiaddr-matcher'
import type { Multiaddr } from '@multiformats/multiaddr'

const ABBREVIATIONS: Record<string, string> = {
// operations
'ipfs.resolve': 'i',
'dnsLink.resolve': 'd',
'ipns.resolve': 'n',
'found-provider': 'p',
'find-providers': 'f',
connect: 'c',
block: 'b',

// routers
'http-gateway-router': 'h',
'libp2p-router': 'l',

// block brokers
'trustless-gateway': 't',
bitswap: 'b'
}

export function abbreviate (str: string): string {
return ABBREVIATIONS[str] ?? str
}

export function abbreviateAddress (ma: Multiaddr): string {
if (TCP.exactMatch(ma)) {
return 't'
}

if (HTTP.exactMatch(ma) || HTTPS.exactMatch(ma)) {
return 'h'
}

if (WebSockets.exactMatch(ma) || WebSocketsSecure.exactMatch(ma)) {
return 'w'
}

if (WebRTC.exactMatch(ma)) {
return 'r'
}

if (WebRTCDirect.exactMatch(ma)) {
return 'd'
}

if (QUIC.exactMatch(ma) || QUIC_V1.exactMatch(ma)) {
return 'q'
}

if (WebTransport.exactMatch(ma)) {
return 'b'
}

return 'u'
}
2 changes: 1 addition & 1 deletion packages/verified-fetch/src/utils/server-timing.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,6 @@ export class ServerTiming {
}

add (name: string, description: string, duration: number | string): void {
this.headers.push(`${name};dur=${Number(duration).toFixed(this.precision)};desc="${description}"`)
this.headers.push(`${name};dur=${Number(duration).toFixed(this.precision)}${description === '' ? '' : `;desc="${description}"`}`)
}
}
Loading
Loading