Skip to content
This repository was archived by the owner on Feb 12, 2024. It is now read-only.
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: ipfs/js-ipfs
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: ipfs-http-server@0.1.4
Choose a base ref
...
head repository: ipfs/js-ipfs
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: ipfs-http-server@0.2.0
Choose a head ref
  • 11 commits
  • 190 files changed
  • 5 contributors

Commits on Dec 18, 2020

  1. chore: broken links in docs (#3457)

    Fixes #3031
    Gozala authored Dec 18, 2020
    Configuration menu
    Copy the full SHA
    18d6359 View commit details
    Browse the repository at this point in the history
  2. chore: refactor common types (#3449)

    Addresses #3442 (comment) by refactoring some of the common types used by root APIs as per #3413 
    
    Co-authored-by: achingbrain <alex@achingbrain.net>
    Gozala and achingbrain authored Dec 18, 2020
    Configuration menu
    Copy the full SHA
    34e1492 View commit details
    Browse the repository at this point in the history
  3. feat: add grpc server and client (#3403)

    Adds a server running a gRPC endpoint over websockets running on port 5003, a `ipfs-grpc-client` module to access the server and a `ipfs-client` module that uses the gRPC client with HTTP fallback.
    
    This is to solve shortcomings and limitations of the existing HTTP API and addresses the concerns raised in the 'Streaming HTTP APIs and errors, y u no work?' session we had at IPFS team week in NYC.
    
    ## Key points
    
    1. Enables full duplex communication with a remote node
    
    When making an HTTP request in the browser, a [FormData][] object must be created. In order to add all the values to the FormData object, an incoming stream must be consumed in its entirety before the first byte is sent to the server.
    
    This means you cannot start processing a response before the request has been sent, so you cannot have full-duplex communication between client and server over HTTP. This seems unlikely to change in the near future.
    
    With a websocket transport for gRPC-web, individual messages can be sent backwards and forwards by the client or the server enabling full-duplex communication.  This is essential for things like progress events from `ipfs.add` in the short term, and exposing the full stream capabilities of libp2p via remote client in the long term.
    
    2. Enables streaming errors
    
    The existing HTTP API sends errors as HTTP trailers.  No browser supports HTTP trailers so when a stream encounters an error, from the client's point of view the stream just stops with no possibility of finding out what happened.
    
    This can also mask intended behaviour cause users to incorrectly interpret the API. For example if you specify a timeout to a DHT query and that timeout is reached, in the browser the stream ends without an error and you take away the results you've received thinking all is well but on the CLI the same operation results in a non-zero exit code.
    
    A websocket transport has no restrictions here, since full-duplex communication is possible, errors can be received at any time.
    
    3. Listens on websockets with no HTTP fallback
    
    gRPC-web exists and is a way of exposing a gRPC service over HTTP.  Whereas gRPC supports four modes (unary, e.g. one request object and one response object, client streaming, server streaming and bidirectional streaming), gRPC-web only supports [unary and server streaming](https://github.com/grpc/grpc-web#wire-format-mode).  This is due to limitations of the web platform mentioned above and doesn't give us anything over our existing HTTP API.
    
    The gRPC-web team are evaluating several options for client and bidirectional streaming, all of which require new capabilities to be added to browsers and none of which will be available in a reasonable time frame.
    
    Notably they have [no plans to use websockets](https://github.com/grpc/grpc-web/blob/master/doc/streaming-roadmap.md#issues-with-websockets) as a transport, even though it solves the problems we have today.
    
    The team from [improbable](https://improbable.io/) maintain a [gRPC-web-websockets bridge](https://github.com/improbable-eng/grpc-web) which the client added by this PR is compatible with.  Their bridge also has a go implementation of a [reverse proxy](https://github.com/improbable-eng/grpc-web/tree/master/go/grpcwebproxy) for use with gRPC servers to turn them into gRPC-web servers with an optional websocket transport.
    
    My proposal is to embrace the use of websockets to solve our problems right now, then move to whatever streaming primitive the gRPC-web team settle on in the years to come.
    
    As implemented there's only websockets here and no HTTP fallback as the existing HTTP API works fine for unary operations so there's little to be gained by blocking this work on reimplementing the whole of the HTTP API in gRPC-web, and the client can pick and choose which API it'll use per-call.
    
    By running the websocket server on a different port to the existing HTTP API it gives us room to add gRPC-web fallback for the API if we find that useful.
    
    4. Has protobuf definitions for all requests/responses
    
    See the [ipfs-grpc-protocol](https://github.com/ipfs/js-ipfs/tree/feat/add-grpc-server-and-client/packages/ipfs-grpc-protocol) module, which contains definitions for API requests/reponses.
    
    They've been ported from the existing API and will need some checking.
    
    The [ipfs-grpc-server/README.md](https://github.com/ipfs/js-ipfs/blob/feat/add-grpc-server-and-client/packages/ipfs-grpc-server/README.md) has a rundown of the websocket communication protocol that was ported from [improbable-eng/grpc-web](https://github.com/improbable-eng/grpc-web).
    
    5. Options as metadata
    
    When making a request, metadata is sent during the preamble - these take the form of a string identical to HTTP headers as the initial websocket message - I've used this mechanism to send the options for a given invocation.
    
    Notably these are not defined as a protocol buffer, just an unspecified list of simple key/value pairs - maybe they should be to ensure compatibility between implementations?
    
    This will be trivial in the implementation in the PR as it contains a server implementation too but to do it in go will require patching or forking the improbable gRPC proxy.
    
    6. Errors as metadata
    
    Similar to the existing HTTP API, message trailers are used to send errors.  Four fields are used to re-construct the error on the client side:
    
    | Field | Notes | 
    | ----- | ----- |
    | grpc-status  | 0 for success, 1+ for error |
    | grpc-message | An error message |
    | grpc-stack   | A stack trace with `\n` delimited lines |
    | grpc-code    | A string code such as `'ERROR_BAD_INPUT'` that may be used for i18n translations to show a message to the user in their own language |
    
    Similar to options these fields are unspecified, if a convention is not enough, perhaps they should be specified as a protobuf and the trailer sent as binary?
    
    7. Streams
    
    When sending data as part of an `ipfs.add`, we send repeated messages that contain a path, a content buffer and an index.  The index is used to differentiate between streams - path cannot be used as it could be empty.  Only the first supplied `path` is respected for a given index. On the server separate input streams are created for each file being added.  A file stream is considered closed when an unset or empty content buffer is received.  Ultimately this will allow us to apply backpressure on a per-file basis and read from different file streams in parallel and asymmetrically based on the available server capacity.
    
    8. Performance
    
    Observed performance pegs gRPC-web over websockets as similar to the HTTP Client with pretty much zero optimisation work performed
    
    9. Security
    
    Browsers require TLS for all use of websocket connections to localhost. They do not require it for the loopback address, however, which this PR uses, though loopback means the traffic will not leave the local machine.
    
    The incoming requests start as HTTP requests so have a referer header and user agent so would follow the same restrictions as the existing HTTP API.
    
    Fixes #2519
    Fixes #2838
    Fixes #2943
    Fixes #2854
    Fixes #2864
    
    [FormData]: https://developer.mozilla.org/en-US/docs/Web/API/FormData
    achingbrain authored Dec 18, 2020
    Configuration menu
    Copy the full SHA
    a9027e0 View commit details
    Browse the repository at this point in the history
  4. docs(browsers): remove websocket-star (#3444)

    It is deprecated, no need for it to be on the list
    lidel authored Dec 18, 2020
    Configuration menu
    Copy the full SHA
    7b48f14 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2021

  1. feat: allow passing a http.Agent to ipfs-http-client in node (#3474)

    Right now no `http.Agent` is used for requests made using the http client in node, which means each request opens a new connection which can end up hitting process resource limits which means connections get dropped.
    
    The change here sets a default `http.Agent` with a `keepAlive: true` and `maxSockets` of 6 which is consistent with [browsers](https://tools.ietf.org/html/rfc2616#section-8.1.4) and [native apps](https://developer.apple.com/documentation/foundation/nsurlsessionconfiguration/1407597-httpmaximumconnectionsperhost?language=objc).
    
    The user can override the agent passed to the `ipfs-http-client` constructor to restore the previous functionality:
    
    ```js
    const http = require('http')
    const createClient = require('ipfs-http-client')
    
    const client = createClient({
      url: 'http://127.0.0.1:5002',
      agent: new http.Agent({
        keepAlive: false,
        maxSockets: Infinity
      })
    })
    ```
    
    Refs: #3464
    achingbrain authored Jan 13, 2021
    Configuration menu
    Copy the full SHA
    fe93ba0 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2021

  1. feat: allow passing a http.Agent to the grpc client (#3477)

    Follows on from #3474 and allows using http.Agents with node.js to
    control the behaviour of the underlying node http client.
    achingbrain authored Jan 14, 2021
    Configuration menu
    Copy the full SHA
    c5f0bc5 View commit details
    Browse the repository at this point in the history

Commits on Jan 15, 2021

  1. chore: update libp2p to 0.30 (#3427)

    Integrates `libp2p@0.30` release.
    
    BREAKING CHANGE: The websocket transport will only dial DNS+WSS addresses - see https://github.com/libp2p/js-libp2p-websockets/releases/tag/v0.15.0
    
    Co-authored-by: Hugo Dias <hugomrdias@gmail.com>
    vasco-santos and hugomrdias authored Jan 15, 2021
    Configuration menu
    Copy the full SHA
    a39e6fb View commit details
    Browse the repository at this point in the history
  2. docs: document the ipfs http client constructor arguments (#3478)

    Documents the existing constructor arguments as I don't think they're written down anywhere right now.
    achingbrain authored Jan 15, 2021
    Configuration menu
    Copy the full SHA
    7ef3adf View commit details
    Browse the repository at this point in the history
  3. chore: re-enable testing on node 15 (#3483)

    Node 15.6.x was released with the fix for hapijs/hapi#4208
    so re-enable testing in CI.
    achingbrain authored Jan 15, 2021
    Configuration menu
    Copy the full SHA
    b7436f2 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    43eff60 View commit details
    Browse the repository at this point in the history
  5. chore: publish

     - interface-ipfs-core@0.143.0
     - ipfs-cli@0.3.0
     - ipfs-client@0.2.0
     - ipfs-core-types@0.2.0
     - ipfs-core-utils@0.6.0
     - ipfs-core@0.4.0
     - ipfs-daemon@0.4.0
     - ipfs-grpc-client@0.1.0
     - ipfs-grpc-protocol@0.1.0
     - ipfs-grpc-server@0.1.0
     - ipfs-http-client@48.2.0
     - ipfs-http-gateway@0.2.0
     - ipfs-http-server@0.2.0
     - ipfs-message-port-client@0.4.4
     - ipfs-message-port-protocol@0.5.0
     - ipfs-message-port-server@0.5.0
     - ipfs@0.53.0
    achingbrain committed Jan 15, 2021
    Configuration menu
    Copy the full SHA
    09f6f0c View commit details
    Browse the repository at this point in the history
Loading