Skip to content
This repository was archived by the owner on Mar 10, 2020. It is now read-only.

Adds wiring for a progress bar #596

Closed
wants to merge 1 commit into from
Closed

Adds wiring for a progress bar #596

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/files/add.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,11 @@ module.exports = (send) => {
return callback(new Error('"files" must be a buffer, readable stream, or array of objects'))
}

const request = { path: 'add', files: files, qs: opts }
const request = { path: 'add', files: files, qs: opts, progress: opts.progress }

// Transform the response stream to DAGNode values
const transform = (res, callback) => DAGNodeStream.streamToValue(send, res, callback)

send.andTransform(request, transform, callback)
})
}
13 changes: 12 additions & 1 deletion src/utils/request-api.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ const getFilesStream = require('./get-files-stream')
const streamToValue = require('./stream-to-value')
const streamToJsonValue = require('./stream-to-json-value')
const request = require('./request')
const Transform = require('readable-stream').Transform

// -- Internal

Expand Down Expand Up @@ -160,7 +161,17 @@ function requestAPI (config, options, callback) {
})

if (options.files) {
stream.pipe(req)
if (options.progress && typeof options.progress === 'function') {
const progressStream = new Transform({
transform: (chunk, encoding, cb) => {
options.progress(chunk.byteLength)
cb(null, chunk)
}
})
stream.pipe(progressStream).pipe(req)
} else {
stream.pipe(req)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Kubuxu how does go-ipfs tell the CLI how much 'data' was ingested files? Isn't it returning a ndjson update?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure but I am not sure what exactly you are asking for.

how much 'data' was ingested files?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Kubuxu when running a ipfs add BIGFILE, how does the CLI know how much data has been processed by go-ipfs?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following object is being set by the daemon to the CLI as file is being digested:

type AddedObject struct {
	Name  string
	Hash  string `json:",omitempty"`
	Bytes int64  `json:",omitempty"`
	Size  string `json:",omitempty"`
}

It causes update of the progress bar.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just confirmed with @whyrusleeping, go-ipfs http-api does indeed send a ndjson stream of updates when adding files if --progress is passed.

We can support streaming both in node and in the browser with stream-http (like we already do).

@bmordan your PR needs to change to capture that and instead of measuring on bytes sent, measure on bytes reported back. Could you change to that, please?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@diasdavid OK I'm having a look at this.

} else {
req.end()
}
Expand Down
9 changes: 9 additions & 0 deletions test/files.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,15 @@ describe('.files (the MFS API part)', function () {
})
})

it.only('files.add with progress options', (done) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only needs to be removed.

ipfs.files.add(testfile, {progress: false}, (err, res) => {
expect(err).to.not.exist()

expect(res).to.have.length(1)
done()
})
})

it('files.mkdir', (done) => {
ipfs.files.mkdir('/test-folder', done)
})
Expand Down