Skip to content
This repository was archived by the owner on Aug 12, 2020. It is now read-only.

Builder refactoring, trickle builder and balanced builder #118

Merged
merged 30 commits into from
Jan 11, 2017
Merged

Builder refactoring, trickle builder and balanced builder #118

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
511c746
builder refactoring. trickle builder. balanced builder
pgte Dec 28, 2016
0e3f158
removed unused experimental builder
pgte Dec 28, 2016
6af7458
documented importer options
pgte Dec 28, 2016
977f41b
default builder strategy is now the balanced strategy
pgte Dec 28, 2016
c767848
removed unused test
pgte Dec 28, 2016
7854682
removed superfluous comment
pgte Dec 28, 2016
df48f69
fixed trickle builder
pgte Dec 29, 2016
50c5d35
removed superfluous comment
pgte Dec 29, 2016
cbe2ce4
using options.chunkerOptions for chunker-specific options
pgte Dec 29, 2016
f8b9e80
docs: corrected option name
pgte Dec 29, 2016
fedfc30
fix: error handling in trickle reducer
pgte Dec 29, 2016
8e8d3d6
using pull-pair instead of backpressure-less bespoke pair
pgte Dec 30, 2016
7647657
fixed trickle builder tests
pgte Jan 3, 2017
74482f3
recursive streaming trickle builder
pgte Jan 3, 2017
2b92345
missing dep
pgte Jan 3, 2017
01d8583
some style corrections
pgte Jan 3, 2017
0036314
importing multiple roots yields an error
pgte Jan 3, 2017
02cdefd
reinstated testing importing using flat and balanced strategies
pgte Jan 3, 2017
8ac163c
asserting that root node is one and only one
pgte Jan 3, 2017
e723586
testing import and export using various builder strategies
pgte Jan 3, 2017
b9a01f8
fixed error propagation into push streams
pgte Jan 3, 2017
03f49d4
simplified some iteration logic
pgte Jan 3, 2017
fedbe5f
default for maximum children pre node is 174
pgte Jan 7, 2017
180b808
by default, only reduces one leaf to self if specific option is present
pgte Jan 8, 2017
937c292
test results reflect new default config
pgte Jan 8, 2017
0f706df
testing against big files genearted from a pseudo random byte stream gen
pgte Jan 9, 2017
0d3602e
added missing dep
pgte Jan 9, 2017
973c483
removed unnecessary dev dependency
pgte Jan 10, 2017
67fbf87
go-ipfs parity: no root node with single leaf
pgte Jan 10, 2017
ff6cce5
docs: corrected the default maximum number of children nodes
pgte Jan 10, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ In the second argument of the importer constructor you can specify the following
* `trickle`: builds [a trickle tree](https://github.com/ipfs/specs/pull/57#issuecomment-265205384)
* `maxChildrenPerNode` (positive integer, defaults to `172`): the maximum children per node for the `balanced` and `trickle` DAG builder strategies
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now defaults to 174

* `layerRepeat` (positive integer, defaults to 4): (only applicable to the `trickle` DAG builder strategy). The maximum repetition of parent nodes for each layer of the tree.

* `reduceSingleLeafToSelf` (boolean, defaults to `false`): optimization for, when reducing a set of nodes with one node, reduce it to that node.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👌🏽


### Example Exporter

Expand Down
2 changes: 1 addition & 1 deletion src/builder/balanced/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
const balancedReducer = require('./balanced-reducer')

const defaultOptions = {
maxChildrenPerNode: 172
maxChildrenPerNode: 174
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a comment with a link to https://github.com/ipfs/go-ipfs/blob/master/importer/helpers/helpers.go#L16-L35 so that we remember where this value comes from


module.exports = function (reduce, _options) {
Expand Down
2 changes: 1 addition & 1 deletion src/builder/builder.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ module.exports = function (createChunker, ipldResolver, createReducer, _options)
return callback(new Error('invalid content'))
}

const reducer = createReducer(reduce(file, ipldResolver), options)
const reducer = createReducer(reduce(file, ipldResolver, options), options)

pull(
file.content,
Expand Down
3 changes: 2 additions & 1 deletion src/builder/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ const reducers = {

const defaultOptions = {
strategy: 'balanced',
highWaterMark: 100
highWaterMark: 100,
reduceSingleLeafToSelf: false
}

module.exports = function (Chunker, ipldResolver, flushTree, _options) {
Expand Down
4 changes: 2 additions & 2 deletions src/builder/reduce.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ const CID = require('cids')
const DAGLink = dagPB.DAGLink
const DAGNode = dagPB.DAGNode

module.exports = function (file, ipldResolver) {
module.exports = function (file, ipldResolver, options) {
return function (leaves, callback) {
if (leaves.length === 1) {
if (leaves.length === 1 && options.reduceSingleLeafToSelf) {
const leave = leaves[0]
callback(null, {
path: file.path,
Expand Down
2 changes: 1 addition & 1 deletion src/builder/trickle/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
const trickleReducer = require('./trickle-reducer')

const defaultOptions = {
maxChildrenPerNode: 172,
maxChildrenPerNode: 174,
layerRepeat: 4
}

Expand Down
48 changes: 24 additions & 24 deletions test/test-importer.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ const smallFile = loadFixture(__dirname, 'fixtures/200Bytes.txt')
const baseFiles = {
'200Bytes.txt': {
path: '200Bytes.txt',
multihash: 'QmQmZQxSKQppbsWfVzBvg59Cn3DKtsNVQ94bjAxg2h3Lb8',
size: 211,
multihash: 'QmZU6acAmfBHBa4esfNfSH9G9Sx1h49KGpqYzPvXjTCWFe',
size: 264,
name: '',
leafSize: 200
},
Expand Down Expand Up @@ -68,16 +68,16 @@ const strategyOverrides = {
size: 1335478
},
pim: {
multihash: 'QmUpzaN4Jio2GB3HoPSRCMQD5EagdMWjSEGD4SGZXaCw7W',
size: 1335744
multihash: 'QmSxR2CPcJqE3WVV9rT4mn7ZB3F3NwyyLUABBFb8MZRABf',
size: 1335797
},
'pam/pum': {
multihash: 'QmUpzaN4Jio2GB3HoPSRCMQD5EagdMWjSEGD4SGZXaCw7W',
size: 1335744
multihash: 'QmSxR2CPcJqE3WVV9rT4mn7ZB3F3NwyyLUABBFb8MZRABf',
size: 1335797
},
pam: {
multihash: 'QmVoVD4fEWFLJLjvRCg4bGrziFhgECiaezp79AUfhuLgno',
size: 2671269
multihash: 'QmdQaTo7JUZ1otkpnLuRdmL75PZUUDovVBxEU3XzKCKsJt',
size: 2671322
}
},
trickle: {
Expand All @@ -87,16 +87,16 @@ const strategyOverrides = {
size: 1334657
},
pim: {
multihash: 'QmPAn3G2x2nrq4A1fu2XUpwWtpqG4D1YXFDrU615NHvJbr',
size: 1334923
multihash: 'QmYHiiqpWySwPS4zRCv4976baa9epk2VYY6GaAEWhEboZu',
size: 1334976
},
'pam/pum': {
multihash: 'QmPAn3G2x2nrq4A1fu2XUpwWtpqG4D1YXFDrU615NHvJbr',
size: 1334923
multihash: 'QmYHiiqpWySwPS4zRCv4976baa9epk2VYY6GaAEWhEboZu',
size: 1334976
},
pam: {
multihash: 'QmZTJah1xpG9X33ZsPtDEi1tYSHGDqQMRHsGV5xKzAR2j4',
size: 2669627
multihash: 'QmTWpcXJD4mph83PgXnEGqNXGxM3JgADM2hZmYtWpscCj7',
size: 2669680
}
}

Expand All @@ -111,13 +111,13 @@ module.exports = (repo) => {
}),
foo: {
path: 'foo',
multihash: 'QmQrb6KKWGo8w7zKfx2JksptY6wN7B2ysSBdKZr4xMU36d',
size: 320
multihash: 'QmZcDXF22anrX3vBbDxMThN5BVCpkrg1KAAXhzdJ6S7g6G',
size: 373
},
'foo/bar': {
path: 'foo/bar',
multihash: 'Qmf5BQbTUyUAvd6Ewct83GYGnE1F6btiC3acLhR8MDxgkD',
size: 270
multihash: 'QmQE64CC7KR8uL75WLAXXtZ9Ms1ZXDM9ptaqGbpLfGog52',
size: 323
},
'foo-big/1.2MiB.txt': extend({}, baseFiles['1.2MiB.txt'], {
path: 'foo-big/1.2MiB.txt'
Expand All @@ -135,21 +135,21 @@ module.exports = (repo) => {
}),
pim: {
path: 'pim',
multihash: 'QmNk8VPGb3fkAQgoxctXo4Wmnr4PayFTASy4MiVXTtXqiA',
size: 1328386
multihash: 'QmZ8gnWwvoyYXd4S1WsnQSKP7MhLhnJkEotEnxgkenyaib',
size: 1328439
},
'empty-dir': {
path: 'empty-dir',
multihash: 'QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn',
size: 4
},
'pam/pum': {
multihash: 'QmNk8VPGb3fkAQgoxctXo4Wmnr4PayFTASy4MiVXTtXqiA',
size: 1328386
multihash: 'QmZ8gnWwvoyYXd4S1WsnQSKP7MhLhnJkEotEnxgkenyaib',
size: 1328439
},
pam: {
multihash: 'QmPAixYTaYnPe795fcWcuRpo6tfwHgRKNiBHpMzoomDVN6',
size: 2656553
multihash: 'QmZ4veCEhzr9wPekpfHvYisK1BqaYJJ4pHf2uwGdYgrsYG',
size: 2656606
}
}, strategyOverrides[strategy])

Expand Down