Added StreamUpload #59

cnlangzi · 2021-05-27T06:57:04Z

Added StreamUpload with more clean design with parts

load/save progress
split data with ErasureEncoder, encrypt it if need
processUpload
processWriteMarker
processCommit

Sriep

Do we want to be including white papers in the project, rather than a link to the whitepaper? Along with an explanation as to why it's relevant.

docs/merkle/streaming-merkle-hasher.txt

Sriep · 2021-05-31T15:11:15Z

docs/merkle/streaming-merkle-hasher.txt

+commit to that point and easily roll back if there is an interruption in
+the upload.
+
+In the code that I had sent previously, this would be at line 61 where


Give a link; Line 61 of what?

It is an email from Tom's forwarded by Saswata. just attach it as a reference. So I don't make any change on it.

The problem is that most readers will not understand this. I think you should either turn this into a *.md and reformat it yourself or create a docs/merkle/streaming-merkle-hasher.md where you introduce this file and docs/merkle/streaming-merkle-hasher.node.

BTW is this the reference to streaming-merkle-hasher.node, because if so its by no mean clear and you should replace In the code that I had sent previously, this would be at line 61 with something like (use 0chain repository)

In [streaming-merkle-hasher.node line 61](https://github.com/cnlangzi/gosdk/blob/featur/continuous_upload/docs/merkle/streaming-merkle-hasher.node#L61)

Alternately you could say in the .md file, that the line 61 mentioned at the end of the letter is here and then give the link.

streaming-merkle-hasher.txt is original content of Tom's email. I think It is unnecessary to update it. It has been implemented in stream_merkle_hasher.go. It is safe to remove them now.
And @guruhubb maybe better to explain it than me.

Sriep · 2021-05-31T15:12:45Z

docs/merkle/streaming-merkle-hasher.node

@@ -0,0 +1,122 @@
+"use strict";


What is the purpose of this streaming-merkle-hasher.node file?

It is an example code in nodejs. Please check streaming-merkle-hasher.txt. Tom explains it there.

Where exactly? I found a reference in stream_merkle_hasher.go, is this an algorithm you have converted from javascript to go?

Does this file exist in an external GitHub repository? If so, I think a link to the file in its home repository would provide better context.

Yes. It is written in go with some improvement in stream_merkle_hasher.go

The nodejs is written by Tom. They are forwarded by @guruhubb. just add it here as a reference. It is safe to remove now. I will remove them.

zboxcore/encoder/erasurecode.go

cnlangzi · 2021-06-02T07:35:38Z

I am testing StreamUpload. so some code maybe be changed. please review it after I get test passed. thanks.

cnlangzi · 2021-06-11T13:56:17Z

FYI: devserver is linked to 0chain/blobber#145 . It is basic feature now. full feature will be followed in 0chain/blobber#145.

cnlangzi · 2021-06-11T15:43:32Z

@Sriep please help me to review PR if you get some time left , thanks.

Sriep

For unit tests you can check out 0chain/0chain#325. Also recent unit tests in zboxcore/sdk might help https://github.com/0chain/gosdk/blob/master/zboxcore/sdk/filestatsworker_test.go

core/util/httpnet.go

core/util/httpnet_context.go

core/util/devserver.yml

core/util/httpnet_middleware.go

Sriep · 2021-06-11T17:45:25Z

docs/uml/upload.puml

+FileStore <- FileSystem: FileRef with merkle leave hash
+StorageHandler <- FileStore: FileOutputData
+StorageHandler -> AllocationChangeProcessor: Add `InitFileChange`
+UploadHandler <- StorageHandler: UploadResult


Is this all happening in gosdk? If this for example some of these participants are in the blobber maybe you could indicate this.

That is done in zbox and blobber with some update. I will update this UML. do you suggest where it should be put ?

Wherever you are using it. Is this the same file you put in the blobber PR description?

Yes. the UML diagram is built based on upload.puml. do you know https://plantuml.com/?

Sriep · 2021-06-11T17:53:13Z

zboxcore/sdk/stream_upload_blobber.go

+
+}
+
+func (sb *StreamUploadBobbler) processUpload(su *StreamUpload, chunkIndex int, fileBytes, thumbnailBytes []byte, isFinal bool, wg *sync.WaitGroup) {


Please can we have unit tests for new functions.

Sriep · 2021-06-11T17:55:19Z

zboxcore/sdk/stream_upload_option.go

+type StreamUploadOption func(su *StreamUpload)
+
+// WithThumbnail add thumbnail. stream mode is unnecessary for thumbnail
+func WithThumbnail(buf []byte) StreamUploadOption {


Please can we have unit tests for new functions.

core/util/httpnet_middleware.go

Sriep · 2021-06-11T18:05:31Z

zboxcore/zboxutil/http.go

@@ -354,6 +354,25 @@ func NewListRequest(baseUrl, allocation string, path string, auth_token string)
 	return req, nil
 }

+// NewUploadRequestWithMethod create a http reqeust of upload
+func NewUploadRequestWithMethod(baseURL, allocation string, body io.Reader, method string) (*http.Request, error) {


There should be a unit test to confirm NewUploadRequestWithMethod returns the right http.Request. I know no one else has done it, but that's now reason why this PR should make the same mistake.

zboxcore/zboxutil/http.go

cnlangzi · 2021-06-11T21:07:01Z

For unit tests you can check out 0chain/0chain#325. Also recent unit tests in zboxcore/sdk might help https://github.com/0chain/gosdk/blob/master/zboxcore/sdk/filestatsworker_test.go

@Sriep devserver is full feature blockchain mock for local development(it is not for unit tests). I have got them dropped from this PR. let us move and follow it on 0chain/blobber#145 thanks.

Sriep · 2021-06-12T07:55:56Z

docs/merkle/streaming-merkle-hasher.txt

+
+2) Otherwise:
+  2.a) set H to hash(HashNodes[index], H)
+  2.b) delete HashNodes[index] (i.e. set this position to empty)


Add comma, i.e. should be i.e.,

Sriep · 2021-06-12T08:12:26Z

docs/merkle/streaming-merkle-hasher.node

@@ -0,0 +1,122 @@
+"use strict";


Where exactly? I found a reference in stream_merkle_hasher.go, is this an algorithm you have converted from javascript to go?

Does this file exist in an external GitHub repository? If so, I think a link to the file in its home repository would provide better context.

Sriep · 2021-06-12T08:21:16Z

docs/merkle/streaming-merkle-hasher.txt

+commit to that point and easily roll back if there is an interruption in
+the upload.
+
+In the code that I had sent previously, this would be at line 61 where


The problem is that most readers will not understand this. I think you should either turn this into a *.md and reformat it yourself or create a docs/merkle/streaming-merkle-hasher.md where you introduce this file and docs/merkle/streaming-merkle-hasher.node.

BTW is this the reference to streaming-merkle-hasher.node, because if so its by no mean clear and you should replace In the code that I had sent previously, this would be at line 61 with something like (use 0chain repository)

In [streaming-merkle-hasher.node line 61](https://github.com/cnlangzi/gosdk/blob/featur/continuous_upload/docs/merkle/streaming-merkle-hasher.node#L61)

Alternately you could say in the .md file, that the line 61 mentioned at the end of the letter is here and then give the link.

cnlangzi · 2021-06-14T23:05:31Z

ignore it first. thanks.

cnlangzi added 2 commits May 27, 2021 14:45

feat(util):added StreamingMerkleHasher

cc37116

fix(merkle): renamed StreamingMerklHasher with StreamMerklHasher

b38e676

Sriep suggested changes May 31, 2021

View reviewed changes

cnlangzi added 4 commits June 2, 2021 15:21

feat(upload):added StreamUpload

e47708f

Merge branch '0chain:master' into featur/continuous_upload

481d138

fix(erasurecode): removed TODO

e84384d

fix(docs): removed whitepaper

1b9db13

cnlangzi changed the title ~~feat(util):added StreamingMerkleHasher~~ [WIP]: Added StreamUpload Jun 2, 2021

cnlangzi added 7 commits June 3, 2021 11:26

fix(upload): updated fields on UploadFormData

d991b62

fix(httpnet):added mock feature on httpnet library

96edb46

fix(httpnet):print statuscode for PASS reqeust

cf86dca

fix(upload): fixed bugs on read shards from fileReader

b68e232

fix(upload):fixed actual hash of file and thumbnail

49d310a

fix(upload):0chain/blobber#111 fixed bugs on hash part

3e4af36

fix(git): merged latest code from master

8b60fdc

cnlangzi changed the title ~~[WIP]: Added StreamUpload~~ Added StreamUpload Jun 11, 2021

cnlangzi requested a review from Sriep June 11, 2021 15:42

Sriep suggested changes Jun 11, 2021

View reviewed changes

fix(upload):#111 drop devserver part

ab34823

Sriep suggested changes Jun 12, 2021

View reviewed changes

cnlangzi mentioned this pull request Jun 12, 2021

Feature/stream upload #106

Closed

cnlangzi closed this Jun 14, 2021


		}

		func (sb StreamUploadBobbler) processUpload(su StreamUpload, chunkIndex int, fileBytes, thumbnailBytes []byte, isFinal bool, wg *sync.WaitGroup) {

Added StreamUpload #59

Added StreamUpload #59

Uh oh!

Conversation

cnlangzi commented May 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sriep left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sriep Jun 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cnlangzi Jun 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cnlangzi commented Jun 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cnlangzi commented Jun 11, 2021

Uh oh!

cnlangzi commented Jun 11, 2021

Uh oh!

Sriep left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cnlangzi commented Jun 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sriep Jun 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cnlangzi commented Jun 14, 2021

Uh oh!

Uh oh!

cnlangzi commented May 27, 2021 •

edited

Loading

Sriep left a comment •

edited

Loading

Sriep Jun 12, 2021 •

edited

Loading

cnlangzi Jun 12, 2021 •

edited

Loading

cnlangzi commented Jun 2, 2021 •

edited

Loading

cnlangzi commented Jun 11, 2021 •

edited

Loading

Sriep Jun 12, 2021 •

edited

Loading