Skip to content

fix: Implement random reader for generating data.#780

Merged
deekoder merged 1 commit into
minio:masterfrom
harshavardhana:fix-random-reader
Aug 14, 2017
Merged

fix: Implement random reader for generating data.#780
deekoder merged 1 commit into
minio:masterfrom
harshavardhana:fix-random-reader

Conversation

@harshavardhana
Copy link
Copy Markdown
Member

Fixes #742

@deekoder deekoder requested review from krisis and vadmeste August 9, 2017 16:31
Comment thread functional_tests.go Outdated
currentReadBytes int
}

func (r *randomReader) Read(p []byte) (n int, err error) {
Copy link
Copy Markdown
Member

@vadmeste vadmeste Aug 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add some comments in this function ? @harshavardhana, it is little complicated to read

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure.. will do.

Comment thread functional_tests.go Outdated

if n != int64(len(buf)) {
logger().Fatalf("Error: number of bytes does not match, want %v, got %v\n", len(buf), n)
if n != int64(MinPartSize) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are not using a random data source, then we get our data from 'datafile-65-MB' which means 65MB and not MinPartSize (64MB)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you are right .. looks like it was an existing problem too @vadmeste

Comment thread functional_tests.go
return &randomReader{
TotalBytes: int64(size),
RandomSeedChar: "a",
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could remove code for counting bytes read by composing a simple random reader with io.LimitReader.
E.g,

  return io.LimitReader{R: randomReader{seed: "a"}, N: size}

where

type randomReader struct {
  seed string
}

func (r *randomReader) Read (b []byte) (int, error) {
   return copy(b, bytes.Repeat(r.seed, len(b))), nil
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@harshavardhana harshavardhana force-pushed the fix-random-reader branch 2 times, most recently from 18b23a6 to 68ac3f7 Compare August 11, 2017 23:07
Comment thread functional_tests.go Outdated

// Perform standard FPutObject with contentType provided (Expecting application/octet-stream)
n, err := c.FPutObject(bucketName, objectName+"-standard", fileName, objectContentType)
ny, err := c.FPutObject(bucketName, objectName+"-standard", fileName, objectContentType)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like a typo. s/ny/n

@vadmeste
Copy link
Copy Markdown
Member

@harshavardhana, it looks like functional tests are not passing

$ go run functional_tests.go
...
FATA[0032] Error:Put http://localhost:9000/minio-go-testa0mxfe1pv2yymobng/minio-go-testa0mxfe1pv2yymobngFPutObject-standard?partNumber=1&uploadId=ee43d6b8-bc7e-4f3b-b6bb-6a634459d6b9: Connection closed by foreign host http://localhost:9000/minio-go-testa0mxfe1pv2yymobng/minio-go-testa0mxfe1pv2yymobngFPutObject-standard?partNumber=1&uploadId=ee43d6b8-bc7e-4f3b-b6bb-6a634459d6b9. Retry again.  file=functional_tests.go function:=main.testFPutObject line#=1038

...

Minio server log:

ERRO[0027] Unable to create object part.                 cause=#has incomplete body source=[object-handlers.go:850:objectAPIHandlers.PutObjectPartHandler()] stack=fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:388:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm
ERRO[0027] Unable to create object part.                 cause=#has incomplete body source=[object-handlers.go:850:objectAPIHandlers.PutObjectPartHandler()] stack=fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:388:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm
ERRO[0029] Unable to create object part.                 cause=#has incomplete body source=[object-handlers.go:850:objectAPIHandlers.PutObjectPartHandler()] stack=fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:388:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm
ERRO[0032] Unable to create object part.                 cause=#has incomplete body source=[object-handlers.go:850:objectAPIHandlers.PutObjectPartHandler()] stack=fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:388:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm
ERRO[0036] Unable to create object part.                 cause=#has incomplete body source=[object-handlers.go:850:objectAPIHandlers.PutObjectPartHandler()] stack=fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:388:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm
ERRO[0567] Unable to create object part.                 cause=#has incomplete body source=[object-handlers.go:850:objectAPIHandlers.PutObjectPartHandler()] stack=fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:388:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm
ERRO[0567] Unable to create object part.                 cause=#has incomplete body source=[object-handlers.go:850:objectAPIHandlers.PutObjectPartHandler()] stack=fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:388:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm
ERRO[0569] Unable to create object part.                 cause=#has incomplete body source=[object-handlers.go:850:objectAPIHandlers.PutObjectPartHandler()] stack=fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:388:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm
ERRO[0572] Unable to create object part.                 cause=#has incomplete body source=[object-handlers.go:850:objectAPIHandlers.PutObjectPartHandler()] stack=fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:388:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm
ERRO[0578] Unable to create object part.                 cause=#has incomplete body source=[object-handlers.go:850:objectAPIHandlers.PutObjectPartHandler()] stack=fs-v1-helpers.go:272:fsCreateFile fs-v1-multipart.go:547:fsObjects.PutObjectPart <autogenerated>:388:(*fsObjects).PutObjectPart object-handlers.go:828:objectAPIHandlers.PutObjectPartHandler api-router.go:46:(objectAPIHandlers).PutObjectPartHandler-fm

@harshavardhana
Copy link
Copy Markdown
Member Author

Already opened a bug for this @vadmeste #784 it has nothing to do with this PR.

Copy link
Copy Markdown
Member

@vadmeste vadmeste left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harshavardhana
Copy link
Copy Markdown
Member Author

@krisis any reviews?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants