DataEntry and BlobStore Options #550
Conversation
Affie
force-pushed
the
maint/20Q3/datablob_concepts
branch
from
2d038ca to
91c1245
Compare
Codecov Report
@@ Coverage Diff @@
## master #550 +/- ##
==========================================
+ Coverage 74.14% 79.49% +5.34%
==========================================
Files 34 33 -1
Lines 2054 2126 +72
==========================================
+ Hits 1523 1690 +167
+ Misses 531 436 -95
Continue to review full report at Codecov.
|
| mydata = Dict(:soundBite => randn(Float32, 10000), :meta => "something about lazy foxes and fences.") | ||
|
|
||
| # store the data | ||
| addDataEntry!( fg, :x0, datastore, :SOUND_DATA, "application/json", Vector{UInt8}(JSON2.write( mydata )) ) |
| Related | ||
|
|
||
| addDataEntry!, addData!, fetchData, fetchDataEntryElement | ||
| """ |
There was a problem hiding this comment.
if we deprecating this, what is the direct replacement for this documented example in the new API -- specifically the combo?
addDataEntry!( fg, :x0, datastore, :SOUND_DATA, "application/json", Vector{UInt8}(JSON2.write( mydata )) )
entry, rawData = fetchData(fg, :x0, datastore, :SOUND_DATA)There was a problem hiding this comment.
I don't know what default to use yet. For persistence: the FolderStore, for quick and easy (not persisted or copied to the cloud) I like the in-memory data entry.
We currently have:
dataset1 = rand(UInt8, 1000)
ade,adb = addData!(InMemoryDataEntry, dfg, :x1, :random, dataset1)
#or
ade,adb = addData!(FileDataEntry, dfg, :x1, :random, "/tmp/dfgFilestore", dataset1)
#or
ds = FolderStore{Vector{UInt8}}(:filestore, "/tmp/dfgFilestore")
addBlobStore!(dfg, ds)
ade,adb = addData!(dfg, :filestore, :x1, :random, dataset1)The docs are a bit behind and we need good examples. The main purpose of PR was to have a clean API.
There was a problem hiding this comment.
Direct replacement should be (if the store is not saved in the DFG):
blobstore = FolderStore{Vector{UInt8}}(:filestore, "/tmp/dfgFilestore")
addData!(fg, blobstore, :x0, :SOUND_DATA, Vector{UInt8}(JSON2.write( mydata )), mimeType="application/json")
rawData, entry = getData(fg, blobstore, :x0, :SOUND_DATA)There was a problem hiding this comment.
Sorry for changing the parameter order. The DFG and BlobStore are mutated so they should go first and there were ambiguities with dispatch.
The idea is to only need the CRUD verbs getData/addData!/updateData!/deleteData! and not have other verbs such as fetch.
There was a problem hiding this comment.
CRUD API
- Verbs: get,add,update,delete
- Nouns: Data, DataBlob, DataEntry
There was a problem hiding this comment.
Sorry for changing the parameter order.
Totally fine, I'm just trying to follow
Note: I gave functions new names to avoid having to deprecate everything. It's also temporary in a different file structure to make it easier.
There are basically 2 design concepts: BlobStoreEntry vs <: AbstractDataEntry
BlobStoreEntry has a generic entry with the blobstores for every type.
<: AbstractDataEntry has a custom type with no blobstore.
TODO
Maybe choose oneEDIT: keep both for now, but focus on blobstoreSome test and all methodsExpand tests and methods