Split traits to get image bytes and metadata bytes

[kylebarron]

This allows for separate caching/buffering of image data and metadata.

Closes #78 . I think this probably also closes #76

cc @feefladder what do you think?

[feefladder]

I think it's a good change, but the metadata bytes differ greatly from COG to COG, from a few kB to a few MB, so it's impossible for any middleware to be aware of that, unless we tell it how big tag data is.

I do think it's a good change, please don't misunderstand, but here's the point:

• africa-scale 30m maps: 4-5MB
• small-scale heightmap: 32 kB

What middleware could possibly account for both of these files? - I think none, unless #76 is landed. In #74, and the corresponding example, I did a:

PrefetchReader<(Metadata)CacheReader<ReqwestReader>>, which worked quite well (most COG ifds read in 2 requests). The only problem there was that CacheReader also doubled and cached the prefetch, which is a bit pointless. That would be solved with this pr, with my proposed changes.

[kylebarron]

unless we tell it how big tag data is

Who is we? The developer at compile time or the end user at runtime?

What middleware could possibly account for both of these files?

Sure you can; you can have an exponential metadata reader. Start with 32kb; if you end up needing more than 32kb while still reading metadata you read the next 256kb. If you need more than that you read the next 2MB, and so on. This is still all at the AsyncFileReader layer.

I think none, unless #76 is landed

#76 only works on a single tag at a time, so I don't see how it solves the problem.

[kylebarron]

PrefetchReader<(Metadata)CacheReader<ReqwestReader>>

With this PR, we can have a MetadataPrefetchReader and a ImageCacheReader, so that there's no double fetching or caching going on.

[kylebarron]

Sure you can; you can have an exponential metadata reader. Start with 32kb; if you end up needing more than 32kb while still reading metadata you read the next 256kb. If you need more than that you read the next 2MB, and so on. This is still all at the AsyncFileReader layer.

Was chatting with @vincentsarago and apparently this is roughly the same approach that GDAL takes to read header metadata

[feefladder]

unless we tell it how big tag data is

Who is we? The developer at compile time or the end user at runtime?

Dev at compile time

I think none, unless #76 is landed

#76 only works on a single tag at a time, so I don't see how it solves the problem.

Because of how a file is laid out in a COG and we're reading the full-res image first:

all calculations are in bytes, since that is what the middleware sees
- TileOffsets0 (Long8 in big, Long in small), largest  -\_ together tile_info_0
- TileByteCounts0 (Long) = 1/2 TileOffsets0 in BigTiff -/

- tile_info_1 ~= 1/4 tile_info_0
- tile_info_2 ~= 1/4 tile_info_1
- continue like that

sum to infinity = geometric series => tile_info_0/(1-1/4) = 4/3*tile_info_0
- bigtiff: tile_info_0 = 3/2*TileOffsets0 => total_bytes ~= 2*TileOffsets0
- smalltiff: tile_info_0 = 2*TileOffsests0 => total_bytes ~= 8/3*TileOffsets0

Since we're only really expecting to have large tile_info arrays in BigTiff, I went with the BigTiff assumption.

Sure you can; you can have an exponential metadata reader. Start with 32kb; if you end up needing more than 32kb while still reading metadata you read the next 256kb. If you need more than that you read the next 2MB, and so on. This is still all at the AsyncFileReader layer.

Was chatting with @vincentsarago and apparently this is roughly the same approach that GDAL takes to read header metadata

I think #74 does better than exponential prefetching in limiting the number of requests and not over-estimating if we're almost at the end of tile_info.

But in any case, my proposed changes can also not be made, and then the logic for the MetadataCacheReader does both the initial prefetch and the subsequent estimates.

[feefladder]

aka LGTM, but please also #76 :)