Skip to content

index reading #294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 57 commits into from
Jan 12, 2022
Merged

index reading #294

merged 57 commits into from
Jan 12, 2022

Conversation

Byron
Copy link
Member

@Byron Byron commented Jan 8, 2022

Tasks

  • basic test case for most simple of index files/figure out how to get test cases
  • stop using FileBuffer due to lack of maintenance and shortcomings in windows
  • parse header
  • parse EOIE (and refactor to get rid of struct)
  • parse TREE extension
  • parse entries V2
  • load extensions threaaded/unthreaded machinery
  • parse entries V4
  • parse IEOT extension
  • actual multi-threaded tests (like in git-pack crate)
  • use multi-threading as intended

Related to #293

Byron added 30 commits January 8, 2022 12:51
Note that in-code we must make sufficiently clear where a particular
fixture is coming from, or we name it after the test and file right
away.
It deals with comparing items from the work tree and the index,
and is generally what makes use of exclude specificiations.
It should be easy enough to learn from git tests to generate whichever
kind of index we need.
This is now sufficiently well implemented in the standard library.
It's sufficiently well supported using the standard library now.
For now the data structure is just 'as-written' and we see what
needs to change there as we have to maintain it.
Byron added 27 commits January 10, 2022 14:58
…even though it doesn't work yet as the flags don't pass an assertion.
Now it works more, but for some reason we don't see the trailer
checksum.
It seems extensions consume too much.
This leads to the first seemingly correct parsing of simple index files.
It's here only so that we can share the code across crates, for now
without any feature toggles.
Unfortunately we are a little more inefficient there as we have to copy
the shared portion into a buffer before we can use these bytes to extend
the backing storage with. Fair enough, it's most definitely not
measurable.
Now for actually using it, that needs some work.
This allows more delicate threading control like is required for the
index.
This iterator makes possible identifies results using a sequence id
and returns only consecutive items.

Use it to collect unordered results produced by threads.
It's advantage to collecting yourself and sorting is the potential
for a smaller memory footprint of in-flight results, one doesn't
have to collect them all for ordering, necessarily.
@Byron Byron merged commit 995994a into main Jan 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant