git error `invalid data in index` with index.skipHash config

[nekopsykose]

Problem

before 1.68, this worked:

# be in a git repository
$ git init .
# create a junk commit (required; any history)
$ touch x && git add x && git commit -m x
$ mkdir -p a/b && cd a/b
# grab a rust project, that does not have a git repository in it (github tag tarball, .crate, ...)
$ curl https://github.com/sharkdp/hyperfine/archive/refs/tags/v1.16.0.tar.gz | tar -xz --strip-components=1
# build the project somehow
$ cargo build

however, since 1.68, you now get:

>>> hyperfine: Unpacking /var/cache/distfiles/hyperfine-1.16.0.tar.gz...
error: failed to determine package fingerprint for build script for hyperfine v1.16.0 (/home/demon/src/aports/community/hyperfine/src/hyperfine-1.16.0)

Caused by:
  failed to determine the most recently modified file in /home/demon/src/aports/community/hyperfine/src/hyperfine-1.16.0

Caused by:
  failed to determine list of files in /home/demon/src/aports/community/hyperfine/src/hyperfine-1.16.0

Caused by:
  failed to open git index at /home/demon/src/aports/.git/

Caused by:
  invalid data in index - calculated checksum does not match expected; class=Index (10)
>>> ERROR: hyperfine: build failed

of course, there is a workaround:

• git init . inside the source directory

however, i'm not sure if it's necessarily "correct" to break here, or why that is the case.

Steps

# be in a git repository
$ git init .
# create a junk commit (required; any history)
$ touch x && git add x && git commit -m x
$ mkdir -p a/b && cd a/b
# grab a rust project, that does not have a git repository in it (github tag tarball, .crate, ...)
$ curl https://github.com/sharkdp/hyperfine/archive/refs/tags/v1.16.0.tar.gz | tar -xz --strip-components=1
# build the project somehow
$ cargo build

Possible Solution(s)

revert to old behaviour? i don't know the semantics for why this is the case now.

Notes

for background, the reason why the build is in an arbitrary git repository is because it is ran inside a ports tree (alpine linux aports in this case), and that is inside git.

as an aside (does not impact the above), GIT_CEILING_DIRECTORIES is set to prevent actual access to the repository metadata (versioning), but it does not affect this issue (whether it is set or not).

my question is- is there a reason cargo now 'requires' git to be clean/new or not exist to succeed and fails inside an existing repository?

Version

$ cargo version --verbose
cargo 1.68.0
release: 1.68.0
host: x86_64-alpine-linux-musl
libgit2: 1.5.0 (sys:0.16.0 vendored)
libcurl: 7.88.1 (sys:0.4.59+curl-7.86.0 system ssl:OpenSSL/3.1.0)
os: Alpine Linux 3.18_alpha20230208 [64-bit]

[ehuss]

Hm, I can't reproduce with those steps. Which version of git are you using? Can you show the output of git config --list? How did you install cargo? Does it happen with the binaries distributed by the rust-lang org?

[nekopsykose]

hm, ok, let me look into it deeper and get all that info and see if it is more niche..,

[weihanglo]

Cannot reproduce on my machine. With rust 1.64 it still compiles. Cargo is installed via rustup on macOS with git 2.39.2.

[pickfire]

I encountered the same issue for helix although I am not in a nested repository. I tested with the latest commit on helix helix-editor/helix@583f6a1.

Maybe it's a different issue?

> cargo install --path helix-term --locked
  Installing helix-term v0.6.0 (/home/ivan/src/pickfire/rs/helix-test/helix-term)
    Updating crates.io index
error: failed to compile `helix-term v0.6.0 (/home/ivan/src/pickfire/rs/helix-test/helix-term)`, intermediate artifacts can be found at `/home/ivan/src/pickfire/rs/helix-test/target`

Caused by:
  failed to determine package fingerprint for build script for helix-loader v0.6.0 (/home/ivan/src/pickfire/rs/helix-test/helix-loader)

Caused by:
  failed to determine the most recently modified file in /home/ivan/src/pickfire/rs/helix-test/helix-loader

Caused by:
  failed to determine list of files in /home/ivan/src/pickfire/rs/helix-test/helix-loader

Caused by:
  failed to open git index at /home/ivan/src/pickfire/rs/helix-test/.git/

Caused by:
  invalid data in index - calculated checksum does not match expected; class=Index (10)

[nekopsykose]

ok, i figured it out. this seems to actually indeed be the gitconfig.

essentially, my gitconfig had feature.manyFiles=true. this itself toggles some options.

the specific option in this case that causes this failure is index.skipHash=true. that sounds like it causes the exact error. from the manpage:

 •   index.skipHash=true speeds up index writes by not computing a
     trailing checksum. Note that this will cause Git versions
     earlier than 2.13.0 to refuse to parse the index and Git
     versions earlier than 2.40.0 will report a corrupted index
     during git fsck.

note that my git version is 2.40.0, but i'm not sure if this is relevant when cargo utilises libgit(?).

an easy reproducer via dockerfile:

FROM alpine:edge

RUN apk upgrade -Ua && apk add git curl tar rustup build-base
RUN rustup-init -y
WORKDIR /test1
RUN git init . && git config user.name s && git config user.email s@s
RUN git config index.skipHash true
RUN touch x && git add . && git commit -m init
RUN mkdir -p a/b && cd a/b && curl -L https://github.com/sharkdp/hyperfine/archive/refs/tags/v1.16.0.tar.gz | tar -xz --strip-components=1
RUN cd a/b && env PATH="$PATH:/root/.cargo/bin" CARGO_REGISTRIES_CRATES_IO_PROTOCOL="sparse" cargo build

[nekopsykose]

so, basically it's git that breaks this. but i will note that i have had these options set for a really long time. so, either git 2.40 changed some behaviour here, or cargo changed what it expects. my guess is the former, but i'm not sure if there's anything to handle here or the resolution is "don't use fancy git features and expect it to work"

[nekopsykose]

if i change only to alpine:3.17 with git 2.38.4, the failure goes away. almost for sure git then..

[nekopsykose]

index.skipHash
    When enabled, do not compute the trailing hash for the index file.
    This accelerates Git commands that manipulate the index, such as
    git add, git commit, or git status. Instead of storing the
    checksum, write a trailing set of bytes with value zero, indicating
    that the computation was skipped.

    If you enable index.skipHash, then Git clients older than 2.13.0
    will refuse to parse the index and Git clients older than 2.40.0
    will report an error during git fsck.

i think the 'earlier than 2.40' here implies they made a breaking change in 2.40. since cargo is using libgit, it has not prepared to handle the error yet, and fails with the above. makes sense for a start?

[pickfire]

essentially, my gitconfig had feature.manyFiles=true. this itself toggles some options.

Ah, I have this enabled too.

After I disabled it and re-clone helix, then cargo install --path helix-term --locked works now. But without git clone or git reset --hard, it seemed to not work.

[nekopsykose]

But without git clone, it seemed to not work.

it's to do with the current 'state'. the moment you enable it, some hashes will be in that intentionally slightly broken state (because stuff gets regenerated)- you'd have to probably run some very specific commands to get it into a 'pure' state again that doesn't fail. cloning just starts fresh anew. (in very wishy-washy non-technical terms)

[ehuss]

Thanks for investigating! I have filed the upstream issue at libgit2/libgit2#6531