LFS: failed concurrent upload deletes the MetaObject created by a successful retry, silently orphaning content (404 despite content in store)

[sasuw]

Description

When a git-lfs client stalls during an upload and transparently retries (standard git-lfs behavior via its activity timeout), the server-side error cleanup of the abandoned first PUT can run after the retry PUT has already succeeded. That cleanup unconditionally deletes the lfs_meta_object row for the oid — including the row just created by the successful retry.

Result: the pointer is in git history, the content file exists in the LFS content store, but there is no meta object row → all subsequent downloads of that oid return 404, and subsequent pushes are told the object exists so they never re-upload it. The push that triggered the bug reports success to the user, so the data loss is completely silent and is typically discovered much later, on the first fresh clone.

Additionally, in this state the content file is "orphaned" from the database's point of view, so running orphaned-content cleanup would permanently delete the data.

Timeline from server logs (Debug level, redacted)

Push of 3 LFS objects (~5 MB, ~2.8 GB, ~3.2 GB) over a connection that stalled mid-transfer. git-lfs 3.7.1 cancelled the stalled transfers and retried automatically; the user never interrupted anything.

22:15:01 completed POST /<owner>/<repo>.git/info/lfs/objects/batch 200 OK
         (three PUTs start, then stall)
22:22:56 completed POST /<owner>/<repo>.git/info/lfs/objects/batch 200 OK          <- retry batch
22:22:56 completed PUT  .../objects/a83317b5…/5223725 200 OK in 430.2ms            <- RETRY SUCCEEDS
22:22:56 completed POST /<owner>/<repo>.git/info/lfs/verify 200 OK                 <- verify OK
22:22:59 ContentStore.Put() [E] Whilst putting LFS OID[a83317b5…]:
         Failed to copy to tmpPath: … Error: unexpected EOF                        <- abandoned 1st PUT dies
22:22:59 UploadHandler.1() [E] Error putting LFS MetaObject [a83317b5…] into content store. Error: unexpected EOF
22:22:59 UploadHandler()   [E] Error whilst uploadOrVerify LFS OID[a83317b5…]: unexpected EOF
22:22:59 completed PUT  .../objects/a83317b5…/5223725 500 in 397429.7ms            <- cleanup deletes meta row
22:25:18 completed PUT  .../objects/87e8f57f…/2964341467 200 OK in 142445.5ms      <- big files' retries finish
22:25:32 completed PUT  .../objects/12b298ae…/3455470147 200 OK in 156170.7ms         after cleanup: unaffected

The content file for a83317b5… remained on disk in the content store (mtime matching the successful 22:22:56 retry) while its meta row was gone; downloads 404'd.

Root cause

services/lfs/server.go, UploadHandler: on any error from uploadOrVerify() it unconditionally removes the meta object:

if err := uploadOrVerify(); err != nil {
    ...
    if _, err = git_model.RemoveLFSMetaObjectByOid(ctx, repository.ID, p.Oid); err != nil {
        log.Error("Error whilst removing MetaObject for LFS OID[%s]: %v", p.Oid, err)
    }
    return
}

There is no check whether a concurrent upload of the same oid completed successfully in the meantime (its verify already passed). A failed/cancelled duplicate PUT therefore clobbers the winner's metadata. Verified present in the release/v1.26 branch.

Reproduction sketch

1. Start git push of an LFS object over a connection that stalls (or pause the client process mid-PUT so git-lfs's activity timeout triggers a retry, or run two pushes of the same new oid concurrently and kill one).
2. Let the retry PUT + verify complete while the first PUT's connection is still open.
3. Close/kill the first connection → its handler errors with unexpected EOF → RemoveLFSMetaObjectByOid deletes the row.
4. git lfs fetch of that oid now returns 404; a re-push is skipped because batch reports the object as existing… while git push overall reported success in step 3.

Suggested fix

Make the error cleanup safe against concurrent success, e.g. only remove the meta object if no successfully verified content exists for the oid (or track per-request ownership of the meta row / take the row's creation into account instead of deleting blindly by (repo, oid)).

Workaround / detection

Recovery (before any orphan cleanup is run!): re-create the meta row by re-pushing the object from a machine that has it: git lfs push --object-id origin <oid>.

Detection script (batch API only, downloads nothing) to verify all LFS objects referenced in a checkout are actually downloadable:

#!/usr/bin/env bash
# lfs_verify_uploads.sh - verify that every LFS object referenced in the
# current checkout is actually downloadable from the LFS server.
# Uses only the batch API; nothing is downloaded.
# Run inside a repo clone. Optional basic auth: LFS_AUTH="user:token".
set -euo pipefail

endpoint=$(git lfs env | sed -n 's/^Endpoint=\([^ ]*\).*/\1/p')
[ -n "$endpoint" ] || { echo "no LFS endpoint found" >&2; exit 2; }

req=$(git lfs ls-files --json |
  jq '{operation:"download", transfers:["basic"],
       objects:([.files[] | {oid:.oid, size:.size}] | unique)}')

resp=$(curl -sS ${LFS_AUTH:+-u "$LFS_AUTH"} \
  -X POST "$endpoint/objects/batch" \
  -H 'Accept: application/vnd.git-lfs+json' \
  -H 'Content-Type: application/vnd.git-lfs+json' \
  -d "$req")

echo "$resp" | jq -r '.objects[] | .oid[0:12] + "  " +
  (if .error then "MISSING: \(.error.code) \(.error.message)" else "ok" end)'

echo "$resp" | jq -e '[.objects[] | select(.error)] | length == 0' >/dev/null

Environment

• Gitea 1.26.4 (binary install, PostgreSQL, local LFS content store)
• Client: git-lfs 3.7.1, git 2.53.0, Linux
• Can it be reproduced on Gitea demo site: not attempted (requires controlling a mid-transfer stall)