[Inquiry] Error compiling when go modules are vendored

[keongalvin]

Inquiry

When trying to build the example with modules vendored (go mod tidy && go mod vendor) i get an error along the lines of

glide_ffi: No such file or directory

The same code works after removing the vendored directory. This behavior seems like it might be by design of go module system(?)

General advice would be appreciated here

Language

go

Language Version

25.1

Engine Version

No response

Operating System

MacOS, Linux

Additional Technical Information

last few lines in the output of go build -x main.go (main.go from https://github.com/valkey-io/valkey-glide/tree/main/go#standalone-example)

mkdir -p $WORK/b001/exe/
cd .
GOROOT='/usr/local/go' /usr/local/go/pkg/tool/linux_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=xW-AqCU91aP0l23ds1ii/DuYaiW1S3f4ZjC2p47rj/whA61L2R5UxIkDqRApy3/xW-AqCU91aP0l23ds1ii -extld=gcc $WORK/b001/_pkg_.a
# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/gcc -m64 -Wl,--build-id=0xab0ddc099935c18c8a25a0fdaba97c01c2b38ce7 -o $WORK/b001/exe/a.out -Wl,--export-dynamic-symbol=_cgo_panic -Wl,--export-dynamic-symbol=_cgo_topofstack -Wl,--export-dynamic-symbol=crosscall2 -Wl,--export-dynamic-symbol=failureCallback -Wl,--export-dynamic-symbol=pubSubCallback -Wl,--export-dynamic-symbol=successCallback -Wl,--compress-debug-sections=zlib /tmp/go-link-1263405046/go.o /tmp/go-link-1263405046/000000.o /tmp/go-link-1263405046/000001.o /tmp/go-link-1263405046/000002.o /tmp/go-link-1263405046/000003.o /tmp/go-link-1263405046/000004.o /tmp/go-link-1263405046/000005.o /tmp/go-link-1263405046/000006.o /tmp/go-link-1263405046/000007.o /tmp/go-link-1263405046/000008.o /tmp/go-link-1263405046/000009.o /tmp/go-link-1263405046/000010.o /tmp/go-link-1263405046/000011.o /tmp/go-link-1263405046/000012.o /tmp/go-link-1263405046/000013.o /tmp/go-link-1263405046/000014.o /tmp/go-link-1263405046/000015.o /tmp/go-link-1263405046/000016.o /tmp/go-link-1263405046/000017.o /tmp/go-link-1263405046/000018.o /tmp/go-link-1263405046/000019.o /tmp/go-link-1263405046/000020.o /tmp/go-link-1263405046/000021.o /tmp/go-link-1263405046/000022.o /tmp/go-link-1263405046/000023.o /tmp/go-link-1263405046/000024.o /tmp/go-link-1263405046/000025.o /tmp/go-link-1263405046/000026.o /tmp/go-link-1263405046/000027.o /tmp/go-link-1263405046/000028.o /tmp/go-link-1263405046/000029.o /tmp/go-link-1263405046/000030.o /tmp/go-link-1263405046/000031.o -O2 -g -lglide_ffi -lm -L/home/ec2-user/example/vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin/x86_64-unknown-linux-gnu -O2 -g -O2 -g -lpthread -O2 -g -O2 -g -O2 -g -no-pie
/usr/bin/ld: cannot find -lglide_ffi: No such file or directory
collect2: error: ld returned 1 exit status

[jamesx-improving]

Hi @keongalvin , thank you for creating this issue!

I'll try to reproduce that and update you soon.

[jamesx-improving]

Hi @keongalvin ,

Apparently our Go binding is not fully compatible with the go mod vendor command at the moment. The binary files needed for cgo linking under rustbin/ is not being copied to the vendor directory by go mod vendor.

As a temporary workaround, please try to copy the rustbin/ directory over. An example command would be (assuming you're in project root):

cp -r ~/go/pkg/mod/github.com/valkey-io/valkey-glide/go/v2@v2.0.1/rustbin ./vendor/github.com/valkey-io/valkey-glide/go/v2

I understand this is inconvenient as the vendor directory will get wiped out every time you execute go mod vendor, but this is the only workaround we can provide at the moment. Sorry about that.

We'll see what we can do to permanently fix this in our next release (v2.2). Based on my limited research on the topic, not a straightforward one.

Stay turned!

[tdschwarz]

Apparently our Go binding is not fully compatible with the go mod vendor command at the moment. The binary files needed for cgo linking under rustbin/ is not being copied to the vendor directory by go mod vendor.

@jamesx-improving I did a bit of research into this and the reason rustbin/*/libglide_ffi.a isn't getting picked up by go mod vendor is because the go toolchain doesn't recognize the rustbin/ subdirectories as valid go packages and ignores them.

Adding a go/tools.go file with some no-op _ imports and dummy go files in the rustbin arch specific subdirs allows go mod vendor to detect and copy the .a files without the manual cp -r workaround.

tools.go would look something like

//go:build tools

package glide

import (
	_ "github.com/valkey-io/valkey-glide/go/v2/rustbin/aarch64-apple-darwin"
	_ "github.com/valkey-io/valkey-glide/go/v2/rustbin/aarch64-unknown-linux-gnu"
	_ "github.com/valkey-io/valkey-glide/go/v2/rustbin/aarch64-unknown-linux-musl"
	_ "github.com/valkey-io/valkey-glide/go/v2/rustbin/x86_64-apple-darwin"
	_ "github.com/valkey-io/valkey-glide/go/v2/rustbin/x86_64-unknown-linux-gnu"
	_ "github.com/valkey-io/valkey-glide/go/v2/rustbin/x86_64-unknown-linux-musl"
)

I was able to confirm locally that adding tools.go and the marker files to my go mod cache did fix the issue. I'm happy to open a PR, but am not sure whether the preferred approach would be to

1. Update the .gitignore pattern from rustbin/ to rustbin/*/* and !rustbin/*/*.go so the placeholder/marker files are committed to the repo, OR
2. Leave the current .gitignore pattern intact and generate the marker files/tools.go in the Commit and push tag step of go-cd.yml workflow where the .a files are added with git add -f go

Thoughts?

[jamesx-improving]

Hi @tdschwarz , thank you for your effort in investigating this issue!

I appreciate your attention to detail in finding our trick buried in the Go CD workflow, which (the 2nd approach) is what I would recommend.

It'll be great if you could help create the PR. Please let me know if you have capacity to do that.

I'll verify the solution manually once the PR is up, considering it is hard to verify in automatic testing given our tricky implementation.

[tdschwarz]

@jamesx-improving PR is up! Let me know if you need anything else from me, I'm happy to add a bash script to the PR description to manually create the doc.go marker files and tools.go file for testing

[jamesx-improving]

Hi @tdschwarz thank you for the PR!

However, after manually test your approach on my local machine, I couldn't make it work.

Could you elaborate your steps? Like the sequence of your operation, after you add all those files into the go mod cache, what specific go commands did you execute? Moreover, is there any special setting in your consuming project that either directly imports any of those rustbin packages, or has a tools.go of your own?

Sorry but without concrete proof, I can not approve the PR yet.

[tdschwarz]

@jamesx-improving

Our setup (answering your question about "special settings"):

• We only import the public Go API (github.com/valkey-io/valkey-glide/go/v2 and subpackages)
• NO direct imports of rustbin packages in our project
• NO custom tools.go in our project
• No special configuration whatsoever - just a standard Go project using valkey-glide

The sequence that triggers the issue:

# someone updates/adds go deps in our main branch
git pull              # Get changes with updated deps
go mod tidy           # Updates go.mod/go.sum
# gopls complains: "inconsistent vendoring"
go mod vendor         # Skips rustbin/ (no .go files)

gopls expects a complete vendor/ directory when one exists and incomplete vendoring causes "inconsistent vendoring" errors

Our current workaround is a custom script that adjusts file permissions and manually copies .a files from GOMODCACHE into the /rustbin dir inside of the project's vendor/ dir after vendoring

To reproduce:
The PoC script below demonstrates the full cycle - build failure without the fix, then success after applying it:

[4/6] ...
  Build failed:
    ld: library 'glide_ffi' not found
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
  → Cannot build: FFI library missing from vendor/

[6/6] Testing: go mod vendor AFTER fix
  vendor/:    6 .a files, 6 doc.go, 1 tools.go  [OK: rustbin/ vendored]
  ✓ Build successful with vendored FFI libraries

test_vendor_fix.sh:

#!/bin/bash
set -e

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

echo -e "${BLUE}=== Valkey GLIDE Vendoring Fix POC ===${NC}"
echo ""

[ -d "vendor_fix_poc" ] && rm -rf vendor_fix_poc

# Step 1: Create minimal Go project
echo -e "${GREEN}[1/6] Creating test project${NC}"
mkdir -p vendor_fix_poc
cd vendor_fix_poc

cat > main.go << 'EOF'
package main

import (
	"context"
	"fmt"
	"time"
	
	glide "github.com/valkey-io/valkey-glide/go/v2"
	"github.com/valkey-io/valkey-glide/go/v2/config"
)

func main() {
	// We actually have to create a client for the FFI library to be used,
    // triggering linker errors if .a files are missing
	clientConfig := config.NewClientConfiguration().
		WithAddress(&config.NodeAddress{Host: "localhost", Port: 6379}).
		WithRequestTimeout(500 * time.Millisecond)
	
	client, err := glide.NewClient(clientConfig)
	if err != nil {
		fmt.Printf("Expected error creating client (no server): %v\n", err)
		return
	}
	defer client.Close()
	
	ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
	defer cancel()
	
	_, err = client.Ping(ctx)
	if err != nil {
		fmt.Printf("Expected error pinging (no server): %v\n", err)
	}
}
EOF

go mod init vendor-fix-poc > /dev/null 2>&1

# Step 2: Add dependency and tidy
echo -e "${GREEN}[2/6] Adding valkey-glide v2.2.3${NC}"
go mod tidy

# Step 3: Try go mod vendor from GOMODCACHE
echo ""
echo -e "${GREEN}[3/6] Vendoring from GOMODCACHE (published v2.2.3)${NC}"
go mod vendor 2>&1 | grep -v "^$" || true

GOMODCACHE=$(go env GOMODCACHE)
GLIDE_CACHE_DIR="$GOMODCACHE/github.com/valkey-io/valkey-glide/go/v2@v2.2.3"

# Check FFI libraries
FFI_IN_CACHE=$(find "$GLIDE_CACHE_DIR/rustbin" -name "libglide_ffi.a" 2>/dev/null | wc -l | tr -d ' ')
FFI_IN_VENDOR=$(find vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin -name "libglide_ffi.a" 2>/dev/null | wc -l | tr -d ' ')

# Check for marker files in GOMODCACHE
DOC_IN_CACHE=$(find "$GLIDE_CACHE_DIR/rustbin" -name "doc.go" 2>/dev/null | wc -l | tr -d ' ')
TOOLS_IN_CACHE=$([ -f "$GLIDE_CACHE_DIR/tools.go" ] && echo "1" || echo "0")

echo -e "  GOMODCACHE: ${GREEN}$FFI_IN_CACHE${NC} .a files, ${RED}$DOC_IN_CACHE${NC} doc.go, ${RED}$TOOLS_IN_CACHE${NC} tools.go"
echo -e "  vendor/:    ${RED}$FFI_IN_VENDOR${NC} .a files  [${RED}MISSING${NC}: rustbin/ not vendored]"

# Assertion: marker files should NOT exist in published v2.2.3
if [ "$DOC_IN_CACHE" -ne 0 ] || [ "$TOOLS_IN_CACHE" -ne 0 ]; then
    echo -e "  ${RED}ASSERTION FAILED:${NC} Published v2.2.3 should not have doc.go or tools.go files"
    echo -e "  ${RED}This likely means you're testing against a modified GOMODCACHE${NC}"
    exit 1
fi
echo -e "  ${GREEN}✓ Verified: Published v2.2.3 lacks marker files (as expected)${NC}"

rm -rf vendor

# Step 4: Create local copy and test BEFORE fix
echo ""
echo -e "${GREEN}[4/6] Using local copy via replace directive${NC}"

# Create a local copy from GOMODCACHE (as-is from published v2.2.3)
LOCAL_GLIDE_DIR="$PWD/local-valkey-glide"
cp -R "$GLIDE_CACHE_DIR" "$LOCAL_GLIDE_DIR"
chmod -R u+w "$LOCAL_GLIDE_DIR"

RUSTBIN_DIR="$LOCAL_GLIDE_DIR/rustbin"
PLATFORMS=()

# Collect platform names
for platform_dir in "$RUSTBIN_DIR"/*/ ; do
    if [ -d "$platform_dir" ]; then
        platform_name=$(basename "$platform_dir")
        PLATFORMS+=("$platform_name")
    fi
done

# Add replace directive
echo "" >> go.mod
echo "replace github.com/valkey-io/valkey-glide/go/v2 => ./local-valkey-glide" >> go.mod
go mod tidy 2>&1 | grep -v "^$" || true

echo -e "  Testing: go mod vendor from local copy (not fixed yet)..."
rm -rf vendor
go mod vendor 2>&1 | grep -v "^$" || true

echo -e "  ${YELLOW}Note: go mod vendor silently skips directories without .go files${NC}"

# Count FFI files
if [ -d "vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin" ]; then
    FFI_LOCAL=$(find vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin -name "libglide_ffi.a" 2>/dev/null | wc -l | tr -d ' ')
else
    FFI_LOCAL=0
fi

# Check if rustbin has doc.go files (are they valid packages?)
DOC_COUNT=$(find vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin -name "doc.go" 2>/dev/null | wc -l | tr -d ' ')
TOOLS_COUNT=$([ -f "vendor/github.com/valkey-io/valkey-glide/go/v2/tools.go" ] && echo "1" || echo "0")

echo -e "  Local copy: ${GREEN}$FFI_IN_CACHE${NC} .a files, ${RED}$DOC_IN_CACHE${NC} doc.go, ${RED}$TOOLS_IN_CACHE${NC} tools.go"
echo -e "  vendor/:    ${RED}$FFI_LOCAL${NC} .a files, ${RED}$DOC_COUNT${NC} doc.go, ${RED}$TOOLS_COUNT${NC} tools.go  [not valid Go packages]"

echo ""
echo -e "  ${YELLOW}Demonstrating the problem: rustbin/ not in vendor means no FFI libs${NC}"
echo -e "  Simulating: user checks out code, runs go mod vendor..."
echo -e "  Testing: go build (rustbin/ not vendored)..."
BUILD_OUTPUT=$(timeout 30 go build ./main.go 2>&1 || true)
if echo "$BUILD_OUTPUT" | grep -q "library 'glide_ffi' not found"; then
    echo -e "  ${RED}Build failed:${NC}"
    echo "$BUILD_OUTPUT" | sed 's/^/    /'
    echo -e "  ${RED}→ Cannot build: FFI library missing from vendor/${NC}"
else
    echo -e "  ${YELLOW}(Build may work if using GOMODCACHE fallback)${NC}"
fi

# Step 5: Apply fix
echo ""
echo -e "${GREEN}[5/6] Applying fix to local copy${NC}"
echo -e "  Creating doc.go + tools.go..."

# Create doc.go for each platform
for platform_dir in "$RUSTBIN_DIR"/*/ ; do
    if [ -d "$platform_dir" ]; then
        platform_name=$(basename "$platform_dir")
        package_name="rustbin_${platform_name//-/_}"
        
        cat > "$platform_dir/doc.go" << EOF
// Copyright Valkey GLIDE Project Contributors - SPDX Identifier: Apache-2.0

// Package ${package_name} exists solely to ensure this directory is included
// when using 'go mod vendor'.
//
// This directory contains libglide_ffi.a, the precompiled Rust FFI library
// for the ${platform_name} target.
//
// DO NOT IMPORT THIS PACKAGE. It contains no usable Go code and is never
// compiled into binaries. This file only exists to make the directory a valid
// Go package for vendoring purposes.
//
// See: https://github.com/valkey-io/valkey-glide/issues/4721
package ${package_name}
EOF
    fi
done

# Create tools.go
cat > "$LOCAL_GLIDE_DIR/tools.go" << 'EOF'
package glide

// This file ensures rustbin platform directories are included when using 'go mod vendor'.
//
// This file is AUTO-GENERATED during releases. Do not edit manually.
// It forces vendoring to include all platform-specific rustbin packages.
//
// See: https://github.com/valkey-io/valkey-glide/issues/4721
import (
EOF

for platform in "${PLATFORMS[@]}"; do
    echo "	_ \"github.com/valkey-io/valkey-glide/go/v2/rustbin/${platform}\"" >> "$LOCAL_GLIDE_DIR/tools.go"
done
echo ")" >> "$LOCAL_GLIDE_DIR/tools.go"

echo -e "  Created doc.go for ${#PLATFORMS[@]} platforms + tools.go"
go mod tidy 2>&1 | grep -v "^$" || true

# Step 6: Try go mod vendor again
echo ""
echo -e "${GREEN}[6/6] Testing: go mod vendor AFTER fix${NC}"
rm -rf vendor
go mod vendor 2>&1 | grep -v "^$" || true

if [ -d "vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin" ]; then
    FFI_COUNT_AFTER=$(find vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin -name "libglide_ffi.a" 2>/dev/null | wc -l | tr -d ' ')
else
    FFI_COUNT_AFTER=0
fi

DOC_COUNT_AFTER=$(find vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin -name "doc.go" 2>/dev/null | wc -l | tr -d ' ')
TOOLS_COUNT_AFTER=$([ -f "vendor/github.com/valkey-io/valkey-glide/go/v2/tools.go" ] && echo "1" || echo "0")

# Count files in local copy
DOC_IN_LOCAL=$(find local-valkey-glide/rustbin -name "doc.go" 2>/dev/null | wc -l | tr -d ' ')
TOOLS_IN_LOCAL=$([ -f "local-valkey-glide/tools.go" ] && echo "1" || echo "0")

echo -e "  Local copy: ${GREEN}$FFI_IN_CACHE${NC} .a files, ${GREEN}$DOC_IN_LOCAL${NC} doc.go, ${GREEN}$TOOLS_IN_LOCAL${NC} tools.go"
echo -e "  vendor/:    ${GREEN}$FFI_COUNT_AFTER${NC} .a files, ${GREEN}$DOC_COUNT_AFTER${NC} doc.go, ${GREEN}$TOOLS_COUNT_AFTER${NC} tools.go  [${GREEN}OK${NC}: rustbin/ vendored]"
echo ""

# Show vendored platforms
if [ -d "vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin" ]; then
    for platform_dir in vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin/*/; do
        if [ -d "$platform_dir" ]; then
            platform_name=$(basename "$platform_dir")
            if [ -f "$platform_dir/libglide_ffi.a" ]; then
                lib_size=$(ls -lh "$platform_dir/libglide_ffi.a" | awk '{print $5}')
                echo -e "  ${GREEN}+${NC} $platform_name ($lib_size)"
            fi
        fi
    done
fi

echo ""
echo -e "  Testing: go build with vendored code (should work)..."
BUILD_OUTPUT=$(timeout 30 go build -mod=vendor ./main.go 2>&1)
if [ $? -eq 0 ]; then
    echo -e "  ${GREEN}✓ Build successful with vendored FFI libraries${NC}"
    rm -f main
else
    echo -e "  ${RED}Build failed:${NC}"
    echo "$BUILD_OUTPUT"
fi

echo ""
echo -e "${GREEN}SUCCESS${NC} - With doc.go + tools.go, go mod vendor includes FFI libraries."
echo ""
echo -e "Note: GOMODCACHE is read-only by design, so to simulate future releases"
echo -e "      we create a local copy and use the replace directive."
echo -e "      Real releases will have these files committed via go-cd.yml"
echo ""
echo -e "Cleanup: ${BLUE}rm -rf vendor_fix_poc${NC}"

[tdschwarz]

@jamesx-improving I decided to try removing any "works on my machine" issues since I'm testing from a mac and test the script above in docker. Turns out the error output was slightly different in linux, so the script didn't show the build error output.

Anyway, a slightly modified version of the script and the accompanying dockerfile are below. This should help reproduce the error/fix regardless of environment

Updated script:

#!/bin/bash
set -e

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'

echo -e "${BLUE}=== Valkey GLIDE Vendoring Fix POC ===${NC}"
echo ""

[ -d "vendor_fix_poc" ] && rm -rf vendor_fix_poc

# Step 1: Create minimal Go project
echo -e "${GREEN}[1/6] Creating test project${NC}"
mkdir -p vendor_fix_poc
cd vendor_fix_poc

cat > main.go << 'EOF'
package main

import (
	"context"
	"fmt"
	"time"
	
	glide "github.com/valkey-io/valkey-glide/go/v2"
	"github.com/valkey-io/valkey-glide/go/v2/config"
)

func main() {
	// We actually have to create a client for the FFI library to be used,
    // triggering linker errors if .a files are missing
	clientConfig := config.NewClientConfiguration().
		WithAddress(&config.NodeAddress{Host: "localhost", Port: 6379}).
		WithRequestTimeout(500 * time.Millisecond)
	
	client, err := glide.NewClient(clientConfig)
	if err != nil {
		fmt.Printf("Expected error creating client (no server): %v\n", err)
		return
	}
	defer client.Close()
	
	ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
	defer cancel()
	
	_, err = client.Ping(ctx)
	if err != nil {
		fmt.Printf("Expected error pinging (no server): %v\n", err)
	}
}
EOF

go mod init vendor-fix-poc > /dev/null 2>&1

# Step 2: Add dependency and tidy
echo -e "${GREEN}[2/6] Adding valkey-glide v2.2.3${NC}"
go mod tidy

# Step 3: Try go mod vendor from GOMODCACHE
echo ""
echo -e "${GREEN}[3/6] Vendoring from GOMODCACHE (published v2.2.3)${NC}"
go mod vendor 2>&1 | grep -v "^$" || true

GOMODCACHE=$(go env GOMODCACHE)
GLIDE_CACHE_DIR="$GOMODCACHE/github.com/valkey-io/valkey-glide/go/v2@v2.2.3"

# Check FFI libraries
FFI_IN_CACHE=$(find "$GLIDE_CACHE_DIR/rustbin" -name "libglide_ffi.a" 2>/dev/null | wc -l | tr -d ' ')
FFI_IN_VENDOR=$(find vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin -name "libglide_ffi.a" 2>/dev/null | wc -l | tr -d ' ')

# Check for marker files in GOMODCACHE
DOC_IN_CACHE=$(find "$GLIDE_CACHE_DIR/rustbin" -name "doc.go" 2>/dev/null | wc -l | tr -d ' ')
TOOLS_IN_CACHE=$([ -f "$GLIDE_CACHE_DIR/tools.go" ] && echo "1" || echo "0")

echo -e "  GOMODCACHE: ${GREEN}$FFI_IN_CACHE${NC} .a files, ${RED}$DOC_IN_CACHE${NC} doc.go, ${RED}$TOOLS_IN_CACHE${NC} tools.go"
echo -e "  vendor/:    ${RED}$FFI_IN_VENDOR${NC} .a files  [${RED}MISSING${NC}: rustbin/ not vendored]"

# Assertion: marker files should NOT exist in published v2.2.3
if [ "$DOC_IN_CACHE" -ne 0 ] || [ "$TOOLS_IN_CACHE" -ne 0 ]; then
    echo -e "  ${RED}ASSERTION FAILED:${NC} Published v2.2.3 should not have doc.go or tools.go files"
    echo -e "  ${RED}This likely means you're testing against a modified GOMODCACHE${NC}"
    exit 1
fi
echo -e "  ${GREEN}✓ Verified: Published v2.2.3 lacks marker files (as expected)${NC}"

rm -rf vendor

# Step 4: Create local copy and test BEFORE fix
echo ""
echo -e "${GREEN}[4/6] Using local copy via replace directive${NC}"

# Create a local copy from GOMODCACHE (as-is from published v2.2.3)
LOCAL_GLIDE_DIR="$PWD/local-valkey-glide"
cp -R "$GLIDE_CACHE_DIR" "$LOCAL_GLIDE_DIR"
chmod -R u+w "$LOCAL_GLIDE_DIR"

RUSTBIN_DIR="$LOCAL_GLIDE_DIR/rustbin"
PLATFORMS=()

# Collect platform names
for platform_dir in "$RUSTBIN_DIR"/*/ ; do
    if [ -d "$platform_dir" ]; then
        platform_name=$(basename "$platform_dir")
        PLATFORMS+=("$platform_name")
    fi
done

# Add replace directive
echo "" >> go.mod
echo "replace github.com/valkey-io/valkey-glide/go/v2 => ./local-valkey-glide" >> go.mod
go mod tidy 2>&1 | grep -v "^$" || true

echo -e "  Testing: go mod vendor from local copy (not fixed yet)..."
rm -rf vendor
go mod vendor 2>&1 | grep -v "^$" || true

echo -e "  ${YELLOW}Note: go mod vendor silently skips directories without .go files${NC}"

# Count FFI files
if [ -d "vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin" ]; then
    FFI_LOCAL=$(find vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin -name "libglide_ffi.a" 2>/dev/null | wc -l | tr -d ' ')
else
    FFI_LOCAL=0
fi

# Check if rustbin has doc.go files (are they valid packages?)
DOC_COUNT=$(find vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin -name "doc.go" 2>/dev/null | wc -l | tr -d ' ')
TOOLS_COUNT=$([ -f "vendor/github.com/valkey-io/valkey-glide/go/v2/tools.go" ] && echo "1" || echo "0")

echo -e "  Local copy: ${GREEN}$FFI_IN_CACHE${NC} .a files, ${RED}$DOC_IN_CACHE${NC} doc.go, ${RED}$TOOLS_IN_CACHE${NC} tools.go"
echo -e "  vendor/:    ${RED}$FFI_LOCAL${NC} .a files, ${RED}$DOC_COUNT${NC} doc.go, ${RED}$TOOLS_COUNT${NC} tools.go  [not valid Go packages]"

echo ""
echo -e "  ${YELLOW}Demonstrating the problem: rustbin/ not in vendor means no FFI libs${NC}"
echo -e "  Simulating: user checks out code, runs go mod vendor..."
echo -e "  Testing: go build (rustbin/ not vendored)..."
BUILD_OUTPUT=$(timeout 120 go build ./main.go 2>&1) && BUILD_SUCCESS=true || BUILD_SUCCESS=false
if [ "$BUILD_SUCCESS" = "false" ]; then
    echo -e "  ${RED}Build failed:${NC}"
    echo "$BUILD_OUTPUT" | tail -5 | sed 's/^/    /'
    echo -e "  ${RED}→ Cannot build: FFI library missing from vendor/${NC}"
else
    echo -e "  ${YELLOW}(Build unexpectedly succeeded)${NC}"
fi

# Step 5: Apply fix
echo ""
echo -e "${GREEN}[5/6] Applying fix to local copy${NC}"
echo -e "  Creating doc.go + tools.go..."

# Create doc.go for each platform
for platform_dir in "$RUSTBIN_DIR"/*/ ; do
    if [ -d "$platform_dir" ]; then
        platform_name=$(basename "$platform_dir")
        package_name="rustbin_${platform_name//-/_}"
        
        cat > "$platform_dir/doc.go" << EOF
// Copyright Valkey GLIDE Project Contributors - SPDX Identifier: Apache-2.0

// Package ${package_name} exists solely to ensure this directory is included
// when using 'go mod vendor'.
//
// This directory contains libglide_ffi.a, the precompiled Rust FFI library
// for the ${platform_name} target.
//
// DO NOT IMPORT THIS PACKAGE. It contains no usable Go code and is never
// compiled into binaries. This file only exists to make the directory a valid
// Go package for vendoring purposes.
//
// See: https://github.com/valkey-io/valkey-glide/issues/4721
package ${package_name}
EOF
    fi
done

# Create tools.go
cat > "$LOCAL_GLIDE_DIR/tools.go" << 'EOF'
package glide

// This file ensures rustbin platform directories are included when using 'go mod vendor'.
//
// This file is AUTO-GENERATED during releases. Do not edit manually.
// It forces vendoring to include all platform-specific rustbin packages.
//
// See: https://github.com/valkey-io/valkey-glide/issues/4721
import (
EOF

for platform in "${PLATFORMS[@]}"; do
    echo "	_ \"github.com/valkey-io/valkey-glide/go/v2/rustbin/${platform}\"" >> "$LOCAL_GLIDE_DIR/tools.go"
done
echo ")" >> "$LOCAL_GLIDE_DIR/tools.go"

echo -e "  Created doc.go for ${#PLATFORMS[@]} platforms + tools.go"
go mod tidy 2>&1 | grep -v "^$" || true

# Step 6: Try go mod vendor again
echo ""
echo -e "${GREEN}[6/6] Testing: go mod vendor AFTER fix${NC}"
rm -rf vendor
go mod vendor 2>&1 | grep -v "^$" || true

if [ -d "vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin" ]; then
    FFI_COUNT_AFTER=$(find vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin -name "libglide_ffi.a" 2>/dev/null | wc -l | tr -d ' ')
else
    FFI_COUNT_AFTER=0
fi

DOC_COUNT_AFTER=$(find vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin -name "doc.go" 2>/dev/null | wc -l | tr -d ' ')
TOOLS_COUNT_AFTER=$([ -f "vendor/github.com/valkey-io/valkey-glide/go/v2/tools.go" ] && echo "1" || echo "0")

# Count files in local copy
DOC_IN_LOCAL=$(find local-valkey-glide/rustbin -name "doc.go" 2>/dev/null | wc -l | tr -d ' ')
TOOLS_IN_LOCAL=$([ -f "local-valkey-glide/tools.go" ] && echo "1" || echo "0")

echo -e "  Local copy: ${GREEN}$FFI_IN_CACHE${NC} .a files, ${GREEN}$DOC_IN_LOCAL${NC} doc.go, ${GREEN}$TOOLS_IN_LOCAL${NC} tools.go"
echo -e "  vendor/:    ${GREEN}$FFI_COUNT_AFTER${NC} .a files, ${GREEN}$DOC_COUNT_AFTER${NC} doc.go, ${GREEN}$TOOLS_COUNT_AFTER${NC} tools.go  [${GREEN}OK${NC}: rustbin/ vendored]"
echo ""

# Show vendored platforms
if [ -d "vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin" ]; then
    for platform_dir in vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin/*/; do
        if [ -d "$platform_dir" ]; then
            platform_name=$(basename "$platform_dir")
            if [ -f "$platform_dir/libglide_ffi.a" ]; then
                lib_size=$(ls -lh "$platform_dir/libglide_ffi.a" | awk '{print $5}')
                echo -e "  ${GREEN}+${NC} $platform_name ($lib_size)"
            fi
        fi
    done
fi

echo ""
echo -e "  Testing: go build with vendored code (should work)..."
BUILD_OUTPUT=$(timeout 30 go build -mod=vendor ./main.go 2>&1)
if [ $? -eq 0 ]; then
    echo -e "  ${GREEN}✓ Build successful with vendored FFI libraries${NC}"
    rm -f main
else
    echo -e "  ${RED}Build failed:${NC}"
    echo "$BUILD_OUTPUT"
fi

echo ""
echo -e "${GREEN}SUCCESS${NC} - With doc.go + tools.go, go mod vendor includes FFI libraries."
echo ""
echo -e "Note: GOMODCACHE is read-only by design, so to simulate future releases"
echo -e "      we create a local copy and use the replace directive."
echo -e "      Real releases will have these files committed via go-cd.yml"
echo ""
echo -e "Cleanup: ${BLUE}rm -rf vendor_fix_poc${NC}"

Dockerfile.vendor-poc:

# Dockerfile for Valkey GLIDE Vendoring Fix POC
# Build:
#   docker build --platform linux/amd64 -f Dockerfile.vendor-poc -t valkey-glide-vendor-poc .
#
# Run:
#   docker run --platform linux/amd64 --rm valkey-glide-vendor-poc

FROM golang:1.24-bookworm

RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc \
    libc6-dev \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /poc

COPY test_vendor_fix.sh .

RUN chmod +x test_vendor_fix.sh

CMD ["bash", "./test_vendor_fix.sh"]

Docker build and run output:

❯ docker build --platform linux/amd64 -f Dockerfile.vendor-poc -t valkey-glide-vendor-poc . --no-cache
[+] Building 4.9s (9/9) FINISHED                                                                                                                                                                                                                                                                                                                             docker:desktop-linux
 => [internal] load build definition from Dockerfile.vendor-poc                                                                                                                                                                                                                                                                                                              0.0s
 => => transferring dockerfile: 511B                                                                                                                                                                                                                                                                                                                                         0.0s
 => [internal] load metadata for docker.io/library/golang:1.24-bookworm                                                                                                                                                                                                                                                                                                      0.4s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                              0.0s
 => CACHED [1/4] FROM docker.io/library/golang:1.24-bookworm@sha256:328a50f29619c1a805851b8e0606ce94637d65c1b239d2fdccdbc3662c4d7313                                                                                                                                                                                                                                         0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                            0.0s
 => => transferring context: 9.33kB                                                                                                                                                                                                                                                                                                                                          0.0s
 => [2/4] RUN apt-get update && apt-get install -y --no-install-recommends     gcc     libc6-dev     && rm -rf /var/lib/apt/lists/*                                                                                                                                                                                                                                          4.3s
 => [3/4] WORKDIR /poc                                                                                                                                                                                                                                                                                                                                                       0.0s 
 => [4/4] COPY test_vendor_fix.sh .                                                                                                                                                                                                                                                                                                                                          0.0s 
 => exporting to image                                                                                                                                                                                                                                                                                                                                                       0.0s 
 => => exporting layers                                                                                                                                                                                                                                                                                                                                                      0.0s 
 => => writing image sha256:fefb27dac960fbacfcb98bf80d4612c28388a7fdb903341b269303a87b5df64b                                                                                                                                                                                                                                                                                 0.0s 
 => => naming to docker.io/library/valkey-glide-vendor-poc                                                                                                                                                                                                                                                                                                                   0.0s 

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/w9x5sqanyleiiwbaltb8upjaa

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview 
❯ docker run --platform linux/amd64 --rm valkey-glide-vendor-poc

=== Valkey GLIDE Vendoring Fix POC ===

[1/6] Creating test project
[2/6] Adding valkey-glide v2.2.3
go: finding module for package github.com/valkey-io/valkey-glide/go/v2/config
go: finding module for package github.com/valkey-io/valkey-glide/go/v2
go: downloading github.com/valkey-io/valkey-glide/go/v2 v2.2.3
go: downloading github.com/valkey-io/valkey-glide/go v1.3.4
go: downloading github.com/valkey-io/valkey-glide v2.2.4+incompatible
go: found github.com/valkey-io/valkey-glide/go/v2 in github.com/valkey-io/valkey-glide/go/v2 v2.2.3
go: found github.com/valkey-io/valkey-glide/go/v2/config in github.com/valkey-io/valkey-glide/go/v2 v2.2.3
go: downloading github.com/stretchr/testify v1.8.4
go: downloading google.golang.org/protobuf v1.33.0
go: downloading github.com/google/uuid v1.6.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/google/go-cmp v0.6.0

[3/6] Vendoring from GOMODCACHE (published v2.2.3)
  GOMODCACHE: 6 .a files, 0 doc.go, 0 tools.go
  vendor/:    0 .a files  [MISSING: rustbin/ not vendored]
  ✓ Verified: Published v2.2.3 lacks marker files (as expected)

[4/6] Using local copy via replace directive
  Testing: go mod vendor from local copy (not fixed yet)...
  Note: go mod vendor silently skips directories without .go files
  Local copy: 6 .a files, 0 doc.go, 0 tools.go
  vendor/:    0 .a files, 0 doc.go, 0 tools.go  [not valid Go packages]

  Demonstrating the problem: rustbin/ not in vendor means no FFI libs
  Simulating: user checks out code, runs go mod vendor...
  Testing: go build (rustbin/ not vendored)...
  Build failed:
    # command-line-arguments
    /usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
    /usr/bin/gcc -m64 -Wl,--build-id=0xca7a6928f08ade5f192f3eb6db5322ccdcc082a3 -o $WORK/b001/exe/a.out -Wl,--export-dynamic-symbol=_cgo_panic -Wl,--export-dynamic-symbol=_cgo_topofstack -Wl,--export-dynamic-symbol=crosscall2 -Wl,--export-dynamic-symbol=failureCallback -Wl,--export-dynamic-symbol=pubSubCallback -Wl,--export-dynamic-symbol=successCallback -Wl,--compress-debug-sections=zlib /tmp/go-link-2574830239/go.o /tmp/go-link-2574830239/000000.o /tmp/go-link-2574830239/000001.o /tmp/go-link-2574830239/000002.o /tmp/go-link-2574830239/000003.o /tmp/go-link-2574830239/000004.o /tmp/go-link-2574830239/000005.o /tmp/go-link-2574830239/000006.o /tmp/go-link-2574830239/000007.o /tmp/go-link-2574830239/000008.o /tmp/go-link-2574830239/000009.o /tmp/go-link-2574830239/000010.o /tmp/go-link-2574830239/000011.o /tmp/go-link-2574830239/000012.o /tmp/go-link-2574830239/000013.o /tmp/go-link-2574830239/000014.o /tmp/go-link-2574830239/000015.o /tmp/go-link-2574830239/000016.o /tmp/go-link-2574830239/000017.o /tmp/go-link-2574830239/000018.o /tmp/go-link-2574830239/000019.o /tmp/go-link-2574830239/000020.o /tmp/go-link-2574830239/000021.o /tmp/go-link-2574830239/000022.o /tmp/go-link-2574830239/000023.o /tmp/go-link-2574830239/000024.o /tmp/go-link-2574830239/000025.o /tmp/go-link-2574830239/000026.o /tmp/go-link-2574830239/000027.o /tmp/go-link-2574830239/000028.o /tmp/go-link-2574830239/000029.o /tmp/go-link-2574830239/000030.o /tmp/go-link-2574830239/000031.o -O2 -g -lglide_ffi -lm -L/poc/vendor_fix_poc/vendor/github.com/valkey-io/valkey-glide/go/v2/rustbin/x86_64-unknown-linux-gnu -O2 -g -O2 -g -lpthread -O2 -g -O2 -g -O2 -g -no-pie
    /usr/bin/ld: cannot find -lglide_ffi: No such file or directory
    collect2: error: ld returned 1 exit status
  → Cannot build: FFI library missing from vendor/

[5/6] Applying fix to local copy
  Creating doc.go + tools.go...
  Created doc.go for 6 platforms + tools.go

[6/6] Testing: go mod vendor AFTER fix
  Local copy: 6 .a files, 6 doc.go, 1 tools.go
  vendor/:    6 .a files, 6 doc.go, 1 tools.go  [OK: rustbin/ vendored]

  + aarch64-apple-darwin (17M)
  + aarch64-unknown-linux-gnu (28M)
  + aarch64-unknown-linux-musl (28M)
  + x86_64-apple-darwin (20M)
  + x86_64-unknown-linux-gnu (29M)
  + x86_64-unknown-linux-musl (29M)

  Testing: go build with vendored code (should work)...
  ✓ Build successful with vendored FFI libraries

SUCCESS - With doc.go + tools.go, go mod vendor includes FFI libraries.

Note: GOMODCACHE is read-only by design, so to simulate future releases
      we create a local copy and use the replace directive.
      Real releases will have these files committed via go-cd.yml

Cleanup: rm -rf vendor_fix_poc

[jamesx-improving]

Hi @tdschwarz , thank you for the detailed script (and the updated version of it)!

What I'm trying to validate is not the reproducibility of the issue, but the effectiveness of the fix. I see that you're not operating on the existing go mod cache, but a copied one which no longer point to the github repo. I'll try to do that as well and will let you know the result.

We might ended up needing to actually release an alpha version of valkey-glide go, just to validate this. Will let you know if that is the case.

[tdschwarz]

What I'm trying to validate is not the reproducibility of the issue, but the effectiveness of the fix.

@jamesx-improving short of running the updated GHA workflow and creating a release I'm not sure how else to demonstrate the effectiveness of the fix than with a script that does the same thing the updated GHA workflow will do in a real release.

I see that you're not operating on the existing go mod cache, but a copied one which no longer point to the github repo.

Correct. The go mod cache is intentionally read-only. The replace directive exists specifically to allow pointing to a path outside of the GOMODCACHE directory.

You can alter the permissions of your $GOMODCACHE/github.com/valkey-io/valkey-glide/go/v2@v2.2.3 to be writeable and point the script to write the files directly to there if you'd prefer to test it without the replace directive.

The documentation for go mod vendor is pretty explicit about only copying packages from the mod cache.

The go mod vendor command constructs a directory named vendor in the main module’s root directory containing copies of all packages needed to build and test packages in the main module.

So for the compiled .a libraries to be copied the directory they're in needs to be:

1. A package
2. Imported by the main module

The first is satisfied by the doc.go marker files. The tooling doesn't care if the package has any business logic, only that it's a package. The second is satisfied by the no-op imports in the tools.go file in the main module's root.

An official tool directive was added in Go 1.24, which eliminates the need for a tools.go file, but since this package still supports 1.22 the imports have to be in tools.go instead of under a tool directive in go.mod.