Severe push latency (~20s) for trivial changes in very large monorepo.

[CrazyFanFan]

Hello josh-proxy team,

First, thank you for this great tool. We are evaluating josh-proxy for our very large monorepo and have encountered significant performance latency during push operations that we would like to understand better.

Environment & Setup

• josh-proxy version: r24.10.04
• josh-proxy Server: 64 vCPUs, 256GB RAM
• Client: git version 2.51.0
• Network: Low-latency, high-bandwidth internal network
• josh-proxy Subscription Usage: We cloned the repository with the following command to subscribe to specific prefixes:
  git clone http://server/repo.git:[::subdira,::subdirb].git
  Our working copy only includes content under the subdira and subdirb directories (the "subscribed prefixes" referenced in subsequent sections).

Repository Scale

Our repository is exceptionally large, which we understand is a primary use case for josh-proxy:

• Full repo size: 5.3 GB
• Git LFS objects size: 55 GB
• Total commits: ~568,000
• Total branches: ~6,300
• Total tags: ~1,500
• Number of files in subscribed prefixes (subdira + subdirb): ~74,000

Problem Description

We are observing consistently high latency (~20,000 ms) when pushing trivial changes through josh-proxy. A common workflow is modifying a single line in a single file (located in either subdira or subdirb) and pushing the change—this process takes approximately 20 seconds.

To isolate the issue, we performed the same push operation directly to our GitLab backend (bypassing josh-proxy): pushing the identical single-line change to the same branch completed in under 2 seconds. This confirms the ~18-second overhead is introduced by josh-proxy's processing.

Our Preliminary Analysis & Core Questions

We have reviewed the code and logs and suspect the bottleneck is related to the complex history and scale of our repository

Our primary goal in opening this issue is to clarify the following:

1. Expected Behavior
   Is 20s of latency for a one-line change in subdira/subdirb an expected outcome for a repository of our size given our server hardware? Or does this indicate a problem in our setup or a known josh-proxy bottleneck?

2. Configuration Optimization
   Are we missing critical configuration parameters for multi-prefix subscriptions? Are there tuning options tailored to large monorepos with multiple subscribed directories?

3. Profiling & Further Assistance
   What is the best way for us to help profile this latency? We have collected debug logs and are ready to provide additional traces, performance profiles, or any other data needed. We are eager to collaborate on resolving this.

Thank you for your time and expertise. We are excited about josh-proxy and hope to help make it work seamlessly for our large-scale use case.

[vlad-ivanov-name]

Hey @CrazyFanFan ,

thanks for the issue report.

Couple of things I can already mention:

• 256GB RAM is a lot for josh. If it's josh only, I would expect you would get more or less equal perf on 32GB RAM or even 16GB.
• Cloning via filters defined in the URL is suboptimal for workflows where you intend to push changes back to the repo.

Ideally, what you would want is:

1. Create a directory in your repo, for example: workspaces/something
2. Create a file named workspace.josh inside the directory, with contents:

:[
  ::subdira/
  ::subdirb/
]

3. Commit the file (it can be on a branch)
4. Push the branch

Then, with josh, try cloning the workspace:

git clone http://josh-host/repo-name.git@refs/heads/your-branch-name:workspace=workspaces/something.git clone-dir

5. Verify the state of clone-dir
6. Iterate: it's best if you amend the commit that created workspace.josh and force-push
7. Once happy with the resulting layout, merge the workspace.josh branch to master.

Then your users will clone the url:

http://josh-host/repo-name.git:workspace=workspaces/something.git

And push to that URL too. This method provides a more reliable way of finding the point of attaching the history, so while it could be that the delay is still substantial, the starting conditions would be more appropriate. I will test the push latency I get with a relatively large repo to confirm

If you'd like, we can set up a call to discuss your use case and also look into perf issues/profiling

[vlad-ivanov-name]

Here's a test push to a workspace:

While not instant, it's also not 20 seconds (and it's running on my laptop)

Workspace is here:

https://github.com/vlad-ivanov-name/binutils-gdb/blob/23c395b07137d467190a37bd9954e814711f0269/workspaces/libs/workspace.josh

[vlad-ivanov-name]

In this scenario I can also see in tracing that finding the original commit takes around a second:

binutils-gdb is big but still has around a quarter of the commits of your repo, so there could be a difference. Either way it's worth trying with a workspace.

[christian-schilling]

Looking at the size numbers of your repo I'm almost certain that the slowness is caused by the number of branches and tags.
The suggestion @vlad-ivanov-name made to use git clone http://josh-host/repo-name.git@refs/heads/your-branch-name:workspace=workspaces/something.git clone-dir should be much faster, not because of using workspaces but rather because the explicit limitation to a single reference refs/heads/your-branch-name using the "@ syntax".

[CrazyFanFan]

Hi there — thank you for the thorough reply and the concrete suggestions. They’re very helpful and gave me several angles to verify on my side.

Apologies for the late response. I’ve been tied up by an urgent internal deliverable over the past weeks and wanted to come back with clearer data rather than a quick note.

I’ve included a compact test matrix below that enumerates the variables we exercised (repo scale, LFS, reduced tracked files, reduced branches/tags) . The attached log is from a docker container capturing a git push via a josh proxy. From the tests I ran, the timings are consistently high on my setup, so the most likely root cause is a misconfiguration or operational detail on my side.

Given your guidance, I’ll focus on narrowing proxy/auth behavior, upstream fetch cadence, ref mapping, caching, and workspace filter scope. If you recommend specific checks or flags (e.g., env vars, pack configuration, or targeted workspace filter tests), I’m happy to run them and report back. Thanks again for the detailed help — I appreciate the time you’ve put into this and will follow up promptly with results.

Mode	Repo scale (commits / branches / tags)	Top files count	LFS presence	Reduced tracked files	Reduced branches/tags	Elapsed time (approx)
directory	commits≈568k / branches≈6.3k / tags≈1.5k	≈97k	Yes	No	No	~18s
directory	commits≈568k / branches≈6.3k / tags≈1.5k	≈97k	No	No	No	~18s
directory	commits≈568k / branches≈6.3k / tags≈1.5k	≈66	Yes	Yes	No	~15s
directory	commits≈568k / branches≈10 / tags≈0	≈97k	Yes	No	Yes	~13s
workspace & @ref	commits≈568k / branches≈6.3k / tags≈1.5k	≈97k	Yes	No	No	~90s
workspace & @ref	commits≈123k / branches≈2 / tags≈0	≈477	No	No	Yes	~25s

git_push_docker_log_.txt.zip

[vlad-ivanov-name]

Hey @CrazyFanFan ,

thanks for the detailed measurements and results.

The last few lines don't really make much sense performance wise -- can't really tell where this much of a delay would come from.

There are a couple of things that would be good to try:

1. Build a more recent josh image from master:

docker buildx build --target run . --tag josh:head --build-context=git=.git --build-context=docker=docker --platform=linux/amd64

2. Run Josh next to Jaeger:

In terminal 1:

docker run \
  -p 8000:8000 \
  -e JOSH_REMOTE=http://yourremote.com \
  -e JOSH_OTLP_ENDPOINT=otlp://localhost:4317 \
  -e JOSH_EXTRA_OPTS="--require-auth" \
  --network=host \
  -v josh:/data josh:head

(adjust options as needed -- the important ones are --network=host and JOSH_OTLP_ENDPOINT)

In terminal 2:

docker run --rm --name jaeger \
   -p 16686:16686 \
   -p 4317:4317 \
   -p 4318:4318 \
   -p 5778:5778 \
   -p 9411:9411 \
   cr.jaegertracing.io/jaegertracing/jaeger:2.9.0

Then run one scenario, e.g. workspace & ref, a couple of times.

The traces should appear in Jaeger and will show what's happening on the inside and whether it's filtering or push taking more time.

[CrazyFanFan]

Hi @vlad-ivanov-name

Thanks for your previous reply and for providing the test repo.
I followed your suggestion — I forked your test repository and also mirrored it into our internal environment for comparison.

Here are my findings:

When using my local josh-proxy connected to GitHub and to our internal mirror, the push performance is quite similar — around 9–10 seconds (1. github-jaeger-log.json 2. codebase-jaeger-log.json), slightly slower than your numbers.

However, when testing with our real large repository, the push takes about 50 seconds.

In this large-repo case, Jaeger fails to download the trace log, and the browser console crashes with the following error:

RangeError: Invalid string length (It seems the trace data might be too large.)

At this point I’m not sure what the next step should be — do you have any advice on how to proceed with diagnosing this?

Thanks again for your help!

[vlad-ivanov-name]

Thank you for the updated trace!

There are two observations that could be made from this screenshot, even without opening the actual trace:

1. Number of child spans -- 7k -- suggests that it's related to the number of branches
2. I see two traces with 7k spans, and only one of them takes 50s. Difficult to say what this means though - but could be it's about upstream latency

Is "download results" button in Jaeger UI also broken for the 50s trace?

other strategies we could try:

• set RUST_LOG=josh=info,josh_proxy=trace when launching josh container
• use a different telemetry backend capable of ingesting spans (anything otel-compatible)

The first suggestion should reduce the amount of spans but leave in e.g. command_env which is executed when you push

PS if you prefer, we can also debug this in a call. NDA is also possible if needed. I will be in UTC+7/ICT timezone for the most of November -- depending on where you are this might be closer to you than now.

[vlad-ivanov-name]

If this doesn't bring clarity, I'll try pushing to an upstream (gitlab?) with a large number of branches to try and reproduce

[CrazyFanFan]

Your input gave me a good direction. I’ve been testing with repos similar to ours in key attributes—like swiftlang/llvm-project (600k+ commits).

Here’s what I did:

1. First, I forked the above repo to my own GitHub account (CrazyFanFan/llvm-project).
2. Cloned it via git clone http://localhost:8000/CrazyFanFan/llvm-project.git:[::libc,::libcxx,::libcxxabi,::libsycl].git local_repo.
3. Made a trivial line change, committed the update, then executed the push operation.
   After multiple runs, the fastest push took ~25s.

Would you mind forking this repo (or my forked version: CrazyFanFan/llvm-project) to run your own performance tests? I’m still hunting for other comparable repos to cross-verify.