Skip to content

Set 4MB stack size for all threads, introduce UV_STACK_SIZE #12839

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 16, 2025
Merged

Set 4MB stack size for all threads, introduce UV_STACK_SIZE #12839

merged 3 commits into from
Apr 16, 2025

Conversation

konstin
Copy link
Member

@konstin konstin commented Apr 11, 2025 •
edited
Loading

See #12769 for the motivation. We set the 4MB not only for the main thread, but also for all tokio and rayon threads to fix a stack overflow while unpacking wheels in production on Windows.

There are two variables for setting the stack size: A new UV_STACK_SIZE that takes precedent, and the existing RUST_MIN_STACK. When setting the stack size, UV_STACK_SIZE should be preferred, since RUST_MIN_STACK affects all Rust applications, including build backends we call (e.g., maturin). The minimum stack size is set to 1MB, the lowest stack size we observed on a platform (Windows main thread).

Fixes #12769

Test Plan

Tested manually with the example from #12769

@konstin konstin added bug Something isn't working windows Specific to the Windows platform labels Apr 11, 2025
@konstin konstin requested a review from Gankra April 11, 2025 16:05
Gankra
Gankra approved these changes Apr 11, 2025
View reviewed changes
crates/uv-configuration/src/threading.rs Outdated
std::env::var(EnvVars::RUST_MIN_STACK)
.ok()
.and_then(|var| var.parse::<usize>().ok())
.unwrap_or(UV_MIN_STACK_DEFAULT)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the unwrap_or(0).max(UV_MIN_STACK_DEFAULT) was imo important for ensuring no errant/unrelated uses of RUST_MIN_STACK broke uv.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't unwrap_or(0).max(UV_MIN_STACK_DEFAULT) mean that we never allow to set a smaller stack size, even with RUST_MIN_STACK? My idea was that we allow RUST_MIN_STACK in both directions, so that we can both test the effect of lower values (is the bound too high?) and allow bumping it higher (for testing in cases #12769 like where we need to bump it higher).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah but at this point we "know" the current value is insufficient, so it's not safe to let people set it lower. And someone might set it in their system for another thing and call an accident? Maybe I'm being too paranoid.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put a 1MB limit in place and switched to UV_STACK_SIZE

@konstin konstin mentioned this pull request Apr 14, 2025
Closed
@konstin konstin changed the title Set 4MB stack size for all threads Set 4MB stack size for all threads, introduce UV_STACK_SIZE Apr 14, 2025
konstin added 3 commits April 14, 2025 16:00
@konstin
Set 4MB stack size for all threads
7aaf26a 
See #12769. We set the 4MB not only for the main thread, but also for all tokio and rayon threads to fix a stack overflow while unpacking wheels in production on Windows.

Fixes #12769
@konstin
Rewrite
2e8f313
@konstin
Generate docs
0a9a2d0
@konstin konstin force-pushed the konsti/set-thread-size-for-rayon-and-tokio-too branch from 195a25f to 0a9a2d0 Compare April 14, 2025 14:00
@konstin konstin merged commit 4d34b28 into main Apr 16, 2025
77 checks passed
@konstin konstin deleted the konsti/set-thread-size-for-rayon-and-tokio-too branch April 16, 2025 07:27
@notatallshaw notatallshaw mentioned this pull request Apr 16, 2025
Closed
@BrewTestBot BrewTestBot mentioned this pull request Apr 22, 2025
Merged
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Apr 26, 2025
@tmeijn
build(deps): update astral-sh/uv to v0.6.16
cb4b910 
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.6.11` -> `0.6.16` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.6.16`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0616)

[Compare Source](astral-sh/uv@0.6.15...0.6.16)

##### Bug fixes

-   Revert "Properly handle authentication for 302 redirect URLs" ([#&#8203;13041](astral-sh/uv#13041))

### [`v0.6.15`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0615)

[Compare Source](astral-sh/uv@0.6.14...0.6.15)

This release includes preliminary support for the `pylock.toml` file format, as standardized in [PEP 751](https://peps.python.org/pep-0751/). `pylock.toml` is an alternative resolution output format intended to replace `requirements.txt` (e.g., in the context of `uv pip compile`, whereby a "locked" `requirements.txt` file is generated from a set of input requirements). `pylock.toml` is standardized and tool-agnostic, such that in the future, `pylock.toml` files generated by uv could be installed by other tools, and vice versa.

As of this release, `pylock.toml` is supported in the following commands:

-   To export a `uv.lock` to the `pylock.toml` format, run: `uv export -o pylock.toml`
-   To generate a `pylock.toml` file from a set of requirements, run: `uv pip compile -o pylock.toml -r requirements.in`
-   To install from a `pylock.toml` file, run: `uv pip sync pylock.toml` or `uv pip install -r pylock.toml`

##### Enhancements

-   Add PEP 751 support to `uv pip compile` ([#&#8203;13019](astral-sh/uv#13019))
-   Add `uv export` support for PEP 751 ([#&#8203;12955](astral-sh/uv#12955))
-   Accept `requirements.txt` (verbatim) as a format on the CLI ([#&#8203;12957](astral-sh/uv#12957))
-   Add `UV_NO_EDITABLE` environment variable to set `--no-editable` on all invocations ([#&#8203;12773](astral-sh/uv#12773))
-   Add `pylock.toml` to `uv pip install` and `uv pip sync` ([#&#8203;12992](astral-sh/uv#12992))
-   Add a brief sleep before sending `SIGINT` to child processes ([#&#8203;13018](astral-sh/uv#13018))
-   Add upload time to `uv.lock` ([#&#8203;12968](astral-sh/uv#12968))
-   Allow updating Git sources by name ([#&#8203;12897](astral-sh/uv#12897))
-   Cache `which git` in `uv init` ([#&#8203;12893](astral-sh/uv#12893))
-   Enable `--dry-run` with `--locked` / `--frozen` for `uv sync` ([#&#8203;12778](astral-sh/uv#12778))
-   Infer output type in `uv export` ([#&#8203;12958](astral-sh/uv#12958))
-   Make `uv init` resilient against broken git ([#&#8203;12895](astral-sh/uv#12895))
-   Respect build constraints for `uv run --with` dependencies ([#&#8203;12882](astral-sh/uv#12882))
-   Split UV_INDEX on all whitespace ([#&#8203;12820](astral-sh/uv#12820))
-   Support build constraints in `uv tool` and PEP723 scripts. ([#&#8203;12842](astral-sh/uv#12842))
-   Use suffix from `uvx` binary when searching for uv binary ([#&#8203;12923](astral-sh/uv#12923))
-   Update version formatting to use cyan color ([#&#8203;12943](astral-sh/uv#12943))
-   Add debug logs for version file search ([#&#8203;12951](astral-sh/uv#12951))
-   Fix `SourceNotAllowed` error message during Python discovery ([#&#8203;13012](astral-sh/uv#13012))
-   Obfuscate password in credentials debug messages ([#&#8203;12944](astral-sh/uv#12944))
-   Obfuscate possible tokens in URL logs ([#&#8203;12969](astral-sh/uv#12969))
-   Validate that PEP 751 entries don't include multiple sources ([#&#8203;12993](astral-sh/uv#12993))

##### Preview features

-   Build backend: Add reference docs and schema ([#&#8203;12803](astral-sh/uv#12803))

##### Bug fixes

-   Align supported `config-settings` with example in docs ([#&#8203;12947](astral-sh/uv#12947))
-   Ensure virtual environment is compatible with interpreter on sync ([#&#8203;12884](astral-sh/uv#12884))
-   Fix `PythonDownloadRequest` parsing for partial keys ([#&#8203;12925](astral-sh/uv#12925))
-   Fix pre-release exclusive comparison operator in `uv-pep440` ([#&#8203;12836](astral-sh/uv#12836))
-   Forward additional signals to the child process in `uv run` ([#&#8203;13017](astral-sh/uv#13017))
-   Omit PEP 751 version for source trees ([#&#8203;13030](astral-sh/uv#13030))
-   Patch `CC` and `CCX` entries in sysconfig for cross-compiled `aarch64` Python distributions ([#&#8203;12239](astral-sh/uv#12239))
-   Properly handle authentication for HTTP 302 redirect URLs ([#&#8203;12920](astral-sh/uv#12920))
-   Set 4MB stack size for all threads, introduce `UV_STACK_SIZE` ([#&#8203;12839](astral-sh/uv#12839))
-   Show PyPy downloads during `uv python list` ([#&#8203;12915](astral-sh/uv#12915))
-   Add `subdirectory` to Direct URL for local directories ([#&#8203;12971](astral-sh/uv#12971))
-   Prefer stable releases over pre-releases in `uv python install` ([#&#8203;12194](astral-sh/uv#12194))
-   Write requested Python variant to pin file in `uv init` ([#&#8203;12870](astral-sh/uv#12870))

##### Documentation

-   Fix CLI reference with code block ([#&#8203;12807](astral-sh/uv#12807))
-   Fix lockfile note ([#&#8203;12793](astral-sh/uv#12793))
-   Fix typo in a reference ([#&#8203;12858](astral-sh/uv#12858))
-   Improve docs for `uv python list --only-downloads` and `--only-installed` ([#&#8203;12916](astral-sh/uv#12916))
-   Update note on lack of musl distributions to ARM-only ([#&#8203;12825](astral-sh/uv#12825))
-   Add section on shebangs for scripts ([#&#8203;11553](astral-sh/uv#11553))
-   Display aliases for long and short args in the CLI reference ([#&#8203;12824](astral-sh/uv#12824))
-   Fix highlight line in explicit index documentation ([#&#8203;12887](astral-sh/uv#12887))
-   Add explicit source (matching PyTorch guide) ([#&#8203;12844](astral-sh/uv#12844))
-   Fix link to issue ([#&#8203;12823](astral-sh/uv#12823))
-   Fix grammatical error in FastAPI guide ([#&#8203;12908](astral-sh/uv#12908))
-   Add `--locked` to `uv sync` in GitHub Actions guide ([#&#8203;12819](astral-sh/uv#12819))
-   Improve formatting for `"all"` `default-groups` setting documentation ([#&#8203;12963](astral-sh/uv#12963))
-   Replace `--frozen` with `--locked` in Docker integration guide ([#&#8203;12818](astral-sh/uv#12818))

### [`v0.6.14`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0614)

[Compare Source](astral-sh/uv@0.6.13...0.6.14)

##### Python versions

The following Python versions have been added:

-   CPython 3.13.3
-   CPython 3.12.10
-   CPython 3.11.12
-   CPython 3.10.17
-   CPython 3.9.22

See the [`python-build-standalone` release notes](https://github.com/astral-sh/python-build-standalone/releases/tag/20250409) for more details.

##### Enhancements

-   Add `uv-build` and `uv_build` aliases to `uv init --build-backend` ([#&#8203;12776](astral-sh/uv#12776))
-   Emit dedicated error message for Conda `environment.yml` files ([#&#8203;12669](astral-sh/uv#12669))

##### Preview features

-   Build backend: Check module dir exists for sdist build ([#&#8203;12779](astral-sh/uv#12779))
-   Build backend: Fix sdist with long directories ([#&#8203;12764](astral-sh/uv#12764))

##### Performance

-   Avoid querying GitHub on repeated install invocations ([#&#8203;12767](astral-sh/uv#12767))

##### Bug fixes

-   Error when `tool.uv.sources` is set in system-level configuration file ([#&#8203;12757](astral-sh/uv#12757))
-   Split workspace members onto their own lines in `uv init` ([#&#8203;12756](astral-sh/uv#12756))

##### Documentation

-   Add lockfile note about PEP 751 ([#&#8203;12732](astral-sh/uv#12732))
-   Extend the reference documentation for `uv pip sync` ([#&#8203;12683](astral-sh/uv#12683))
-   Fix mismatched pip interface header / nav titles ([#&#8203;12640](astral-sh/uv#12640))

### [`v0.6.13`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0613)

[Compare Source](astral-sh/uv@0.6.12...0.6.13)

##### Enhancements

-   Add `--show-version` to `uv python find` ([#&#8203;12376](astral-sh/uv#12376))
-   Remove `--no-config` warning from `uv pip compile` and `uv pip sync` ([#&#8203;12642](astral-sh/uv#12642))
-   Skip repeated directories in `PATH` when searching for Python interpreters ([#&#8203;12367](astral-sh/uv#12367))
-   Unset `SCRIPT_PATH` in relocatable activation script ([#&#8203;12672](astral-sh/uv#12672))
-   Add `UV_PYTHON_DOWNLOADS_JSON_URL` to set custom managed python sources ([#&#8203;10939](astral-sh/uv#10939))
-   Reject `pyproject.toml` files in `uv pip compile -o` ([#&#8203;12673](astral-sh/uv#12673))
-   Respect the `--offline` flag for Git operations ([#&#8203;12619](astral-sh/uv#12619))

##### Bug fixes

-   Warn instead of error if CRC appears to be missing ([#&#8203;12722](astral-sh/uv#12722))
-   Avoid infinite loop in `uv export` with conflicts ([#&#8203;12726](astral-sh/uv#12726))

##### Rust API

-   Update MSRV to 1.84 ([#&#8203;12670](astral-sh/uv#12670))

### [`v0.6.12`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0612)

[Compare Source](astral-sh/uv@0.6.11...0.6.12)

##### Enhancements

-   Report the queried executable path in `uv python list` ([#&#8203;12628](astral-sh/uv#12628))
-   Improve archive unpack error messages ([#&#8203;12627](astral-sh/uv#12627))

##### Bug fixes

-   Respect `authenticate` when using `explicit = true` ([#&#8203;12631](astral-sh/uv#12631))
-   Normalize extra and group names in `uv add` and `uv remove` ([#&#8203;12586](astral-sh/uv#12586))
-   Enforce CRC-32 checks when unpacking archives ([#&#8203;12623](astral-sh/uv#12623))
-   Fix parsing of `python-platform` in settings files ([#&#8203;12592](astral-sh/uv#12592))

##### Documentation

-   Add note about `uv build` to `package = false` ([#&#8203;12608](astral-sh/uv#12608))
-   Add index fallback note to `authenticate = always` documentation ([#&#8203;12498](astral-sh/uv#12498))
-   Fix invalid 'kind' reference in flat index docs ([#&#8203;12583](astral-sh/uv#12583))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4yNTEuMCIsInVwZGF0ZWRJblZlciI6IjM5LjI1My41IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Apr 29, 2025
@0-wiz-0
uv: update to 0.6.17.
21a3169 
## 0.6.17

### Preview features

- Add PyTorch v2.7.0 to GPU backend ([#13072](astral-sh/uv#13072))

### Bug fixes

- Avoid panic for invalid Python versions ([#13077](astral-sh/uv#13077))
- Block scripts from overwriting `python` ([#13051](astral-sh/uv#13051))
- Check distribution names to handle invalid redirects ([#12917](astral-sh/uv#12917))
- Check for mismatched package and distribution names on resolver thread ([#13088](astral-sh/uv#13088))
- Fix panic with invalid last character in PEP 508 name ([#13105](astral-sh/uv#13105))
- Reject `requires-python` even if not listed on the index page ([#13086](astral-sh/uv#13086))

## 0.6.16

### Bug fixes

- Revert "Properly handle authentication for 302 redirect URLs" ([#13041](astral-sh/uv#13041))

## 0.6.15

This release includes preliminary support for the `pylock.toml` file format, as standardized in [PEP 751](https://peps.python.org/pep-0751/). `pylock.toml` is an alternative resolution output format intended to replace `requirements.txt` (e.g., in the context of `uv pip compile`, whereby a "locked" `requirements.txt` file is generated from a set of input requirements). `pylock.toml` is standardized and tool-agnostic, such that in the future, `pylock.toml` files generated by uv could be installed by other tools, and vice versa.

As of this release, `pylock.toml` is supported in the following commands:

- To export a `uv.lock` to the `pylock.toml` format, run: `uv export -o pylock.toml`
- To generate a `pylock.toml` file from a set of requirements, run: `uv pip compile -o pylock.toml -r requirements.in`
- To install from a `pylock.toml` file, run: `uv pip sync pylock.toml` or `uv pip install -r pylock.toml`

### Enhancements

- Add PEP 751 support to `uv pip compile` ([#13019](astral-sh/uv#13019))
- Add `uv export` support for PEP 751 ([#12955](astral-sh/uv#12955))
- Accept `requirements.txt` (verbatim) as a format on the CLI ([#12957](astral-sh/uv#12957))
- Add `UV_NO_EDITABLE` environment variable to set `--no-editable` on all invocations ([#12773](astral-sh/uv#12773))
- Add `pylock.toml` to `uv pip install` and `uv pip sync` ([#12992](astral-sh/uv#12992))
- Add a brief sleep before sending `SIGINT` to child processes ([#13018](astral-sh/uv#13018))
- Add upload time to `uv.lock` ([#12968](astral-sh/uv#12968))
- Allow updating Git sources by name ([#12897](astral-sh/uv#12897))
- Cache `which git` in `uv init` ([#12893](astral-sh/uv#12893))
- Enable `--dry-run` with `--locked` / `--frozen` for `uv sync` ([#12778](astral-sh/uv#12778))
- Infer output type in `uv export` ([#12958](astral-sh/uv#12958))
- Make `uv init` resilient against broken git ([#12895](astral-sh/uv#12895))
- Respect build constraints for `uv run --with` dependencies ([#12882](astral-sh/uv#12882))
- Split UV_INDEX on all whitespace ([#12820](astral-sh/uv#12820))
- Support build constraints in `uv tool` and PEP723 scripts. ([#12842](astral-sh/uv#12842))
- Use suffix from `uvx` binary when searching for uv binary ([#12923](astral-sh/uv#12923))
- Update version formatting to use cyan color ([#12943](astral-sh/uv#12943))
- Add debug logs for version file search ([#12951](astral-sh/uv#12951))
- Fix `SourceNotAllowed` error message during Python discovery ([#13012](astral-sh/uv#13012))
- Obfuscate password in credentials debug messages ([#12944](astral-sh/uv#12944))
- Obfuscate possible tokens in URL logs ([#12969](astral-sh/uv#12969))
- Validate that PEP 751 entries don't include multiple sources ([#12993](astral-sh/uv#12993))

### Preview features

- Build backend: Add reference docs and schema ([#12803](astral-sh/uv#12803))

### Bug fixes

- Align supported `config-settings` with example in docs ([#12947](astral-sh/uv#12947))
- Ensure virtual environment is compatible with interpreter on sync ([#12884](astral-sh/uv#12884))
- Fix `PythonDownloadRequest` parsing for partial keys ([#12925](astral-sh/uv#12925))
- Fix pre-release exclusive comparison operator in `uv-pep440` ([#12836](astral-sh/uv#12836))
- Forward additional signals to the child process in `uv run` ([#13017](astral-sh/uv#13017))
- Omit PEP 751 version for source trees ([#13030](astral-sh/uv#13030))
- Patch `CC` and `CCX` entries in sysconfig for cross-compiled `aarch64` Python distributions ([#12239](astral-sh/uv#12239))
- Properly handle authentication for HTTP 302 redirect URLs ([#12920](astral-sh/uv#12920))
- Set 4MB stack size for all threads, introduce `UV_STACK_SIZE` ([#12839](astral-sh/uv#12839))
- Show PyPy downloads during `uv python list` ([#12915](astral-sh/uv#12915))
- Add `subdirectory` to Direct URL for local directories ([#12971](astral-sh/uv#12971))
- Prefer stable releases over pre-releases in `uv python install` ([#12194](astral-sh/uv#12194))
- Write requested Python variant to pin file in `uv init` ([#12870](astral-sh/uv#12870))

### Documentation

- Fix CLI reference with code block ([#12807](astral-sh/uv#12807))
- Fix lockfile note ([#12793](astral-sh/uv#12793))
- Fix typo in a reference ([#12858](astral-sh/uv#12858))
- Improve docs for `uv python list --only-downloads` and `--only-installed` ([#12916](astral-sh/uv#12916))
- Update note on lack of musl distributions to ARM-only ([#12825](astral-sh/uv#12825))
- Add section on shebangs for scripts ([#11553](astral-sh/uv#11553))
- Display aliases for long and short args in the CLI reference ([#12824](astral-sh/uv#12824))
- Fix highlight line in explicit index documentation ([#12887](astral-sh/uv#12887))
- Add explicit source (matching PyTorch guide) ([#12844](astral-sh/uv#12844))
- Fix link to issue ([#12823](astral-sh/uv#12823))
- Fix grammatical error in FastAPI guide ([#12908](astral-sh/uv#12908))
- Add `--locked` to `uv sync` in GitHub Actions guide ([#12819](astral-sh/uv#12819))
- Improve formatting for `"all"` `default-groups` setting documentation ([#12963](astral-sh/uv#12963))
- Replace `--frozen` with `--locked` in Docker integration guide ([#12818](astral-sh/uv#12818))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Gankra Gankra Gankra approved these changes

Assignees
No one assigned
Labels
bug Something isn't working windows Specific to the Windows platform
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

uv pip insall on Windows causes "thread '<unknown>' has overflowed its stack"
2 participants
@konstin @Gankra