Failed to compile a project with LTO + PGO #115344

zamazan4ik · 2023-08-29T14:28:45Z

In vectordotdev/vector#15631 (comment) I tried to optimize Vector with LTO + PGO enabled. But during the optimization phase (done with cargo pgo optimize) I get the following error: Kobzol/cargo-pgo#32 (comment) . I am not the only person with this error - llvm/llvm-project#57501 (comment) (@kevincox probably can provide more details about they project).

The only way to avoid the bug is disable LTO completely (switching from Fat to Thin mode does not help).

I expected to see this happen: Vector compiles successfully with LTO and PGO enabled.

Instead, this happened: Vector failed to compile with LTO + PGO.

Meta

rustc --version --verbose (from https://github.com/vectordotdev/vector/blob/master/rust-toolchain.toml):

1.71.1

Other links:

Possibly related bug in LLVM (but I am not sure): [15 regression] "module flag identifiers must be unique (or of 'require' type) llvm/llvm-project#57501
A discussion in cargo-pgo: LLVM ERROR: Broken module found, compilation aborted Kobzol/cargo-pgo#32

The text was updated successfully, but these errors were encountered:

Kobzol · 2023-08-29T14:33:37Z

Were you able to reproduce this without using cargo pgo, by using the compiler flags manually? (-Cprofile-generate etc.)

zamazan4ik · 2023-08-29T14:34:34Z

I haven't tried to do it yet but I will (and then report the results here)

kevincox · 2023-08-29T14:36:37Z

My project was done using the flags directly. See https://gitlab.com/kevincox/ricochetrobots/-/blob/6bbc44ddbf2ab8ae9edcbdca62e3035572f9c1ed/default.nix#L41-57

# Cargo.toml
[profile.release]
codegen-units = 1
lto = true

RUSTFLAGS=-Cprofile-use={profile-data}

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ricochetrobots` (bin "ricochetrobots-debug-precompute")
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ricochetrobots` (bin "solve")
[naersk] cargo returned with exit code 101, exiting

caizixian · 2023-09-01T14:15:51Z

I'm using the compiler flag directly (via RUSTFLAGS). RUSTFLAGS="-Cprofile-use=/tmp/pgo-data/merged.profdata -Cllvm-args=-pgo-warn-missing-function" fails with

error: file `/tmp/pgo-data/merged.profdata` passed to `-C profile-use` does not exist.`

The file is confirmed to exist via ls. The merged.profdata is generated using llvm-profdata shipped with Rust 1.71.1.

RUSTFLAGS="-Clto=false -Cprofile-use=/tmp/pgo-data/merged.profdata -Cllvm-args=-pgo-warn-missing-function" works.

My PGO+LTO setup worked with Rust 1.66.1, but doesn't work with 1.71.1.

lqd · 2023-09-01T15:24:40Z

This "file does not exist" error seems like a different one from the OP ("LLVM ERROR: Broken module found, compilation aborted!"), and it would be good to have a small self-contained example for your new error -- and possibly a dedicated issue since it also looks different from #63609. It feels possible to investigate in mmtk-openjdk directly, but to land a fix with a non-regression test, a small reproducer (MCVE) is going to be important.

zamazan4ik · 2023-09-01T23:14:05Z

I got the same issue during the ruff optimization with PGO with cargo-pgo. When I run cargo pgo optimize bench in https://github.com/astral-sh/ruff/tree/main/crates/ruff_benchmark on my Mac machine (Macbook M1 Pro, macOS 13.4 Ventura), I get the following error:

<skipped lines>
Compiling ruff_benchmark v0.0.0 (/Users/zamazan4ik/open_source/ruff/crates/ruff_benchmark)
warning: 19 warnings emitted========>  ] 287/299: ruff_benchmark, libcst, similar, ruff_notebook, ruff_python_parser, pyproject-toml, ruff_python_formatter, criterion

warning: 468 warnings emitted========> ] 290/299: libcst, ruff_notebook, ruff_python_parser, ruff_python_formatter, criterion

warning: 996 warnings emitted========> ] 292/299: libcst, ruff_python_parser, ruff_python_formatter

   Compiling ruff v0.0.287 (/Users/zamazan4ik/open_source/ruff/crates/ruff)
warning: 75 warnings emitted=========> ] 294/299: ruff, ruff_python_parser

warning: 5188 warnings emitted=======> ] 295/299: ruff

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ruff_benchmark` (bench "formatter"); 45 warnings emitted
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ruff_benchmark` (bench "parser"); 42 warnings emitted
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ruff_benchmark` (bench "linter"); 47 warnings emitted
PGO optimized build has failed.
Cargo finished with an error (101)

Note that Ruff also enables LTO as a part of its Release profile. When I switch off LTO, cargo pgo optimize bench builds Ruff successfully.

Additional details - cargo pgo info:

cargo pgo info
[rustc version]: 1.72.0 is recent enough
[llvm-profdata]: found at /Users/zamazan4ik/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/llvm-profdata
[llvm-bolt]: could not be found (Build LLVM with BOLT and add its `bin` directory to PATH.)
[merge-fdata]: could not be found (Build LLVM with BOLT and add its `bin` directory to PATH.)
Some requirements were not satisfied

Tested the same scenario on my Linux machine (Fedora 38, x86-64) for the same scenario with the same Rust version and got the same error.

zamazan4ik · 2023-09-02T12:52:03Z

Another affected project - https://github.com/foundry-rs/foundry

Got the same error during cargo pgo optimize bench -- --workspace on my Linux machine with Rust 1.72:

warning: 381 warnings emitted

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `anvil-server` (lib test)
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `anvil-rpc` (lib test)
warning: 4095 warnings emitted=====> ] 1501/1535: foundry-abi(test), ethers-solc, foundry-abi, foundry-macros(test), anvil, foun...

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `foundry-macros` (lib test)
PGO optimized build has failed.
Cargo finished with an error (101)

Kobzol · 2023-09-04T20:39:10Z

~~The ruff benchmark worked for me with Rust 1.72 on Linux~~, but I haven't investigated further. Edit: it is also broken for me.

caizixian · 2023-09-06T04:33:59Z

This "file does not exist" error seems like a different one from the OP ("LLVM ERROR: Broken module found, compilation aborted!"), and it would be good to have a small self-contained example for your new error -- and possibly a dedicated issue since it also looks different from #63609. It feels possible to investigate in mmtk-openjdk directly, but to land a fix with a non-regression test, a small reproducer (MCVE) is going to be important.

Indeed. I have a workaround for now (cargo clean before build with profile-use). Might be a problem with incremental compilation.

kevincox · 2023-09-06T12:05:17Z

In my case the compilation was happening clean, so incremental doesn't seem like the problem for me.

zamazan4ik · 2023-09-14T02:20:29Z

Another example of hitting this error with cargo-pgo is optimizing https://github.com/bootandy/dust on my both machines (Linux machine with quite recent x86-64 Fedora 38 and Macbook M1 with macOS 13.4 Ventura) with the latest stable Rust 1.72 on both machines:

cargo pgo optimize build

<some logs are skipped>
...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `du-dust` (bin "dust")
PGO optimized build has failed.
Cargo finished with an error (101)

Also, I've checked cleaning the build directory before the PGO optimization build - didn't help. Dust also enables LTO in its Release profile.

Kobzol · 2023-09-14T08:22:34Z

Uhh, so I was able to reproduce this on Linux (with just lto = "thin", no need for fat LTO or 1 CGU) all the way back to 1.60.0 (older rustc versions can't build dust). Did LTO + PGO ever work? 😆

zamazan4ik · 2023-09-14T08:47:28Z

Uhh, so I was able to reproduce this on Linux (with just lto = "thin", no need for fat LTO or 1 CGU) all the way back to 1.60.0 (older rustc versions can't build dust). Did LTO + PGO ever work?

I need to recheck but AFAIK I remember some Rust projects there LTO + PGO worked fine... :)

Kobzol · 2023-09-14T08:48:12Z

Yeah it's something specific in the project causing it, I'm trying to minimize dust to find it.

zamazan4ik · 2023-10-17T02:58:57Z

Yeah it's something specific in the project causing it, I'm trying to minimize dust to find it.

@Kobzol Were you able to reproduce the issue? I have reproduced it once again with the latest Fedora 38 and Rust stable 1.73 and Ruff project on the latest commit 4113d65836e1b3baa3aa77cbdf44f83207e9f424 from its main branch.

zamazan4ik · 2023-10-20T11:33:36Z

Another affected project: https://github.com/googleforgames/quilkin . When lto = "fat" and codegen-units = 1 when optimizing with cargo-pgo I get the following error:

Compiling quilkin v0.8.0-dev (/home/zamazan4ik/open_source/quilkin)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `quilkin` (bin "quilkin")
PGO optimized build has failed.
Cargo finished with an error (101)

cargo-pgo info:

[rustc version]: 1.73.0 is recent enough
[2023-10-20T11:28:40Z WARN  cargo_pgo::pgo::env] llvm-profdata was resolved from PATH. Make sure that its version is compatible with rustc! If not, run `rustup component add llvm-tools-preview`.
[llvm-profdata]: found at /usr/bin/llvm-profdata
[llvm-bolt]: found at /usr/bin/llvm-bolt
[merge-fdata]: found at /usr/bin/merge-fdata

OS: Fedora 38

zamazan4ik · 2023-10-24T12:12:18Z

Another affected project - https://github.com/ImageOptim/gifski . On Linux with Rust 1.73 I get the same error:

cargo pgo optimize build

<logs are skipped>

Compiling gifski v1.13.0 (/home/zamazan4ik/open_source/gifski)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `gifski` (bin "gifski")
PGO optimized build has failed.
Cargo finished with an error (101)

Additional information:

cargo pgo info

[rustc version]: 1.73.0 is recent enough
[llvm-profdata]: found at /home/zamazan4ik/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata
[llvm-bolt]: found at /usr/bin/llvm-bolt
[merge-fdata]: found at /usr/bin/merge-fdata

Kobzol · 2023-10-26T11:15:39Z

Thanks for gifski! I managed to minimize this down to this in main.rs:

use clap::Command;

fn main() {
    Command::new("a").about("a");
}

in combination with this in Cargo.toml:

[lib]
crate-type = ["lib", "cdylib"]

and LTO=thin. I'll probably post this as a new issue and try to investigate more.

zamazan4ik · 2024-09-06T16:48:46Z

Project - https://github.com/infinyon/fluvio-jolt

Using cargo pgo optimize bench -- --workspace --all-features --profile release-lto (before that PGO profiles were collected via cargo pgo optimize bench -- --workspace --all-features --profile release-lto.

Error:

   Compiling jolt v0.0.0 (/home/zamazan4ik/open_source/fluvio-jolt/smartmodule)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `jolt` (lib test)
PGO optimized build has failed.
Cargo finished with an error (101)

OS: Fedora 40
Rustc: 1.80.1
fluvio-jolt version: main branch, 5b85538f553d338997e20bdbb08cd6fd141e9171 commit

zamazan4ik · 2024-09-12T17:28:52Z

Project - https://github.com/anacrolix/possum

When I enable LTO (add lto = true to the root Cargo.toml file) and using cargo pgo optimize bench I get the following error:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `possum-db` (bin "possum"); 901 warnings emitted
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `possum-db` (bench "possum")
PGO optimized build has failed.
Cargo finished with an error (101)

OS: Fedora 40
Rustc: 1.80.1
Possum: main branch, 35d7f9c03581c649b4c4adced8542520a5d1b54 commit

FilipAndersson245 · 2024-09-16T18:11:43Z

Do anyone have any idea what and how we may fix this issue, as it seem to affect a lot of projects? @Kobzol?

Kobzol · 2024-09-16T18:13:20Z

@dianqk did some initial investigation (#117220). I wanted to follow-up, but haven't found the time to do it yet.

alonme · 2024-09-21T20:47:37Z

Also happens when trying to compile polars with PGO
Branch: main
Commit: 2b4986aab4256ac626600e67a0c1f647186def64
rustc: rustc 1.82.0-nightly (c6db1ca3c 2024-08-25)

Edit:
This doesn't seem to happen with lto=thin

zamazan4ik · 2024-09-30T08:18:27Z

Project - https://github.com/mwlon/pcodec

After enabling LTO with lto = true get the error:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `pco_python` (lib); 1572 warnings emitted
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `pco_cli` (bin "pcodec")
PGO optimized build has failed.
Cargo finished with an error (101)

Fedora 40
Rustc 1.81
pcodec version: main branch, 407fc12b9483bddd153432e59cad4ad968949692 commit

zamazan4ik · 2024-11-14T03:55:40Z

Project - https://git.picodata.io/picodata/picodata/picodata

Error:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `picodata` (bin "picodata")
PGO optimized build has failed.
Cargo finished with an error (101)

Fedora 41
Rustc 1.82
picodata version: master branch, a31dd403db0f46c5bdd154eade444c4256c7209f commit

dianqk · 2024-11-14T04:16:07Z

@DianQK did some initial investigation (#117220). I wanted to follow-up, but haven't found the time to do it yet.

I’m considering fixing it, but if you plan to do it, feel free to re-claim. :)

@rustbot claim

insilications · 2024-11-14T07:36:26Z

@dianqk It would be incredible if you managed to fix this

[WIP] The embedded bitcode should always be prepared for LTO/ThinLTO Fixes rust-lang#115344. Fixes rust-lang#117220. r? ghost

dianqk · 2024-11-24T14:09:57Z

A candidate PR: #133250.

berkus · 2024-11-30T11:08:23Z

@dianqk you are an amazing human being! 💟

The embedded bitcode should always be prepared for LTO/ThinLTO Fixes rust-lang#115344. Fixes rust-lang#117220. There are currently two methods for generating bitcode that used for LTO. One method involves using `-C linker-plugin-lto` to emit object files as bitcode, which is the typical setting used by cargo. The other method is through `-C embed-bitcode=yes`. When using with `-C embed-bitcode=yes -C lto=no`, we run a complete non-LTO LLVM pipeline to obtain bitcode, then the bitcode is used for LTO. We run the Call Graph Profile Pass twice on the same module. This PR is doing something similar to LLVM's `buildFatLTODefaultPipeline`, obtaining the bitcode for embedding after running `buildThinLTOPreLinkDefaultPipeline`. r? nikic

Kobzol · 2025-03-01T12:10:21Z

@zamazan4ik I imagine you might want to try out if it fixes the myriad of issues that you have reported 😁 Huge thanks to DianQK for working on this!

zamazan4ik · 2025-03-18T06:00:48Z

Sorry for the delay from my side - I was busy a bit.

I just tested the fix for Vector with the latest Rustc Nightly - vectordotdev/vector#15631 (comment) - the fix works like a charm! Thanks a lot for everyone who were participating in resolving this PGO blocker. I hope with this fix, we will see more PGO adoption in the Rust ecosystem. Maybe later I will perform more PGO benchmarks for the reported above projects, but it's not an urgent task as far as I see since the fix works great.

UPD: I tested some other projects - for all of them the fix works fine.

The embedded bitcode should always be prepared for LTO/ThinLTO Fixes rust-lang#115344. Fixes rust-lang#117220. There are currently two methods for generating bitcode that used for LTO. One method involves using `-C linker-plugin-lto` to emit object files as bitcode, which is the typical setting used by cargo. The other method is through `-C embed-bitcode=yes`. When using with `-C embed-bitcode=yes -C lto=no`, we run a complete non-LTO LLVM pipeline to obtain bitcode, then the bitcode is used for LTO. We run the Call Graph Profile Pass twice on the same module. This PR is doing something similar to LLVM's `buildFatLTODefaultPipeline`, obtaining the bitcode for embedding after running `buildThinLTOPreLinkDefaultPipeline`. r? nikic

zamazan4ik added the C-bug Category: This is a bug. label Aug 29, 2023

rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Aug 29, 2023

zamazan4ik mentioned this issue Aug 29, 2023

PGO applicability to Vector vectordotdev/vector#15631

Open

This was referenced Sep 1, 2023

pgo: -Cprofile-use can't find file #63609

Open

Update pgo-build script to use the pinned Rust toolchain mmtk/mmtk-openjdk#236

Merged

zamazan4ik mentioned this issue Sep 1, 2023

Evaluate Profile-Guided Optimization (PGO) and LLVM BOLT astral-sh/ruff#7055

Open

zamazan4ik mentioned this issue Oct 20, 2023

Enable Profile-Guided Optimization (PGO) or LLVM BOLT googleforgames/quilkin#834

Open

Kobzol mentioned this issue Oct 26, 2023

Combining LTO + PGO + lib/cdylib crashes on LLVM assertion #117220

Closed

jieyouxu added the A-PGO Area: Profile-guided optimizations (PGO) label Aug 13, 2024

zamazan4ik mentioned this issue Sep 12, 2024

Evaluate using more optimizations like LTO and PGO anacrolix/possum#5

Closed

alonme mentioned this issue Sep 21, 2024

Build Python polars wheels with PGO pola-rs/polars#9702

Open

Kobzol mentioned this issue Oct 29, 2024

Add more optimizations to the release build profile. rust-lang/cargo#11298

Open

rustbot assigned dianqk Nov 14, 2024

dianqk mentioned this issue Nov 20, 2024

The embedded bitcode should always be prepared for LTO/ThinLTO #133250

Merged

bors added a commit to rust-lang-ci/rust that referenced this issue Nov 20, 2024

Auto merge of rust-lang#133250 - DianQK:embed-bitcode-pgo, r=<try>

5f7a0d8

[WIP] The embedded bitcode should always be prepared for LTO/ThinLTO Fixes rust-lang#115344. Fixes rust-lang#117220. r? ghost

bors added a commit to rust-lang-ci/rust that referenced this issue Nov 21, 2024

Auto merge of rust-lang#133250 - DianQK:embed-bitcode-pgo, r=<try>

5a6a7f3

[WIP] The embedded bitcode should always be prepared for LTO/ThinLTO Fixes rust-lang#115344. Fixes rust-lang#117220. r? ghost

FilipAndersson245 mentioned this issue Dec 12, 2024

Apply Profile-guided optimization to improve performance rust-lang/rust-analyzer#9412

Closed

GoldenStack mentioned this issue Jan 13, 2025

Enable Link-Time Optimization (LTO) and codegen-units = 1 GoldenStack/stupidfs#1

Closed

bors closed this as completed in 0c72c0d Mar 1, 2025

mtreinish mentioned this issue Mar 10, 2025

Fix PGO on MacOS for 2.0.0 final Qiskit/qiskit#13983

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to compile a project with LTO + PGO #115344

Failed to compile a project with LTO + PGO #115344

zamazan4ik commented Aug 29, 2023 •

edited by rustbot

Loading

Kobzol commented Aug 29, 2023

zamazan4ik commented Aug 29, 2023 •

edited

Loading

kevincox commented Aug 29, 2023 •

edited

Loading

caizixian commented Sep 1, 2023 •

edited

Loading

lqd commented Sep 1, 2023

zamazan4ik commented Sep 1, 2023 •

edited

Loading

zamazan4ik commented Sep 2, 2023

Kobzol commented Sep 4, 2023 •

edited

Loading

caizixian commented Sep 6, 2023

kevincox commented Sep 6, 2023

zamazan4ik commented Sep 14, 2023 •

edited

Loading

Kobzol commented Sep 14, 2023

zamazan4ik commented Sep 14, 2023

Kobzol commented Sep 14, 2023

zamazan4ik commented Oct 17, 2023 •

edited

Loading

zamazan4ik commented Oct 20, 2023

zamazan4ik commented Oct 24, 2023

Kobzol commented Oct 26, 2023 •

edited

Loading

zamazan4ik commented Sep 6, 2024

zamazan4ik commented Sep 12, 2024 •

edited

Loading

FilipAndersson245 commented Sep 16, 2024

Kobzol commented Sep 16, 2024

alonme commented Sep 21, 2024 •

edited

Loading

zamazan4ik commented Sep 30, 2024

zamazan4ik commented Nov 14, 2024

dianqk commented Nov 14, 2024

insilications commented Nov 14, 2024

dianqk commented Nov 24, 2024

berkus commented Nov 30, 2024

Kobzol commented Mar 1, 2025

zamazan4ik commented Mar 18, 2025 •

edited

Loading

Failed to compile a project with LTO + PGO #115344

Failed to compile a project with LTO + PGO #115344

Comments

zamazan4ik commented Aug 29, 2023 • edited by rustbot Loading

Meta

Kobzol commented Aug 29, 2023

zamazan4ik commented Aug 29, 2023 • edited Loading

kevincox commented Aug 29, 2023 • edited Loading

caizixian commented Sep 1, 2023 • edited Loading

lqd commented Sep 1, 2023

zamazan4ik commented Sep 1, 2023 • edited Loading

zamazan4ik commented Sep 2, 2023

Kobzol commented Sep 4, 2023 • edited Loading

caizixian commented Sep 6, 2023

kevincox commented Sep 6, 2023

zamazan4ik commented Sep 14, 2023 • edited Loading

Kobzol commented Sep 14, 2023

zamazan4ik commented Sep 14, 2023

Kobzol commented Sep 14, 2023

zamazan4ik commented Oct 17, 2023 • edited Loading

zamazan4ik commented Oct 20, 2023

zamazan4ik commented Oct 24, 2023

Kobzol commented Oct 26, 2023 • edited Loading

zamazan4ik commented Sep 6, 2024

zamazan4ik commented Sep 12, 2024 • edited Loading

FilipAndersson245 commented Sep 16, 2024

Kobzol commented Sep 16, 2024

alonme commented Sep 21, 2024 • edited Loading

zamazan4ik commented Sep 30, 2024

zamazan4ik commented Nov 14, 2024

dianqk commented Nov 14, 2024

insilications commented Nov 14, 2024

dianqk commented Nov 24, 2024

berkus commented Nov 30, 2024

Kobzol commented Mar 1, 2025

zamazan4ik commented Mar 18, 2025 • edited Loading

zamazan4ik commented Aug 29, 2023 •

edited by rustbot

Loading

zamazan4ik commented Aug 29, 2023 •

edited

Loading

kevincox commented Aug 29, 2023 •

edited

Loading

caizixian commented Sep 1, 2023 •

edited

Loading

zamazan4ik commented Sep 1, 2023 •

edited

Loading

Kobzol commented Sep 4, 2023 •

edited

Loading

zamazan4ik commented Sep 14, 2023 •

edited

Loading

zamazan4ik commented Oct 17, 2023 •

edited

Loading

Kobzol commented Oct 26, 2023 •

edited

Loading

zamazan4ik commented Sep 12, 2024 •

edited

Loading

alonme commented Sep 21, 2024 •

edited

Loading

zamazan4ik commented Mar 18, 2025 •

edited

Loading