Ensmallen graph

[isuruf]

Checklist

• Title of this PR is meaningful: e.g. "Adding my_nifty_package", not "updated meta.yaml".
• License file is packaged (see here for an example).
• Source is from official source.
• Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged).
• If static libraries are linked in, the license of the static library is packaged.
• Package does not ship static libraries. If static libraries are needed, follow CFEP-18.
• Build number is 0.
• A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details).
• GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there.
• When in trouble, please check our knowledge base documentation before pinging a team.

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/bioregistry, recipes/cache_decorator, recipes/compress_json, recipes/curies, recipes/deflate_dict, recipes/dict_hash, recipes/downloaders, recipes/ensmallen, recipes/environments_utils, recipes/jaro-winkler, recipes/more-click, recipes/support_developer, recipes/userinput, recipes/validate_version_code) and found some lint.

Here's what I've got...

For recipes/cache_decorator:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

For recipes/compress_json:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

For recipes/deflate_dict:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

For recipes/dict_hash:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

For recipes/downloaders:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

For recipes/ensmallen:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

For recipes/environments_utils:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

For recipes/jaro-winkler:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

For recipes/support_developer:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

For recipes/userinput:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

For recipes/validate_version_code:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/ensmallen) and found some lint.

Here's what I've got...

For recipes/ensmallen:

• The following maintainers have not yet confirmed that they are willing to be listed here: dillonroach. Please ask them to comment on this PR if they are.

[isuruf]

@dillonroach, can you comment here that you are willing to maintain this recipe?

[isuruf]

@dillonroach, @conda-forge/staged-recipes, I named this ensmallen_graph because there's an unrelated https://github.com/conda-forge/ensmallen-feedstock

[isuruf]

@conda-forge/core @conda-forge/staged-recipes, this is a weird scenario. The package here needs a lot of RAM and the only way to do that is to use Quansight/open-gpu-server. Rust compiler is too memory hungry and gets killed by the OS when it uses too much RAM. You can see conda-forge/cf-autotick-bot-test-package-feedstock#496 where it succeeds with 32GB, but fails with 12 GB of RAM. I tried different options to reduce RAM usage, but it didn't get anywhere.

[github-actions]

To help direct your pull request to the best reviewers, please mention a topic-specifc team if your recipe matches any of the following: conda-forge/help-c-cpp, conda-forge/help-cdts, conda-forge/help-go, conda-forge/help-java, conda-forge/help-julia, conda-forge/help-nodejs, conda-forge/help-perl, conda-forge/help-python, conda-forge/help-python-c, conda-forge/help-r, or conda-forge/help-ruby. Thanks!

[beckermr]

Well that's fun. Also it appears the linter went a bit crazy. Is that a bug we should report?

[isuruf]

You mean #24851 (comment)? That's annoying, but I thought it was intentional.

[dillonroach]

@dillonroach, can you comment here that you are willing to maintain this recipe?

Happy to be added as maintainer

[mbargull]

I tried different options to reduce RAM usage, but it didn't get anywhere.

It would be good to know which options those were.
Is it feasible to somewhat narrow down what part of the build caused the high memory usage?
It would be good to report this upstream -- to the Ensmallen devs and also Rust folks if it's the compiler.

I'd suggest to try out a swap file (see conda-forge/conda-smithy#1751 ) as a workaround instead of using the other runner.
(It's not unlikely that the allocated memory is not read/written to for an extended amount of time in its entirety; hence a swap file would likely not incur much added run time.)

[isuruf]

It would be good to know which options those were.

I tried reducing parallelism to one. (Both cargo parallelism and rustc parallelism)

It would be good to report this upstream -- to the Ensmallen devs and also Rust folks if it's the compiler.

I made an upstream issue at AnacletoLAB/ensmallen#228

I'd suggest to try out a swap file (see conda-forge/conda-smithy#1751 ) as a workaround instead of using the other runner.

Thanks. Let me try that one.

[mbargull]

I'd suggest to try out a swap file (see conda-forge/conda-smithy#1751 ) as a workaround instead of using the other runner.

Thanks. Let me try that one.

Looking at https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=859939&view=results , it unfortunately does seem to incur a much longer compile time (assuming cirun-openstack-cpu-xlarge is not 3+ times faster than the Azure Pipelines runner anyway).

Peeking at the main build files in the Esmallen repo, I noticed
https://github.com/AnacletoLAB/ensmallen/blob/b2492e02dfdb9a7a10ca087d53b3207aadca3b9c/bindings/python/Cargo.toml#L63
and
https://github.com/AnacletoLAB/ensmallen/blob/b2492e02dfdb9a7a10ca087d53b3207aadca3b9c/graph/Cargo.toml#L76
=> setting this to lto = "thin" would make sense.

If it's not (only) the lto = "fat" that causes the enormous memory usage, you could also try to add -C symbol-mangling-version=v0 to the RUSTFLAGS; see rust-lang/rust#82406 and related issues.

[isuruf]

Looking at https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=859939&view=results , it unfortunately does seem to incur a much longer compile time (assuming cirun-openstack-cpu-xlarge is not 3+ times faster than the Azure Pipelines runner anyway).

The agents got killed.

setting this to lto = "thin" would make sense.

Trying this now

If it's not (only) the lto = "fat" that causes the enormous memory usage, you could also try to add -C symbol-mangling-version=v0 to the RUSTFLAGS; see rust-lang/rust#82406 and related issues.

If lto="thin" doesn't work, I'll try this.

[mbargull]

Looks like lto = "thin" already did the trick -- lucky :).

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/ensmallen) and found it was in an excellent condition.

[mbargull]

@conda-forge-admin, please restart ci.
(Due to dabb840 this picked up conda-build=3.28.2 with which linking checks take too long. Restart with 3.28.4 now.)

[isuruf]

Ready for a review @conda-forge/staged-recipes

[github-actions]

To help direct your pull request to the best reviewers, please mention a topic-specifc team if your recipe matches any of the following: conda-forge/help-c-cpp, conda-forge/help-cdts, conda-forge/help-go, conda-forge/help-java, conda-forge/help-julia, conda-forge/help-nodejs, conda-forge/help-perl, conda-forge/help-python, conda-forge/help-python-c, conda-forge/help-r, conda-forge/help-ruby,or conda-forge/help-rust. Thanks!

[isuruf]

Oh. @conda-forge/help-rust

[isuruf]

@carterbox, @xhochy, the bot labelled this as R instead of rust. Probably a pattern matching issue.

[isuruf]

Ping on this

[xhochy]

Looks good!

[isuruf]

Thanks