Skip to content

Pin python version in a strict way #1354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

mathbunnyru
Copy link
Member

Fix: #1352

This is another approach to fix the python version (mamba doesn't change the python version in #1351).

For now, I'm not 100% sure about this PR, because @parente wanted the python patch version to be floating (#778 (comment)).

I want to see how it's gonna work for pyspark image, which was updating python version, even it might not be needed.
Also, it's nice to see the same tags in all the images and to control python version via a variable (it was unexpected for me that two images had different python versions at all).

@mathbunnyru
Copy link
Member Author

mathbunnyru commented Jun 1, 2021

Seems to work fine.
This PR:
https://github.com/jupyter/docker-stacks/pull/1354/checks?check_run_id=2722386028#step:5:6153
Built image size: 3.32GB
jupyterlab_server and python versions stay the same

Master:
https://github.com/jupyter/docker-stacks/runs/2719734258#step:5:6155
Built image size: 3.41GB

Also saves 90 MB of information.

conda list python | grep '^python ' | tr -s ' ' | cut -d '.' -f 1,2 | sed 's/$/.*/' >> $CONDA_DIR/conda-meta/pinned && \
conda list python | grep '^python ' | tr -s ' ' | cut -d ' ' -f 1,2 >> $CONDA_DIR/conda-meta/pinned && \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is too technical for me to parse, can you describe what is changed in what was previously written to the pinned file vs what is now written to the pinned file?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, the current version of python is 3.9.2.
Currently, in the master branch we will pin the version like this python 3.9.*.
So, during conda update (and even install), we might update (and even downgrade) python version to 3.9.x.

The change: write exactly python 3.9.2. This way we pin python versions in a strict way and it won't change during conda update / install.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent thank you!

@consideRatio
Copy link
Member

consideRatio commented Jun 7, 2021

Questions in my mind

  • Are we in agreement to want to pin our Python version while installing the docker image?
    • I think we should enforce a single python version to be used across all our build steps, and if we need to rely on adding a line to the pinned file when using mamba, then I support that at least during the build.
  • Should we have an entry in the $CONDA_DIR/conda-meta/pinned file after our build's various updates to conda packages are complete?
    • If so, how tight should that version pin be?

@mathbunnyru
Copy link
Member Author

Questions in my mind

  • Are we in agreement to want to pin our Python version while installing the docker image?

    • I think we should enforce a single python version to be used across all our build steps, and if we need to rely on adding a line to the pinned file when using mamba, then I support that at least during the build.

As this review shows, we can rely on adding a line to the pinned file (python version is not changing).

  • Should we have an entry in the $CONDA_DIR/conda-meta/pinned file after our build's various updates to conda packages are complete?

    • If so, how tight should that version pin be?

I think, it's ok to pin the version in a strict way - we provide our docker images with complete python tag like jupyter/all-spark-notebook:python-3.9.4, so it won't be obvious for users that they might change the python version after some conda installs / updates.

@mathbunnyru
Copy link
Member Author

And, I think we're updating quite a lot our mambaforge installation (at least we try), so we will be always with quite recent version of python).

@mathbunnyru mathbunnyru changed the title Fix python version in a strict way Pin python version in a strict way Jun 13, 2021
@mathbunnyru
Copy link
Member Author

Let's try this one out.
Now, after switching to mamba, the python version is not changing.
But let's try to merge this to see how it goes for the users. It's easy to revert in case something goes wrong.

@mathbunnyru mathbunnyru merged commit 6e3490c into jupyter:master Jun 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python version changes in pyspark-notebook (and all-spark-notebook)
2 participants