Skip to content

pep517: in_process: try _in_process.pyc file if _in_process.py doesn't exist#130

Closed
commodo wants to merge 1 commit intopypa:mainfrom
commodo:in_process_pyc
Closed

pep517: in_process: try _in_process.pyc file if _in_process.py doesn't exist#130
commodo wants to merge 1 commit intopypa:mainfrom
commodo:in_process_pyc

Conversation

@commodo
Copy link
Copy Markdown

@commodo commodo commented Sep 6, 2021

This change came about to exist after this bug report:
openwrt/packages#11912

The general context is that OpenWrt does not ship Python source-code by
default. Instead, the build byte-compiles all .py files into .pyc files and
removes the .py files.
There's a wide reasoning for this, from performance to reducing the size of
these packages.

This change is kept in the Python3 build as this patch:
https://github.com/openwrt/packages/blob/master/lang/python/python3/patches-pip/001-pep517-pyc-fix.patch

This is an attempt to start a discussion, to see whether it makes sense to
remove it from the OpenWrt packages feed, and live in the pip source-base.

The original author of this patch is Jeffery To.
I am the submitter to this repo.
We're both co-maintainers of the Python3 package in the OpenWrt realm.

Signed-off-by: Jeffery To jeffery.to@gmail.com
Signed-off-by: Alexandru Ardelean ardeleanalex@gmail.com

@jefferyto @commodo
pep517: in_process: try _in_process.pyc file if _in_process.py doesn'…
c610d55 
…t exist

This change came about to exist after this bug report:
  openwrt/packages#11912

The general context is that OpenWrt does not ship Python source-code by
default. Instead, the build byte-compiles all .py files into .pyc files and
removes the .py files.
There's a wide reasoning for this, from performance to reducing the size of
these packages.

This change is kept in the Python3 build as this patch:
  https://github.com/openwrt/packages/blob/master/lang/python/python3/patches-pip/001-pep517-pyc-fix.patch

This is an attempt to start a discussion, to see whether it makes sense to
remove it from the OpenWrt packages feed, and live in the pip source-base.

The original author of this patch is Jeffery To.
I am the submitter to this repo.
We're both co-maintainers of the Python3 package in the OpenWrt realm.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
commodo added a commit to commodo/packages that referenced this pull request Sep 8, 2021
@commodo
python3: bump version to 3.9.7
c3bb374 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using cat + echo.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
commodo added a commit to commodo/packages that referenced this pull request Sep 8, 2021
@commodo
python3: bump version to 3.9.7
d0a9c22 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using cat + echo.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
commodo added a commit to commodo/packages that referenced this pull request Sep 8, 2021
@commodo
python3: bump version to 3.9.7
0decb46 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
commodo added a commit to commodo/packages that referenced this pull request Sep 8, 2021
@commodo
python3: bump version to 3.9.7
114d2b6 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
@commodo commodo mentioned this pull request Sep 8, 2021
Merged
neheb pushed a commit to openwrt/packages that referenced this pull request Sep 9, 2021
@commodo @neheb
python3: bump version to 3.9.7
26988f9 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
commodo added a commit to commodo/packages that referenced this pull request Sep 10, 2021
@commodo
python3: bump version to 3.9.7
9c0a52d 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
(cherry picked from commit 26988f9)
@commodo commodo mentioned this pull request Sep 10, 2021
Merged
neheb pushed a commit to openwrt/packages that referenced this pull request Sep 10, 2021
@commodo @neheb
python3: bump version to 3.9.7
e72cc2b 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
(cherry picked from commit 26988f9)
1715173329 pushed a commit to immortalwrt/packages that referenced this pull request Sep 10, 2021
@commodo @1715173329
python3: bump version to 3.9.7
9d39f30 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
Signed-off-by: Tianling Shen <cnsztl@immortalwrt.org>
@commodo
Copy link
Copy Markdown
Author

commodo commented Oct 27, 2021

ping on this;

not saying this is the best solution, but i would like an idea on how to handle this better;

i'm fine to implement it

utoni pushed a commit to utoni/openwrt-packages that referenced this pull request Jan 21, 2022
@commodo @utoni
python3: bump version to 3.9.7
f5d30b8 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
graysky2 pushed a commit to graysky2/packages that referenced this pull request Feb 3, 2022
@commodo @graysky2
python3: bump version to 3.9.7
55e6666 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
graysky2 pushed a commit to graysky2/packages that referenced this pull request Feb 6, 2022
@commodo @graysky2
python3: bump version to 3.9.7
8d338cd 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
breakings pushed a commit to breakings/packages that referenced this pull request Mar 5, 2022
@commodo
python3: bump version to 3.9.7
e69790f 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
(cherry picked from commit 26988f9)
@commodo
Copy link
Copy Markdown
Author

commodo commented Mar 16, 2022

ping on this :)

utoni pushed a commit to utoni/openwrt-packages that referenced this pull request May 30, 2022
@commodo @utoni
python3: bump version to 3.9.7
9014ae4 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
@ghost ghost mentioned this pull request Aug 26, 2022
Open
@pfmoore
Copy link
Copy Markdown
Member

pfmoore commented Aug 26, 2022

Assuming the .pyc is in the same location as the .py is wrong. You should probably use importlib.util.cache_from_source, but even with that, there could be problems - for example running pip from a metapath hook that references a sourceless pip1 - which should be tested.

Footnotes

  1. This would happen in real life when setting up an isolated build environment from a sourceless pip, for example. ↩

@commodo
Copy link
Copy Markdown
Author

commodo commented Aug 29, 2022

Assuming the .pyc is in the same location as the .py is wrong. You should probably use importlib.util.cache_from_source, but even with that, there could be problems - for example running pip from a metapath hook that references a sourceless pip1 - which should be tested.

Footnotes

1. This would happen in real life when setting up an isolated build environment from a sourceless pip, for example. [leftwards_arrow_with_hook](#user-content-fnref-1-de8c8293481778e25b7a0f5852353604)

phew, it's been a while since i opened this PR;
anyway the use-case for OpenWrt is a bit specific (with regards to .pyc files); we store them along-side with .py usually;

i'll re-review this and comeback;

@pradyunsg
Copy link
Copy Markdown
Member

pradyunsg commented Nov 24, 2022

Closing this out since its not been updated in a while and there's some significant changes needed for this to even be considered.

Let's keep the discussion in #148 before we get back to this.

@pradyunsg pradyunsg closed this Nov 24, 2022
@commodo commodo deleted the in_process_pyc branch December 9, 2022 14:29
ineedfat pushed a commit to ineedfat/rockchip_rk3568_openwrt_packages that referenced this pull request Jun 7, 2024
@commodo @ineedfat
python3: bump version to 3.9.7
29b0b3f 
Refreshed patches.
And apply hack for line-endings in pep517 (from pip).

Hack comment:
  # FIXME: [1] get rid of this asap; 'patch' doesn't like Windows endings, and this file is full of them...
  #        I actually tried this in a number of ways and the only way to fix this is to implement
  #        a poor-man's dos2unix using sed.
  #        The issue is with the pip package; it seems that it throws in some Windows line-endings
  #        and 'patch' won't handle them. So, we do a "dos2unix" and then patch.
  #        We can get rid of this once this is solved upstream and in pip:
  #            pypa/pyproject-hooks#130

Signed-off-by: Alexandru Ardelean <ardeleanalex@gmail.com>
(cherry picked from commit 26988f9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@commodo @pfmoore @pradyunsg @jefferyto