PyCryptodome as replacement for PyCrypto

[drts01]

Description of Issue/Question

PyCrypto is unmaintained. And sometimes fails to build. There several issues open about PyCrypto failing on Python 3, but this is the longest thread, pycrypto/pycrypto#218. And definitely does not build with Pypy3.

PyCryptodome is a fork and (mostly) API compatible replacement, see https://pycryptodome.readthedocs.io/en/latest/src/vs_pycrypto.html.

Adapting Salt for PyCryptodome is a faster near-term solution rather than implementing #30287 or making the crypto backend modular (#44923 (comment))

Versions Report

At least salt 2019.2.0 (Fluorine) and earlier.

[dmurphy18]

@CMeza99 Thanks for your suggestion but but there is already the ability to utilize PyCryptodome instead of PyCrypto (on Redhat family) for those who desire. Currently SaltStack makes available a version of PyCryptodomex (Redhat family) and M2Crypto (all Linux platforms), and there is a picking order as to which package is used as follows:

PyCrypto - basic level
PyCryptodomx - preferred over PyCrypto if installed
M2Crypto - preferred over PyCryptodome and PyCrypto if installed

This pecking order is present on both the 2018.3 and 2019.2 branches.
Support for #30287 if and when, is probably going to occur in Python 3 given the short runway that Python 2.7 has.

Hope this clarifies the issue for you, that PyCyrptodome and M2Crypto can already be used as alternatives to PyCrypto. If this is sufficient information, please consider closing this issue unless there is something else I can help clarify about this for you.

[drts01]

Thanks @dmurphy18. So I suppose my issue is that pycrpto installs as a dependency regardless if one of the preferred libraries are installed.

M2Crypto==0.32.0
pip==19.1
salt==2019.2.0
  - Jinja2 [required: Any, installed: 2.10.1]
    - MarkupSafe [required: >=0.23, installed: 1.1.1]
  - MarkupSafe [required: Any, installed: 1.1.1]
  - msgpack [required: >=0.5,!=0.5.5, installed: 0.6.1]
  - pycrypto [required: >=2.6.1, installed: 2.6.1]

---

I did some digging and it looks like PyCrypto is set as a dependency for ZeroMQ. Is this is necessary? https://github.com/saltstack/salt/blob/develop/requirements/zeromq.txt#L3

[dmurphy18]

@CMeza99 Actually Salt depends on python-crypto in packaged form (commonly PyCrypto) as a hard depend, and has done for years. It is unlikely to be removed from Python 2.7 packages at this point.

Need to cleanup those requirement files when using pip

root@c7master:~# yum deplist salt
Loaded plugins: fastestmirror, langpacks
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Loading mirror speeds from cached hostfile

• base: mirrors.xmission.com
• epel: mirrors.xmission.com
• extras: mirrors.xmission.com
• os_pkgs_repo: mirrors.xmission.com
• updates: mirrors.xmission.com
  package: salt.noarch 2019.2.0-1.el7
  dependency: /bin/sh
  provider: bash.x86_64 4.2.46-31.el7
  dependency: /usr/bin/env
  provider: coreutils.x86_64 8.22-23.el7
  dependency: /usr/bin/python
  provider: python.x86_64 2.7.5-76.el7
  dependency: PyYAML
  provider: PyYAML.x86_64 3.11-1.el7
  dependency: pciutils
  provider: pciutils.x86_64 3.5.1-3.el7
  dependency: python(abi) = 2.7
  provider: python.x86_64 2.7.5-76.el7
  dependency: python-crypto >= 2.6.1
  provider: python2-crypto.x86_64 2.6.1-15.el7
  provider: python-crypto.x86_64 2.6.1-2.el7

When Salt moves to an all Python 3 client (no Python 2.7 built or involved) can look again at removing it, but until the issue with Salt and python-tornado > 5.0 on Python 3 is resolved, have to have Python 2.7 around to build Python 3 packages. Note that fix is currently scheduled for the Neon release.

[drts01]

@dmurphy18 thank you for the info.

I will close this issue for now as there are blockers that will be addressed in an upcoming version. Though it is confusing that the issue is with Salt and python-tornado, but PyCrypto is coming as part of requirements/zeromq.txt.

[udf2457]

@dmurphy18, re: your comment to @CMeza99, may I point out this quote from the PyCryptodome man page (https://pycryptodome.readthedocs.io/en/latest/src/installation.html):

One must avoid having both PyCrypto and PyCryptodome installed at the same time, as they will interfere with each other.

It is therefore in my view imperative that Salt needs to eradicate its dependence on the unsupported PyCrypto ASAP.

[dmurphy18]

@udf2457 I would very much like to drop pycrypto however I still have to support OS's such as Debian 8 (until June 30th) till they are effectively EOL and there is no pycryptodome support on Debian 8. Similarly I ran into issues with pycryptodomex in 32-bit mode on RHEL 6 which does not end of life until 30th Oct 2020. My preference would be for later versions of M2Crypto as used on Py 3 RHEL 7 & 8 as supplied and installed by SaltStack packages.

The pycrypto issue arose two days before final v3000 tag, and it was the quickest way to resolve the issue and not miss the already delayed release. This use of pycrypto was no different that what had been released in previous releases, that is, unchanged from previous releases. Efforts are been made in packaging to install either pycryptodome[x] or M2Crypto where possible for an OS in the next release, and if I could drop Debian 8 (Raspbian 8 too), RHEL 6 and Amazon Linux 1 support, it would be gone already; I have to wait for those OS's to EOL.

I would gladly accept a contribution on a backported version of pycryptodome[x] running on those platforms in order to be rid of pycrypto, note: remember Raspbian 8 is Debian 8 on ARM HL.

[udf2457]

@dmurphy18 Bring on June 2020, finally the demise of Debian 8 !

M2Crypto is not available on Debian 10 for python3, only the obsolete python2:

$ sudo apt-cache search M2Crypto
m2crypto-doc - Python wrapper for the OpenSSL library (docs)
python-m2crypto - Python wrapper for the OpenSSL library (Python 2 modules)
python-m2ext - Extensions to the M2Crypto Python package

Regardign your last paragraph (contribution). I don't do python, not even in newbie spaghetti-code format. So can't help you there. Unfortunatley.

[ymasson]

@udf2457 python3-m2crypto is available in buster-backports.

root@845d019ba068# apt-cache search M2Crypt
m2crypto-doc - Python wrapper for the OpenSSL library (docs)
python-m2crypto - Python wrapper for the OpenSSL library (Python 2 modules)
python-m2ext - Extensions to the M2Crypto Python package
python3-m2crypto - Python wrapper for the OpenSSL library (Python 3 modules)

root@845d019ba068# apt-cache policy python3-m2crypto
python3-m2crypto:
  Installed: (none)
  Candidate: 0.31.0-6~bpo10+1
  Version table:
     0.31.0-6~bpo10+1 100
        100 http://deb.debian.org/debian buster-backports/main amd64 Packages

[udf2457]

@ymasson All very well if your environment permits the use of buster-backports.

Some of us work with environments where the double-edged sword warnings posted on the buster-backports page need to be heeded:

backports are provided on an as-is basis, with risk of incompatibilities with other components in Debian stable. Use with care!