Improve generation of required php extensions

[hostep]

Bug report

Description

The list of required php extensions in https://devdocs.magento.com/guides/v2.4/install-gde/system-requirements-tech.html#required-php-extensions is not accurate.

Currently it's generated from the composer.json file of the latest Magento version, but that's not accurate enough since all its dependency packages are not taken into account.

I suggested a possible solution in #7398 (comment) (also #7398 (comment)) to get a more accurate list, maybe someone can try that and see if that works and then somehow try to automate this.

Steps to reproduce

1. Look at list of php extensions on https://devdocs.magento.com/guides/v2.4/install-gde/system-requirements-tech.html#required-php-extensions

Expected result

1. Finding ext-sockets which became required in Magento 2.4.0

Possible solutions

1. Not finding ext-sockets which became required in Magento 2.4.0

Additional information

This issue was opened at the request of @dshevtsov

[m2-assistant]

Hi @hostep. Thank you for your report.
To help us process this issue please make sure that you provided sufficient information.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[dshevtsov]

Related PR with a workaround: #7953

[dshevtsov]

The input I've got from the packaging team:

the best and the only correct way is rely on the composer.json. Dependency on ext-sockets came from another dependency of Magento php-amqplib/php-amqplib. We should not document dynamic dependencies.
If we have real dependency on ext-sockets in the code and don’t have it in the composer.json - this needs to be fixed

[hostep]

Sorry, but that's just incorrect, Magento code strongly relies on code in the dependent packages, so their dependencies are implicitly also dependencies Magento needs.

Just try executing composer install with Magento 2.4.0 on a system where only the php extensions listed in the composer.json file are installed, it's not going to work.

[hostep]

Code of Magento and code called in its dependencies might not need ext-sockets, but composer will not allow you to install Magento if the required dependencies (including php extensions) aren't installed. Therefore we should document all php extensions required to be able to install Magento.

[dshevtsov]

cc @fascinosum

[dshevtsov]

Just try executing composer install with Magento 2.4.0 on a system where only the php extensions listed in the composer.json file are installed, it's not going to work.

In DevDocs, we use a Docker project that reproduces Magento installation using Composer on an environment with listed prerequisites to test the documentation. Here is the script for an environment setup for 2.4.0.

#!/bin/bash
set -x
apt-get update
export DEBIAN_FRONTEND=noninteractive


# INSTALL MAIN TOOLS
# https://help.ubuntu.com/community/ApacheMySQLPHP
apt-get install -y  curl \
                    git \
                    mysql-server \
                    php \
                    software-properties-common \
                    unzip \
                    vim \
                    wget

# INSTALL PHP EXTENSIONS
# Required: https://devdocs.magento.com/guides/v2.4/install-gde/system-requirements-tech.html#required-php-extensions
# Available: 'apt-cache search php7.4'
# Installation details:
#   ctype: selecting 'php7.4-common' instead of 'php7.4-ctype'
#   dom: selecting 'php7.4-xml' instead of 'php7.4-dom'
#   hash: As of PHP 5.1.2, the Hash extension is bundled and compiled into PHP by default.
#   iconv: System: Note, selecting 'php7.4-common' instead of 'php7.4-iconv'
#   openssl: Already present and enabled in the system: 'php --ri openssl'
#   pdo_mysql: https://www.php.net/manual/en/ref.pdo-mysql.php
#   simplexml: selecting 'php7.4-xml' instead of 'php7.4-simplexml'
#   xsl: installed with 'php7.4-xml'

apt-get install -y  php-bcmath \
                    php-curl \
                    php-xml \
                    php-gd \
                    php-intl \
                    php-mbstring \
                    php-mysql \
                    php-soap \
                    php-zip

# INSTALL ELASTICSEARCH
# https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add -
apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt-get update
apt-get install elasticsearch

# INSTALL COMPOSER GLOBALLY
# https://getcomposer.org/doc/faqs/how-to-install-composer-programmatically.md
EXPECTED_CHECKSUM="$(wget -q -O - https://composer.github.io/installer.sig)"
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
ACTUAL_CHECKSUM="$(php -r "echo hash_file('sha384', 'composer-setup.php');")"

if [ "$EXPECTED_CHECKSUM" != "$ACTUAL_CHECKSUM" ]
then
    >&2 echo 'ERROR: Invalid installer checksum'
    rm composer-setup.php
    exit 1
fi

php composer-setup.php --quiet
RESULT=$?
rm composer-setup.php
mv composer.phar /usr/local/bin/composer

exit $RESULT

[dshevtsov]

It's for ubuntu 20.04

[dshevtsov]

And for Magento installation:

#!/bin/bash
set -x

# For COMPOSER_CREDS see https://devdocs.magento.com/guides/v2.4/install-gde/prereq/connect-auth.html
composer create-project --repository-url=https://${COMPOSER_CREDS}@repo.magento.com/ magento/${EDITION}=${VERSION} /var/www/html/magento2

# RUN AND CONFIGURE MYSQL
service mysql start
mysql -u root -p\n <<SQL
create database magento;
create user 'magento'@'localhost' IDENTIFIED BY 'magento';
GRANT ALL ON magento.* TO 'magento'@'localhost';
flush privileges;
SQL

# RUN ELASTICSEARCH
service elasticsearch start

# INSTALL MAGENTO
# https://devdocs.magento.com/guides/v2.4/install-gde/install/cli/install-cli-install.html#install-cli-example
/var/www/html/magento2/bin/magento setup:install \
                                      --base-url=http://127.0.0.1/magento2/ \
                                      --db-host=localhost \
                                      --db-name=magento \
                                      --db-user=magento \
                                      --db-password=magento \
                                      --admin-firstname=Magento \
                                      --admin-lastname=User \
                                      [email protected] \
                                      --admin-user=admin \
                                      --admin-password=admin123 \
                                      --language=en_US \
                                      --currency=USD \
                                      --timezone=America/Chicago \
                                      --use-rewrites=1

[hostep]

@dshevtsov: maybe sockets is included by default on this particular environment with php, not all users of Magento will use this specific php installation. The docs should make it clear for everybody what is needed exactly.

I just tested this again on my local environment where I've explicitly disabled the sockets php extension, this is the result:

Creating a "magento/project-community-edition:2.4.*" project at "./"
Installing magento/project-community-edition (2.4.0)
  - Installing magento/project-community-edition (2.4.0): Loading from cache
Created project in /Volumes/Projects/magento2/magento24-test/.
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - php-amqplib/php-amqplib v2.10.1 requires ext-sockets * -> the requested PHP extension sockets is missing from your system.
    - php-amqplib/php-amqplib v2.10.0 requires ext-sockets * -> the requested PHP extension sockets is missing from your system.
    - magento/product-community-edition 2.4.0 requires php-amqplib/php-amqplib ~2.10.0 -> satisfiable by php-amqplib/php-amqplib[v2.10.0, v2.10.1].
    - Installation request for magento/product-community-edition 2.4.0 -> satisfiable by magento/product-community-edition[2.4.0].

[dshevtsov]

The docs should make it clear for everybody what is needed exactly.

I agree with that. But since the application is huge and grows rapidly, and our resources are restricted, DevDocs focuses on the content that is directly related to the product. In this case, project dependencies vs dynamic dependencies. Content that is indirectly related to the product typically goes to the Magento Help Center. We cannot control dynamic dependencies, they can change at any time by the moment someone creates a new project, or runs composer install for the first time, or composer update.

And I see your point, that failed installation caused by missing components can be very confusing. Developers do not expect issues when they follow guidelines from official documentation. If we want to add the bare minimum to documentation that would include dynamic dependencies, it has to be automated and separated from project dependencies. The way to get extensions required for production could be composer check-platform-reqs --no-dev.

We could add a subsection that would contain an example list generated at the time of release (or close to it) such as:

Make sure your platform complies with requirements (items and versions can vary):

/var/www/html/magento2# composer check-platform-reqs --no-dev
composer-plugin-api  1.1.0       success  
ext-bcmath           7.4.3       success  
ext-ctype            7.4.3       success  
ext-curl             7.4.3       success  
ext-dom              20031129    success  
ext-gd               7.4.3       success  
ext-hash             7.4.3       success  
ext-iconv            7.4.3       success  
ext-intl             7.4.3       success  
ext-json             7.4.3       success  
ext-libxml           7.4.3       success  
ext-mbstring         7.4.3       success  
ext-openssl          7.4.3       success  
ext-pcre             7.4.3       success  
ext-pdo_mysql        7.4.3       success  
ext-SimpleXML        7.4.3       success  
ext-soap             7.4.3       success  
ext-sockets          7.4.3       success  
ext-xmlwriter        7.4.3       success  
ext-xsl              7.4.3       success  
ext-zip              1.15.6      success  
lib-libxml           2.9.10      success  
php                  7.4.3       success

[dshevtsov]

cc @meker12 , @dobooth

[hostep]

Look, please ask yourself the question why this documentation exist.
I'm going to give you one example.

Suppose a shop owner wants to setup a new Magento shop. He finds a good hosting company and the hosting company asks: "Well, what software should we install on your system?" The shop owner answers with a link to the system requirements documentation from the devdocs site, the hoster starts setting up everything and when push comes to shove and the shop owner wants to install Magento, it will turn out that certain things are still missing. Which is not good.

We - as an agency - have our own documentation for hosting companies with all the requirements, since we don't trust the devdocs system requirements, which is unfortunate.

Also: these "dynamic dependencies", their reliance on php extensions only changes very rarely. So if we would have a script to automatically generate the list of php extensions needed for Magento and run that with every new release of Magento, that would suffice more then enough in my opinion.

I've looked into check-platform-reqs a few months ago when we had our initial discussion about this, and it was not accurate in my opinion back then, but I don't remember why. I would still recommend you try the steps mention in that discussion, it should be a lot more accurate in theory.

[hostep]

Just quickly tried something with the help of a json cli parser:

1. Setup a new Magento OS 2.4.1 installation using composer
2. Executed:

$ cat composer.lock | jq '.packages[] .require | keys?' | grep 'ext-' | tr -d '",' | sort | uniq
  ext-bcmath
  ext-ctype
  ext-curl
  ext-dom
  ext-gd
  ext-hash
  ext-iconv
  ext-intl
  ext-json
  ext-libxml
  ext-mbstring
  ext-openssl
  ext-pcre
  ext-pdo_mysql
  ext-simplexml
  ext-soap
  ext-sockets
  ext-xmlwriter
  ext-xsl
  ext-zip

3. Ran composer update --prefer-lowest
4. Executed:

$ cat composer.lock | jq '.packages[] .require | keys?' | grep 'ext-' | tr -d '",' | sort | uniq
  ext-bcmath
  ext-ctype
  ext-curl
  ext-dom
  ext-gd
  ext-hash
  ext-iconv
  ext-intl
  ext-json
  ext-libxml
  ext-mbstring
  ext-openssl
  ext-pcre
  ext-pdo_mysql
  ext-simplexml
  ext-soap
  ext-sockets
  ext-xmlwriter
  ext-xsl
  ext-zip

If we look at the difference here, there is none, so all the extensions listed here are required for Magento 2.4.1

If we then do the same for Magento 2.3.6, here are the results:
Just as is:

 ext-bcmath
  ext-ctype
  ext-curl
  ext-dom
  ext-gd
  ext-hash
  ext-iconv
  ext-intl
  ext-json
  ext-libxml
  ext-mbstring
  ext-openssl
  ext-pcre
  ext-pdo_mysql
  ext-simplexml
  ext-soap
  ext-sockets
  ext-xmlwriter
  ext-xsl
  ext-zip

After --prefer-lowest:

  ext-bcmath
  ext-ctype
  ext-curl
  ext-dom
  ext-gd
  ext-hash
  ext-iconv
  ext-intl
  ext-json
  ext-mbstring
  ext-openssl
  ext-pcre
  ext-pdo_mysql
  ext-simplexml
  ext-soap
  ext-xmlwriter
  ext-xsl
  ext-zip

If we compare both lists, there are 2 differences here:

@@ -7,14 +7,12 @@
   ext-iconv
   ext-intl
   ext-json
-  ext-libxml
   ext-mbstring
   ext-openssl
   ext-pcre
   ext-pdo_mysql
   ext-simplexml
   ext-soap
-  ext-sockets
   ext-xmlwriter
   ext-xsl
   ext-zip

So in the case of Magento 2.3.x, it means that the sockets php extension isn't required, as is that ext-libxml entry (which seems to have been a mistake)

Also: I don't know if we also have to parse out the lib-* dependencies, not exactly sure what the differences are between ext-* and lib-* ...
Update: you can run cat composer.lock | jq '.packages[] .require | keys?' | egrep '"(ext|lib)-' | tr -d '",' | sort | uniq to fetch those as well...

If I compare the list from my test with Magento 2.4.1 with the devdocs requirements, that list is missing the following extensions:

- json
- pcre
- xmlwriter

[dshevtsov]

@hostep, The packaging team confirmed that this can be used to publish dynamic dependencies. They also recommend to generate both lists for OpenSource and Commerce, because they can be different, even though in most cases they are the same. We will be implementing this into Docs automation.

[medigeek]

we need to list ext-fileinfo
explained previously in #8933

[dshevtsov]

@medigeek, per #8933 (comment), the dependency is not required since the 2.4.2 version.

[dshevtsov]

Internal ticket: DOC-362

[dshevtsov]

I'm going to use the following logic for the suggested scenario:

composer update
EXT_LIST=$(cat composer.lock | jq '.packages[] .require | keys?' | egrep '"(ext|lib)-' | tr -d '",' | sort | uniq)
composer update --prefer-lowest
LOWEST_EXT_LIST=$(cat composer.lock | jq '.packages[] .require | keys?' | egrep '"(ext|lib)-' | tr -d '",' | sort | uniq)

comm -12 <(echo $EXT_LIST | tr ' ' '\n') <(echo $LOWEST_EXT_LIST | tr ' ' '\n')

[dshevtsov]

Resolved in 3ab7dec

[hostep]

Awesome, thanks @dshevtsov !