Many dependencies are out of date

[maxbucknell]

This is not really about the bug I'm raising. That's a vessel to which I'm attaching a larger question of outdated dependencies. The bug is a PITA, though.

Preconditions

For example, today I was trying to compile static assets on an Alpine Linux Docker container. I was unable to, because there is a bug in zendframework/zend-stdlib around usage of the GLOB_BRACE constant, which may or may not be defined depending on the environment, and isn't defined in Alpine Linux.

Environment Details

Using an Alpine Linux PHP Docker container, with the required extensions for running Magento:

• PHP 7.0 (Alpine Linux)
• Magento 2.1.3

Steps to Reproduce

Run bin/magento set:sta:dep.

Expected Result

The command executes successfully.

Actual Result

The following:

 [Exception]
  Notice: Use of undefined constant GLOB_BRACE - assumed 'GLOB_BRACE' in /mnt/www/vendor/zendframework/zend-stdlib/src/Glob.php on line 64

Additional

Of course, I have more to say about this. This is obviously not a bug with Magento, it is a bug with the Zend Framework, and admittedly an edge case. I almost didn't raise this, because I can't imagine it rates highly as a priority.

The catch is that this issue was raised with them, and fixed about 8 months ago. The irony here is that the original issue was created by somebody using Magento!

The problem now is that Magento is not requiring a recent enough version of zendframework/zend-stdlib to take advantage of this wonderful fix that lets me run Magento on a sub 50MB Docker image. So, I would like to open a wider conversation about how we keep dependencies up to date.

Yes we should use the lock file, and yes we should be cautious about unpinned dependencies and Magento is doing everything right so far. But the fact that the version of the Zend Framework currently required by Magento is 2.4.6 tells me that this was set in August 2015, and then not revisited. This is a failing. Perhaps there is a reason for staying on 2.4, and of course updates should be tested rigorously to check for breaking changes, but I don't think anyone knows for sure that the latest version of the Zend Framework causes issues. I don't think anyone has tried.

It's not just the Zend Framework, either. We can run the handy Composer command, composer outdated. On 2.1.3, I obtain this output:

braintree/braintree_php                       3.7.0       3.20.0         Braintree PHP Client Library
colinmollenhour/credis                        1.6         1.7            Credis is a lightweight interface to the Redis key-value store which wraps the phpredis library when available for better performance.
colinmollenhour/php-redis-session-abstract    v1.1        v1.3.1         A Redis-based session handler with optimistic locking
composer/composer                             1.0.0-beta1 1.3.1          Composer helps you declare, manage and install dependencies of PHP projects, ensuring you have the right stack everywhere.
elasticsearch/elasticsearch                   v2.3.0      v5.1.0         PHP Client for Elasticsearch
fabpot/php-cs-fixer                           v1.13.1     v2.0.0         A tool to automatically fix PHP code style
Package fabpot/php-cs-fixer is abandoned, you should avoid using it. Use friendsofphp/php-cs-fixer instead.
justinrainbow/json-schema                     1.6.1       4.1.0          A library to validate a json schema.
league/climate                                2.6.1       3.2.1          PHP's best friend for the terminal. CLImate allows you to easily output colored text, special formats, and more.
lusitanian/oauth                              v0.7.0      v0.8.9         PHP 5.3+ oAuth 1/2 Library
magento/zendframework1                        1.12.16     1.12.16-patch1 Magento Zend Framework 1
monolog/monolog                               1.16.0      1.22.0         Sends your logs to files, sockets, inboxes, databases and various web services
pdepend/pdepend                               2.2.2       2.3.2          Official version of pdepend to be handled with Composer
pelago/emogrifier                             v0.1.1      V1.1.0         Converts CSS styles into inline style attributes in your HTML code
php-amqplib/php-amqplib                       v2.5.2      v2.6.3         This library is a pure PHP implementation of the AMQP protocol. It's been tested against RabbitMQ.
phpunit/php-code-coverage                     2.2.4       4.0.4          Library that provides collection, processing, and rendering functionality for PHP code coverage information.
phpunit/php-file-iterator                     1.3.4       1.4.2          FilterIterator implementation that filters files based on a list of suffixes.
phpunit/phpunit                               4.1.0       5.7.5          The PHP Unit Testing framework.
phpunit/phpunit-mock-objects                  2.3.8       3.4.3          Mock Object library for PHPUnit
sebastian/environment                         1.3.8       2.0.0          Provides functionality to handle HHVM/PHP environments
sebastian/exporter                            1.2.2       2.0.0          Provides the functionality to export PHP variables for visualization
sebastian/phpcpd                              2.0.0       2.0.4          Copy/Paste Detector (CPD) for PHP code.
sebastian/recursion-context                   1.0.2       2.0.0          Provides functionality to recursively process PHP variables
sebastian/version                             1.0.6       2.0.1          Library that helps with managing the version number of Git-hosted PHP projects
sjparkinson/static-review                     4.1.1       5.2.0          An extendable framework for version control hooks.
solarium/solarium                             3.3.0       3.7.0          PHP Solr client
squizlabs/php_codesniffer                     1.5.3       2.7.1          PHP_CodeSniffer tokenises PHP, JavaScript and CSS files and detects violations of a defined set of coding standards.
symfony/config                                v2.8.15     v3.2.1         Symfony Config Component
symfony/console                               v2.6.13     v3.2.1         Symfony Console Component
symfony/dependency-injection                  v2.8.15     v3.2.1         Symfony DependencyInjection Component
symfony/event-dispatcher                      v2.8.15     v3.2.1         Symfony EventDispatcher Component
symfony/filesystem                            v2.8.15     v3.2.1         Symfony Filesystem Component
symfony/process                               v2.8.15     v3.2.1         Symfony Process Component
symfony/yaml                                  v2.8.15     v3.2.1         Symfony Yaml Component
tedivm/jshrink                                v1.0.1      v1.1.0         Javascript Minifier built in PHP
zendframework/zend-code                       2.4.11      3.1.0          provides facilities to generate arbitrary code using an object oriented interface
zendframework/zend-config                     2.4.11      2.6.0          provides a nested object property based user interface for accessing this configuration data within application code
zendframework/zend-console                    2.4.11      2.6.0          
zendframework/zend-crypt                      2.4.11      3.2.0          
zendframework/zend-di                         2.4.11      2.6.1          
zendframework/zend-escaper                    2.4.11      2.5.2          
zendframework/zend-eventmanager               2.4.11      3.1.0          
zendframework/zend-filter                     2.4.11      2.7.1          provides a set of commonly needed data filters
zendframework/zend-form                       2.4.11      2.9.2          
zendframework/zend-http                       2.4.11      2.5.5          provides an easy interface for performing Hyper-Text Transfer Protocol (HTTP) requests
zendframework/zend-i18n                       2.4.11      2.7.3          
zendframework/zend-inputfilter                2.4.11      2.7.3          
zendframework/zend-json                       2.4.11      3.0.0          provides convenience methods for serializing native PHP to JSON and decoding JSON to native PHP
zendframework/zend-loader                     2.4.11      2.5.1          
zendframework/zend-log                        2.4.11      2.9.1          component for general purpose logging
zendframework/zend-math                       2.4.11      3.0.0          
zendframework/zend-modulemanager              2.4.11      2.7.2          
zendframework/zend-mvc                        2.4.11      3.0.4          
zendframework/zend-serializer                 2.4.11      2.8.0          provides an adapter based interface to simply generate storable representation of PHP types by different facilities, and recover
zendframework/zend-server                     2.4.11      2.7.0          
zendframework/zend-servicemanager             2.4.11      3.2.0          
zendframework/zend-soap                       2.4.11      2.6.0          
zendframework/zend-stdlib                     2.4.11      3.1.0          
zendframework/zend-text                       2.4.11      2.6.0          
zendframework/zend-uri                        2.4.11      2.5.2          a component that aids in manipulating and validating » Uniform Resource Identifiers (URIs)
zendframework/zend-validator                  2.4.11      2.8.1          provides a set of commonly needed validators
zendframework/zend-view                       2.4.11      2.8.1          provides a system of helpers, output filters, and variable escaping

I think this is important. The first rule of staying secure online is "Keep your software updated". Magento implores us to update our version of Magento regularly, and I think it's worth Magento doing the same with its dependencies.

I don't want to just send in a pull request that updates all of these, because that doesn't help anyone. But I do believe there should be some process attached to reviewing this list and updating on a case by case basis.

[antonkril]

We're reviewing our development process now. And I agree that we need a formal procedure for dependency update.

[maxbucknell]

It's worth adding that the particular issue described above can be fixed by upgrading zendframework/zend-stdlib manually. Behold:

"require": {
    // ...
    "zendframework/zend-stdlib": "2.7.7. as 2.4.11"
},

The version alias is necessary because Magento is requiring a different version of that package. I have been using this for a while and noticed no regressions.

[TiEul]

Some of these outdated dependencies are seriously creating dependency hell now and they will continue to do so in the future. For example, because of sjparkinson/static-review still being ~4.1, it uses the outdated version 2 of league/climate, which is incompatible with version 3, while other projects have long moved away from version 2. It's a complete mess to work around this and it's rather frustrating to see that it's caused by such a "non-core" dependency.

I am not expecting this to be fixed for 2.1 anymore, but please try and make this a change that makes it into 2.2.

[orlangur]

@TiEul sjparkinson/static-review is a require-dev dependency, how it can affect your project? In your custom project it could be even removed as it is used only in git hook.

[TiEul]

@orlangur Correct me if I am wrong, but if you check the composer.json of Magento 2, you can see it's in the require block. Installing Magento 2 will install sjparkinson/static-review and as such the outdated league/climate dependency, I don't see how it would be possible to prevent that (and even if, it would still be a nasty solution). I have now worked around it, but it was still a major pain.

[orlangur]

OMG :) Nice catch, you are right, it is in require section currently https://github.com/magento/magento2/blob/develop/composer.json#L71

It was in require-dev when I introduced it and in 2.0 https://github.com/magento/magento2/blob/2.0/composer.json#L70

I'll check for any other dependencies wrongly moved out of require-dev section and prepare a PR.

[orlangur]

Hi again, guys! :)

I think this is important. The first rule of staying secure online is "Keep your software updated". Magento implores us to update our version of Magento regularly, and I think it's worth Magento doing the same with its dependencies.

Recently I found out that there is a better way to keep dependencies secure than just try to update all of them as frequently as possible or to monitor security lists. SensioLabsInsight has such control point: "Projects must not depend on dependencies with known security issues".

It is not too expensive (probably even some special agreement is possible considering the scale of Magento 2 product).

Ideally would be to integrate it for PRs the way similar to codacy/pr (Cc: @antonkril as Codacy seems to be tied to you). As a consequence, with the power of properly coordinated crowdsourcing platinum medal could be achieved pretty quick eliminating many other code smell issues.

[mzeis]

I like the idea by @orlangur, this would be a start. What would everybody think about this?

My suggestion with proceeding here is 1. re-run the outdated check against 2.2 and 2. splitting up this issue into more granular issues. Also we may have to accept (didn't check for this) that M2 could use another library in a current version which depends on outdated libraries.

[TiEul]

Hey @orlangur, Magento 2.2 was just released but it looks like a lot of dependencies were not updated or only updated months ago while newer stable releases already exist. Do you have any clue if any plans were made or discussed some other place to get these updated more frequently? Going back to my example from March, is there a reason why sjparkinson/static-review is still in the require block instead of require-dev and in addition to it is still using a pretty outdated version?

[magento-engcom-team]

HI @maxbucknell Thank you for your report. We have the task in our backlog for components update. it would be really helpful if you can submit a pull request for the components which you would like to update. It would be better to have separate PRs for each component so that we can track and test in individually. Closing this issue.