Caching between runs for better performance

[JDGrimes]

First let me thank you for this great tool. 👍

I've been using this on my PHP projects, and I've found that it can take a while to sniff the code, especially on larger projects with complex configurations. Performance will naturally be determined largely by how well the sniffs used are written. However, I think that performance could be increased by caching the hash signatures of the files being sniffed. Then only those files which have changes since the last sniff was conducted would need to be sniffed (there are some caveats which I'll get to in a moment). This wouldn't improve the performance of the initial sniffing (and might even degrade it slightly), but would drastically improve performance for latter sniffings.

As I noted above, there are some caveats:

• The cache needs to be invalidated when the configuration changes. This would be when the XML config file is edited, for example, and possibly at other times as well. This could be facilitated by saving a hash signature of these configuration settings to the cache, and checking whether this matches the current configuration when the cache is being loaded.
• If a file has errors, the errors still need to get reported on subsequent sniffings even if the file hasn't changed. This could be done by not caching files with errors, or by caching a list of the errors found and replaying them to the user without sniffing those files.

There are probably other things I haven't thought of, maybe regarding interactive mode, reports, or automatic fixing, all of which I am unfamiliar with. And there would probably need to be an easy way for the user to bypass the cache as needed.

There are probably also other things that could be cached between runs on a project as well.

Exactly how the cache is saved is up to you. I was thinking of a .phpcs-cache in the root of the project being sniffed that would contain the cache represented as a JSON object.

If this is something that you think could be done, I'd be happy to work up a PR if you'll give me a little guidance on how you'd like this implemented.

[gsherwood]

I am still undecided as to if PHPCS itself should be doing things like this or if a wrapper script should be used, like what is done for hooking into a VCS.

Of course, it's possible, and probably fairly easy to get going, but I would not introduce it into the current 2.x versions regardless. I'd start with the 3.0 version, which is still in heavy development here: https://github.com/squizlabs/PHP_CodeSniffer/tree/3.0

How it is done is 1 of 2 options: something in the core and some command line values, or a new script like the SVN pre-commit hook or phpcbf. Either way, it's going to be easier/cleaner to do in the 3.0 version because of the refactoring I've done there, but I'm not ready to accept any new features on that branch at the moment.

[JDGrimes]

Either way, it's going to be easier/cleaner to do in the 3.0 version because of the refactoring I've done there, but I'm not ready to accept any new features on that branch at the moment.

When you are ready, let me know, and if I can I'll try to make up a PR.

[gsherwood]

When you are ready, let me know, and if I can I'll try to make up a PR.

We can start talking about it now. The most important bit is deciding on an implementation before starting work.

Version 3 has a FileList class that just creates a list of files from a file system location. With some changes, different types of file lists could be provided to the core processing code to generate their file lists from various sources:

1. The current one, which grabs a list of files from the file system
2. An extension to the first, which only includes files that have been modified since they were last checked (cache file required)
3. An VCS-based (or even Git specific) extension to the first, which only includes files that have been modified locally (command needs to be run to get the list)

Or, 2 and 3 could be implemented using some sort of filter class, which is passed to the main FileList. I think I prefer this than extending a base class, but I haven't thought about it long enough to be 100% sure.

Interested to hear your thoughts.

[gsherwood]

One more thing to add. I think it would be good if people could specify their own filters (assuming we go that way) on the command line just like they can do for reports. So a command like --filter=/path/to/my/filter.php should be possible, as well as built-in filter types like --filter=modified and --filter=git (whatever the names are).

[JDGrimes]

I think the filters idea is a good one. I don't think it is a complete solution though, because there are probably other things which could be cached as well. For example, it might be beneficial to cache the parsed rulesets.

[JDGrimes]

One possible drawback to filters versus child classes, is that the FileList class would still be traversing the whole directory unnecessarily (e.g., when using the git filter). Or were you thinking it would be implemented in a way that would avoid that?

[aik099]

Maybe the RecursiveDirectoryIterator can be combined with filter provided RecursiveFilterIterator sub-class to avoid scanning sub-folder of a particular folder, when filter iterator decides that it's not worth scanning.

[gsherwood]

One possible drawback to filters versus child classes, is that the FileList class would still be traversing the whole directory unnecessarily (e.g., when using the git filter). Or were you thinking it would be implemented in a way that would avoid that?

I was thinking more about file modification times more than anything else, which really needs all the normal recursive directory iterator stuff, with an added check (once a file path is found) to check if it needs rechecking.

But finding files is very fast, so I don't see why the same process wont work for Git. You could use the output from a git command and parse the paths in the output to support specified files. directories and the local flag. But it might just be easier to let the recursive scan happen and then match the found file paths to a list of ones found by git.

You don't always want every modified or uncommitted file. Even though there is a cache of the ones that are modified, you may want to limit that by file type, extension, path etc. I think it would be much easier if FileList was in charge of finding candidate files based on that logic, and a filter kicked to limit the candidates based on the filter logic.

If could do this after the candidates have all been found, or during the recursive scan. After is pretty easier, but during might be more efficient. If, for example, a directory does not contain any uncommitted files, you could have the FileList skip that dir and continue with the rest. A filter checking the modification times (or hashes) of files couldn't do that, but that's not a big deal.

[gsherwood]

One thing I actually didn't address in my previous comments was the initial request 😄 I got carried away talking about other related feature requests that I've received.

The filters are good for limiting the things that need to be checked, but we do need to inject the error reports for files that are being skipped from checking (due to cache) but the user still requested to have a report generated for.

In this case, a filter might still be useful as it can block the checking of a file and return the report instead, but I think that would take a fair bit of refactoring in the code, and might make things worse.

What would be better is to just implement this functionality in the LocalFile class, which is loaded with a file path and asked to process (tokenize and check) itself by the Runner. Instead of always choosing to process, it could instead load itself with a set of cached error messages if the file has not changed on disk. The DummyFile class (used mainly for STDIN) would not include this check as it doesn't have a file system location.

So a filter can kick in to limit the files to process, and the file itself could include a hash or modification time check to determine if it needs to be processed again. Two features.

It feels like each checked file should have its own cache so that filters and command line arguments don't get in the way of each other, but that would create a lot of files. Instead, PHPCS might need a Cache handler somewhere where files can register caches with a particular key. Same result, but a single cache file instead of hundreds. The cache handler could be responsible for maintaining the overall state of the run (standard and options used) and so could keep multiple caches if needed.

[gsherwood]

I've put together a quick implementation of a hash-based cached based on my comment above. It's missing a lot of features, but I wanted to see if the implementation would work and what the results would be like. Obviously, a pretty big improvement in performance, for really minimal code.

This is the first run, with no cache file:

$ bin/phpcs --report=summary --standard=PHPCS src -lv
Registering sniffs in the PHP_CodeSniffer standard... DONE (55 sniffs registered)
Creating file list... DONE (5 files in queue)
Changing into directory .../PHP_CodeSniffer/src
Processing Config.php [PHP => 8287 tokens in 1156 lines]... DONE in 367ms (0 errors, 67 warnings)
Processing Fixer.php [PHP => 4533 tokens in 695 lines]... DONE in 185ms (1 errors, 27 warnings)
Processing Reporter.php [PHP => 2564 tokens in 386 lines]... DONE in 101ms (9 errors, 8 warnings)
Processing Ruleset.php [PHP => 7394 tokens in 997 lines]... DONE in 337ms (0 errors, 31 warnings)
Processing Runner.php [PHP => 3708 tokens in 525 lines]... DONE in 161ms (0 errors, 15 warnings)

PHP CODE SNIFFER REPORT SUMMARY
-------------------------------------------------------------------------------------
FILE                                                                 ERRORS  WARNINGS
-------------------------------------------------------------------------------------
.............................../PHP_CodeSniffer/src/Config.php       0       67
.............................../PHP_CodeSniffer/src/Fixer.php        1       27
.............................../PHP_CodeSniffer/src/Reporter.php     9       8
.............................../PHP_CodeSniffer/src/Ruleset.php      0       31
.............................../PHP_CodeSniffer/src/Runner.php       0       15
-------------------------------------------------------------------------------------
A TOTAL OF 10 ERRORS AND 148 WARNINGS WERE FOUND IN 5 FILES
-------------------------------------------------------------------------------------

Time: 1.25 secs; Memory: 20.25Mb

This is the second run, with the cache file in place:

$ bin/phpcs --report=summary --standard=PHPCS src -lv
Registering sniffs in the PHP_CodeSniffer standard... DONE (55 sniffs registered)
Creating file list... DONE (5 files in queue)
Changing into directory .../PHP_CodeSniffer/src
Processing Config.php [loaded from cache]... DONE in 0ms (0 errors, 67 warnings)
Processing Fixer.php [loaded from cache]... DONE in 0ms (1 errors, 27 warnings)
Processing Reporter.php [loaded from cache]... DONE in 0ms (9 errors, 8 warnings)
Processing Ruleset.php [loaded from cache]... DONE in 0ms (0 errors, 31 warnings)
Processing Runner.php [loaded from cache]... DONE in 0ms (0 errors, 15 warnings)

PHP CODE SNIFFER REPORT SUMMARY
-------------------------------------------------------------------------------------
FILE                                                                 ERRORS  WARNINGS
-------------------------------------------------------------------------------------
.............................../PHP_CodeSniffer/src/Config.php       0       67
.............................../PHP_CodeSniffer/src/Fixer.php        1       27
.............................../PHP_CodeSniffer/src/Reporter.php     9       8
.............................../PHP_CodeSniffer/src/Ruleset.php      0       31
.............................../PHP_CodeSniffer/src/Runner.php       0       15
-------------------------------------------------------------------------------------
A TOTAL OF 10 ERRORS AND 148 WARNINGS WERE FOUND IN 5 FILES
-------------------------------------------------------------------------------------

Time: 73ms; Memory: 8.5Mb

Running over the whole PHPCS dir is a difference between 22.13secs and 375ms, so this is good. But the cache file itself (which I'm json pretty printing at the moment) is 7.2M, which isn't that great. If I turn off pretty printing, it comes down to 1.5M, but is now unreadable, so a different format could even be chosen if it ends up being faster. Still, I like JSON.

I'll commit what I have after a bit more cleanup and we can take things from there.

[gsherwood]

I've pushed some commits for this. The main one is this e5cc0ab

But forgot unit testing, so committed these 2 fixes as well: f558de5 1471a78

If you use the --cache command line argument, PHPCS will write a .phpcs.xxxxxxxxxxxx.cache file into the current directory (where xxxxxxxxxxxx is a hash representing the config of the run) and subsequent runs will use the data within if the file has not changed.

If you run over part of the code base in one run, and another part during another run, but use the same config, the same cache file will be used (it will just get bigger). Similarly, if you run over your entire code base and cache everything, you can then do another run limiting the files to check and the cache file will still be used.

I haven't committed anything to do with filtering of the file list.

This is still pretty dirty, so I'd appreciate any testing that anyone can do, and ideas for how to make things better.

One of the decision I had to make was where to put the cache files. I decided on the current working directory instead of the temp dir for 2 reasons (1 good reason, 1 stupid reason):

1. If you are running PHPCS over multiple code bases with the same config, it would end up with a massive cache file containing every file ever checked with that config. The file could get massive, and it will be loading file info that you don't need, so you'll blow the memory limit.
2. If you really want to, you could always commit the cache file to a repo periodically so that all devs can benefit from the current cache.

Number 1 could be worked around by including the current dir in the file hash, but you lose number 2 by doing that. I'm still not sure what the best place for these files is.

[aik099]

If you are running PHPCS over multiple code bases with the same config, it would end up with a massive cache file containing every file ever checked with that config. The file could get massive, and it will be loading file info that you don't need, so you'll blow the memory limit.

Agreed, but current working directory might not be be better place, because:

• when I'm running php using absolute path from / folder, then / is where cache will be saved
• when I'm running phpcs from build server (e.g. Jenkins), then it sets current directory to project directory, which might not be writable at all
• way, how phpcs is executed by PhpStorm (on per file basis) probably uses temp folder as current directory, but I haven't checked

If you really want to, you could always commit the cache file to a repo periodically so that all devs can benefit from the current cache.

Yes, they can, but since code base changes all the time the developers need to run phpcs all the time and commit cache file with every commit. And then we can have huge merge conflicts if 2 developers changed some code, which resulted in non-pretty json change (one large line), which diff would probably fail to merge correctly.

Making cache directory configurable (e.g. --cache <cache dir>, directory name is optional) would solve problem, when phpcs is invoked differently to ensure that it's looking into same cache file no matter of invocation way. The PhpStorm however have no idea about this new option and we'll need to wait for PhpStorm 10 release (current release is 8, but 9 is in EAP state) to adopt that option.

[aik099]

Also the xxxxxxxxxxxx part of cache filename needs to include used sniff names (not just standard name specified in .phpcs file or via --standard command line option).

Any of these changes should invalidate cache:

• standard name was changed (e.g. from PSR2 to Squiz)
• sniff was added/removed from used standard ruleset.xml
• sniff was added/removed in other standard, that is fully (or partially, e.g. sub-dir only) included in used standard
• sniff PHP code changed (specified in used standard or included from another standard)

What I believe would be correct cache key detection is:

• get to the point in code, where we have:
  • all sniffs loaded
  • all exclusion rules (per ruleset, per file, per sniff)
  • all CLI options (part of them can be read from system-wide PHP_CodeSniffer.conf or .phpcs file)
• used that to form cache key

[VasekPurchart]

+1 for configurable cache dir