PHP 8 and trailing newline in comment token

[sebastianbergmann]

• Don't include trailing newline in comment token php/php-src#5182
• http://git.php.net/?p=php-src.git;a=commitdiff;h=47cf18ba4ef9fb9df6c3affff8f2a3f5a3f930a8

[sebastianbergmann]

This could be the root cause for https://github.com/sebastianbergmann/php-code-coverage/runs/810286407.

[sebastianbergmann]

@theseer will (probably) have to adapt theseer/tokenizer for this as well.

[theseer]

Checking...

[sebastianbergmann]

The changes made in 19cf736 re-attach the newline to the comment token and effectively reverse the change in PHP 8 so that code that uses this component continues to work without change in PHP 8.

test.php

<?php declare(strict_types=1);
require __DIR__ . '/vendor/autoload.php';

$i = 1;

foreach ((new PHP_Token_Stream(file_get_contents(__DIR__ . '/tests/_fixture/issue_95.php'))) as $token) {
    print $i++ . "\t" . get_class($token) . "\t'" . $token . "'\n";
}

PHP 7.4

1	PHP_Token_OPEN_TAG	'<?php '
2	PHP_Token_DECLARE	'declare'
3	PHP_Token_OPEN_BRACKET	'('
4	PHP_Token_STRING	'strict_types'
5	PHP_Token_EQUAL	'='
6	PHP_Token_LNUMBER	'1'
7	PHP_Token_CLOSE_BRACKET	')'
8	PHP_Token_SEMICOLON	';'
9	PHP_Token_WHITESPACE	'
'
10	PHP_Token_IF	'if'
11	PHP_Token_WHITESPACE	' '
12	PHP_Token_OPEN_BRACKET	'('
13	PHP_Token_STRING	'false'
14	PHP_Token_CLOSE_BRACKET	')'
15	PHP_Token_WHITESPACE	' '
16	PHP_Token_OPEN_CURLY	'{'
17	PHP_Token_WHITESPACE	'
    '
18	PHP_Token_COMMENT	'// @codeCoverageIgnoreStart
'
19	PHP_Token_WHITESPACE	'    '
20	PHP_Token_COMMENT	'# ...
'
21	PHP_Token_WHITESPACE	'    '
22	PHP_Token_COMMENT	'// @codeCoverageIgnoreEnd
'
23	PHP_Token_CLOSE_CURLY	'}'
24	PHP_Token_WHITESPACE	'
'

PHP 8

1	PHP_Token_OPEN_TAG	'<?php '
2	PHP_Token_DECLARE	'declare'
3	PHP_Token_OPEN_BRACKET	'('
4	PHP_Token_STRING	'strict_types'
5	PHP_Token_EQUAL	'='
6	PHP_Token_LNUMBER	'1'
7	PHP_Token_CLOSE_BRACKET	')'
8	PHP_Token_SEMICOLON	';'
9	PHP_Token_WHITESPACE	'
'
10	PHP_Token_IF	'if'
11	PHP_Token_WHITESPACE	' '
12	PHP_Token_OPEN_BRACKET	'('
13	PHP_Token_STRING	'false'
14	PHP_Token_CLOSE_BRACKET	')'
15	PHP_Token_WHITESPACE	' '
16	PHP_Token_OPEN_CURLY	'{'
17	PHP_Token_WHITESPACE	'
    '
18	PHP_Token_COMMENT	'// @codeCoverageIgnoreStart
'
19	PHP_Token_WHITESPACE	'    '
20	PHP_Token_COMMENT	'# ...
'
21	PHP_Token_WHITESPACE	'    '
22	PHP_Token_COMMENT	'// @codeCoverageIgnoreEnd
'
23	PHP_Token_CLOSE_CURLY	'}'
24	PHP_Token_WHITESPACE	'
'

Differences between PHP 7.4 and PHP 8

$ php test.php > php-74.txt   
$ php-80 test.php > php-8.txt 
$ diff -u php-74.txt php-8.txt

[sebastianbergmann]

Work on this is not complete.