Request: dumper that create php to create AST

[flip111]

Input php code

<?php

class Match
{
}

Now when i look at the AST and ignore the wrapping array (just to clear additional clutter in the following code snippets), i get these outputs:

var_export($ast)

PhpParser\Node\Stmt\Class_::__set_state(array(
   'flags' => 0,
   'extends' => NULL,
   'implements' => 
  array (
  ),
   'name' => 
  PhpParser\Node\Identifier::__set_state(array(
     'name' => 'MyClass',
     'attributes' => 
    array (
      'startLine' => 3,
      'endLine' => 3,
    ),
  )),
   'stmts' => 
  array (
  ),
   'attributes' => 
  array (
    'startLine' => 3,
    'endLine' => 5,
  ),
)

dumper

Stmt_Class(
    flags: 0
    name: Identifier(
        name: Match
    )
    extends: null
    implements: array(
    )
    stmts: array(
    )
)

What i like to have is a dumper that prints PHP code that can construct the AST

new PhpParser\Node\Stmt\Class_('MyClass');

For this the dumper should have some knowledge about the node constructor argument defaults, so that no unneeded arguments are given. For example if the same class had some statements in it, it would need to be:

new PhpParser\Node\Stmt\Class_('MyClass', ['stmts' => [
   /* actual statements here*/
]]);

This would be a very handy tool to create PHP from AST nodes with the help of a "static written" php file used for prototyping dynamic php code creation. While the export of var_export is probably runnable php code .. it's very hard to create code based on this because it's very verbose and __set_state is probably not the best thing to use here either.

[nikic]

Probably something like this can be made based on reflection. Constructor argument names should match subnodes names and it should be possible to generate code based on that.

[flip111]

@nikic maybe reflection can help for the subNodes. However the Class_ has as first argument the $name which is not part of the subNodes. Break_ has as first argument $num (number of loops to break). I see no other solution than to hand craft these things for each class in PHP-Parser. If there is another solution i'm interested to hear about it. Suppose now that each class needs it's own customized code for this type of dumper. Where should the dumper code live and how to keep it in sync with the node constructor arguments? Or is the structure of the PHP-Parser nodes already stable?

[nikic]

@flip111 Class_::$name is a subnode. Break_::$num is a subnode. With reflection, the property Class_::$name can be associated with the $name argument of the constructor. For Class_ in particular there is also the $subNodes argument, into which everything should go for which there is no explicit constructor argument. I think with these two things combines (checking for argument with the same name as property, and handling $subNodes specially) it should be possible to cover all nodes.

[flip111]

@nikic when i look here name is not part of the subNodes of class https://github.com/nikic/PHP-Parser/blob/master/lib/PhpParser/Node/Stmt/Class_.php#L31-L34 I don't understand your comment that name is also a subnode

[nikic]

@flip111 By subnodes I mean anything returned here: https://github.com/nikic/PHP-Parser/blob/master/lib/PhpParser/Node/Stmt/Class_.php#L47 All subnodes are public properties of the class.

[flip111]

@nikic ok i can try to make this. Is PR welcome? Where should this code live?

[flip111]

Found a class that is irregular.. the constructor can not be created with the same name as the subNodes.

https://github.com/nikic/PHP-Parser/blob/master/lib/PhpParser/Node/Name.php#L20-L29

There needs to be a translation from parts to name first.

[flip111]

I have this now .. it works for some code, but not fully tested. Maybe i can finish this and send PR ?

<?php

use PhpParser\Node as N;
use PhpParser\Node\Expr as E;
use PhpParser\Node\Scalar as SV;

// assumes values have already been transformed into AST nodes
function wrapArray($array) {
  $wrapped = [];

  $i = 0;
  $nonAssoc = true;
  foreach ($array as $key => $value) {
    if (is_array($value)) {
      $value = wrapArray($value);
    }
    if (is_string($key)) {
      $wrapped[] = new E\ArrayItem($value, new SV\String_($key));
    } elseif (is_int($key)) {
      if ($key !== $i) {
        $nonAssoc = false;
      }

      // don't set an explicit key when we can use php auto indexing
      if ($nonAssoc) {
        $wrapped[] = new E\ArrayItem($value);
      } else {
        $wrapped[] = new E\ArrayItem($value, new SV\LNumber($key));
      }
    }

    $i++;
  }

  return new E\Array_($wrapped);
};

function astToAstBuild($node) {
  if (is_array($node)) {
    array_walk($node, function(&$value, $key) { $value = astToAstBuild($value); });
    return $node;
  } elseif (is_object($node)) {
    $args = array_map(function($refl_par) {
      return [ 'name' => $refl_par->name
                // empty array means "no value"
                // filled array means "a value"
             , 'default' => $refl_par->isOptional() ? [$refl_par->getDefaultValue()] : []
             ];
    }, (new \ReflectionClass($node))->getConstructor()->getParameters());

    $nonConstructorParameterSubnodes = array_diff($node->getSubNodeNames(), array_column($args, 'name'));

    foreach ($args as $key => ['name' => $name, 'default' => $default]) {
      if ($name === 'subNodes') {
        $subNodes = [];

        // only process array items that were listed as being a subNode
        // but not an explicit constructor parameter
        foreach ($nonConstructorParameterSubnodes as $k) {
          $v = $node->{$k};

          if ($k === 'stmts') {
            if ($v !== []) {
              $subNodes[$k] = $v;
            }
          } else {
            // can put some custom rules per class here
            if (get_class($node) === 'PhpParser\Node\Stmt\Class_') {
              if (  ($k === 'flags' && $v !== 0)
                 || ($k === 'extends' && $v !== null)
                 || ($k === 'implements' && $v !== [])
                 ) {
                $subNodes[$k] = $v;
              }
            // when not specifying rules per class it can lead to more noise in the output
            } else {
              $subNodes[$k] = $v;
            }
          }
        }

        $args[$key]['value'] = $subNodes;
      } elseif ($name === 'attributes') {
        $args[$key]['value'] = []; // ignoring attributes for the moment
      } elseif (($node instanceof \PhpParser\Node\Name) && $name === 'name') {
        $args[$key]['value'] = $node->toString();
      } else {
        $args[$key]['value'] = $node->{$name};
      }

      unset($args[$key]['name']); // don't need this information anymore
    }

    // find out which constructor parameters can use the default value
    $args = array_reverse($args, true);
    $can_use_default = true;
    foreach ($args as $k => $v) {
      if (count($v['default']) === 0 || $v['value'] !== $v['default'][0]) {
        $can_use_default = false;
      }

      $args[$k]['can_use_default'] = $can_use_default;
      unset($args[$k]['default']); // don't need this information anymore
    }
    $args = array_reverse($args, true);

    $argsBuild = [];
    foreach ($args as $arg) {
      if ($arg['can_use_default']) {
        break;
      }

      $ret = astToAstBuild($arg['value']);

      if (is_array($ret)) {
        $ret = wrapArray($ret);
      }

      $argsBuild[] = new N\Arg($ret);
    }

    return new E\New_(new N\Name(get_class($node)), $argsBuild);
  } elseif (is_string($node)) {
    return new SV\String_($node);
  } elseif (is_int($node)) {
    return new SV\LNumber($node);
  } elseif ($node === null) {
    return new E\ConstFetch(new Node\Name('null'));
  } elseif (is_float($node)) {
    return new SV\DNumber($node);
  } elseif (is_bool($node)) {
    return new E\ConstFetch(new Node\Name($node ? 'true' : 'false'));
  } else {
    printf("Not handled:\n");
    var_dump($node);
    die();
  }
}

[matthiasnoback]

Nice idea!
Although it might be nice to have the functionality inside this library, it's possible to implement in a separate library as well. So I've started a dedicated project for this: https://github.com/matthiasnoback/php-parser-instantiation-printer

[matthiasnoback]

FYI: I've just wrapped up the initial version: https://github.com/matthiasnoback/php-parser-instantiation-printer I've created this in a test-driven way and have added several examples to show that certain edge cases are handled (thanks @flip111 for the inspiration, great suggestions regarding default constructor arguments, constructor arguments that are also subnodes, and subnodes that have a default value.

[flip111]

Thanks for the library looks pretty nice. I would add the following things to it:

• (More) motivation why it is useful. For me personally this was that to create a lot of code to generate PHP code it would have been easier to first write a (static) php file of how the result should look like. Then i would take this code and turn it into the construction of AST nodes, which i can then change manually to put in all (dynamic) information i wanted.
• Ability to use aliases for namespaces so that the code is not cluttered with namespaces
• Use a quickcheck library https://github.com/giorgiosironi/eris or https://github.com/steos/php-quickcheck with following steps:
  • Create arbitrary instances (if not available somewhere already) for nikic/PHP-Parser nodes
  • Prettyprint the tree (from previous step) to PHP code using nikic/PHP-Parser prettyprinter. Save this as "input.php"
  • Read this tree with your own code, and output the file to construct the AST nodes
  • Run the file that constructs AST nodes
  • Then print those AST nodes again with nikic/PHP-Parser prettyprinter and save it as "output.php"
  • Compare input and output
  • get rid of static tests and test fixtures all together & stay updated automatically with changed to nikic/PHP-Parser and by implication features in new php versions.

[matthiasnoback]

Interesting suggestions, @flip111 ! I'll see what I can do.