/usr/portage

The state of meta programming in PHP 11

Quoting Wikipedia

Metaprogramming is the writing of computer programs that write or manipulate other programs (or themselves) as their data, or that do part of the work at compile time that would otherwise be done at runtime

Metaprogramming is quite an interesting sub-discipline and knowing about certain techniques and tools allows you to cut corners quite dramatically for certain tasks. As always, don’t overdo but to find out when you are overdoing, first start doing, get excited, overdo, find out the right dose. Let’s have a look at what kind of tools you have available in PHP to solve typical meta programming problems.

What kind of questions can meta programmatic APIs answer?

I would group metaprogramming into three sub areas: type introspection, lower level syntax inspection and metadata management. Typical type introspection questions are:

On a lower level you typically interact with a certain kind of syntax tree to answer questions like:

A third category is adding metadata to the declared types: Java, C# and a few others have first-class Annotation support for this kind of things but PHP only has user space solutions so far. A few things you need metadata for:

The toolkit

Reflection APIs

PHP core delivers 2.5 key APIs for meta programming. The first one is ext/reflection. You can create reflection classes form a lot of things, functions, classes, extensions and use them to make programming assumptions about the APIs you are introspecting.

A simple example to find out the number of required parameters for each method in the class DirectoryIterator:

<?php
$class = new ReflectionClass('DirectoryIterator');
foreach ($class->getMethods() as $method) {
    $numberOfRequiredParameters = $method->getNumberOfRequiredParameters();
}

Refection is all nice and shiny, except when you don’t want to include everything you want to inspect. This is of interest if you inspect various source trees at once that declare duplicate symbols. To do so, there is PHP-Token-Reflection by Ondřej Nešpor. It’s a pretty nifty replacement for ext/reflection completely built in user land and on top of ext/tokenizer that even copes with invalid declarations. Additionally it fixes some oddities of the internal reflection API but tries to keep it as close as possible. I’ve played around with it a bit and I quite like it.

<?php
$broker = new TokenReflection\Broker(new TokenReflection\Backend\Memory());
$broker->processDirectory("path/to/src");
$class = $broker->getClass('MyClass');
foreach ($class->getMethods() as $method) {
   ...
}

Tokenizer

Another core API, this time much more low level, is ext/tokenizer. If enabled at compile time it allows you to parse PHP source code into a list of tokens. Because the API is so low level it is quite hard to use without a proper abstraction layer on top of it. Most of the successful projects built upon ext/tokenizer have built one. One of them is phpcs by Greg Sherwood that built an Token Stream abstraction on top of ext/tokenizer that allows much more convenient navigation in the token stream. Another one shipping its own token stream abstraction is pdepend by Manuel Pichler. Another noteworthy, standalone abstraction is php-manipulator.
For an example on how the raw API can be used, I once wrote this little script to apply a few transformations to source trees to ease converting source trees to PHP 5.4.

PHP Parser: a fully fledged AST parser for PHP

Between a high level API like Reflection and a low level API like ext/tokenizer there surely is a gap: what if I want to work on an AST data structure. There is this beautiful project PHP-Parser by Nikita Popov. This is quite interesting for more complex transformations like user space AOP, all kinds of static code analysis and so on. If ext/tokenizer feels way underpowered, have a look at this project.

Aspect oriented programming

While we are talking about AOP: a relative newcomer is PECL AOP that provides a quite simple API for aspect orientated programming in PHP. For Zend Framework 2 there is also an AOP module available. Let’s stick to AOP for a moment: for Symfony 2 there is JMSAopBundle by Johannes Schmitt. It provides basic AOP functionality for Symfony 2. JMSSecurityExtraBundle and JMSDiExtraBundle use it to provide annotation support for Symfony security bundle and the Symfony dependency injection component.

Metadata management

Traditionally, every docblock documentation parser rolled it’s own annotation system. This changed a little with the rise of Symfony and Doctrine 2. Doctrine 2 allows you to use annotations for persistence definition and Symfony allows you to use annotations for a lot of things (routes, security, etc.). While Doctrine still ships it’s own metadata handling component in doctrine-common, there is another library by Johannes Schmitt, Metadata that aims to consolidate metadata handling for PHP. The API of the Metadata library as well as the one of doctrine-common is quite simple: you have some sort of annotation reader that maps metadata information to classes. Think about this annotation:

<?php
use My\Annotation\Some;
/**
  * @Some(foo="bar")
  */
class MyClass
{}

This kind of annotation will map to an instance of My\Annotation\Some with the property $foo set to “bar”.

Radioactive, specialized or obscure

Ever dreamed of renaming functions, redeclaring classes and so on? Let us not discuss whether this is a good idea or not, but if you would like, look no further: there is runkit for that (I think this is the most current fork).

If you want to access the opcodes of a your code, Stefan Esser wrote bytekit for you (bytekit.org is no longer available, I only found Tyrael/bytekit and Mayflower/Bytekit). To make working with bytekit data a little more convenient, Sebastian Bergmann wrote bytekit-cli.

To register callbacks at every function call, there is funcall by Chen Ze and intercept by Gabriel Ricard.

One should not forget about xdebug by Derick Rethans that provides a quite specialized sub-sub-sub-discipline: code coverage analyis.

The future

PHP core itself could really use native support for annotations. This would fix little differences in how annotations are used nowadays by major projects. Another very interesting development is quite definitely PHP AOP. I would consider that a candidate for core inclusion at some point.

The userland libraries could see some consolidation and now that we have composer dependency management isn’t so much of a problem. Especially in the Symfony 2 world, reusing the same metadata framework would make totally sense. A first step is that Zend Framework 2 uses doctrine-common for annotations support.

Filed on 02-12-2012, 17:05 under , , , , , , , & eleven comments & no trackbacks

Trackbacks

Trackback specific URI for this entry

No Trackbacks

Comments

  1. Thomas Maroschik returns:
    published on December 2nd 2012, 08:15:30 pm *

    Great overview. But I’m missing TYPO3 Flow (http://flow.typo3.org) in this article as it implements – in my opinion – the most advanced PHP meta programming support currently available.

    Reply

  2. Lars Strojny answers:
    published on December 2nd 2012, 08:48:46 pm *

    Thanks for the hint, Thomas. Are you referring to Flow3’s AOP capabilities or anything else? As I don’t know that much of Flow3, are Flow3 components usable separately?

    Reply

  3. Thomas Maroschik answers:
    published on December 4th 2012, 12:47:36 pm *

    Currently Flow (has been renamed from FLOW3 to TYPO3 Flow) is still a full stack framework. But the team is looking forward to divide it into components afaik.
    Beside AOP, Flow has a great reflection service, where you can ask questions like "what are the implemenations of this interface" and take care of dependency injection in a much developer friendlier way than most other implementations. I think its worth a look here: http://flow.typo3.org/about/features.html

    Reply

  4. Ivan Kurnosov states:
    published on December 2nd 2012, 10:40:04 pm *

    PHP core itself could really use native support for annotations.

    But please, only as an additional language construction, not as an advanced-doc-block parser.

    Reply

  5. Lars Strojny replys:
    published on December 2nd 2012, 10:55:59 pm *

    The only concept I’ve read so far is "this RFC":https://wiki.php.net/rfc/annotations. If you have any thought on it, feel free to get in touch with the authors.

    Reply

  6. Ivan Kurnosov states:
    published on December 2nd 2012, 11:02:31 pm *

    Yep, looks acceptable.

    Like the attributes in C#: http://msdn.microsoft.com/en-us/library/z0w1kczw.aspx but with triangle braces.

    Reply

  7. Chris Alfano answers:
    published on December 3rd 2012, 12:27:02 am *

    Hey Lars, I’ve been following your blog for awhile and am working on an open source project to provide intelligently managed space for PHP applications. It has a couple years in production but not much yet in the way of documentation and marketing materials.

    If you are interested in taking a look and lending your thoughts please drop me an email so we can setup a quick skype/ghangout

    Reply

  8. Lars Strojny supposes:
    published on December 3rd 2012, 12:28:33 am *

    Sure, just drop me a mail: lars@strojny.net.

    Reply

  9. michael kimsal states:
    published on December 3rd 2012, 02:35:15 pm *

    Would love to be able to do something like

    class foo { }

    foo->myCalc = function ($a) { // do something }

    $t = new foo();
    $t->myCalc(5);

    Will we ever get to something as straightforward as that in PHP?

    Reply

  10. Lars Strojny opines:
    published on December 3rd 2012, 03:18:07 pm *

    Michael, this was discussed at length when we introduced closures. Reason not to do it is that resolution rules esp. in combination with __call are not straight-forward. I guess this is a topic we need to reiterate once extended property support is implemented.
    Meanwhile, you can do https://gist.github.com/4195275.

    Reply

Add a Comment & let me know what you think