PHP - Tokenizer Functions

Tokenizer functions can provide an interface to the PHP tokenizer embedded in Zend Engine. By using these functions, we may write our own PHP source analyzing or modification tools without having to deal with a language specification at the lexical level.

Example

<?php
   if (!defined('T_ML_COMMENT')) {
      define('T_ML_COMMENT', T_COMMENT);
   } else {
      define('T_DOC_COMMENT', T_ML_COMMENT);
   }

   $source = file_get_contents('example.php');
   $tokens = token_get_all($source);

   foreach($tokens as $token) {
      if(is_string($token)) {
         //  simple 1-character token
         echo $token;
      } else {
         //  token array
         list($id, $text) = $token;
         switch ($id) { 
            case T_COMMENT: 
            case T_ML_COMMENT: // we've defined this
            case T_DOC_COMMENT: // and this
            
			//  no action on comments
            break;

            default:
               //  anything else -> output "as is"
               echo $text;
               break;
         }
      }
   }
?>