PHP - token_get_all() Function

token_get_all() function can split a given source into PHP tokens.

Syntax

array token_get_all( string $source [, int $flags = 0 ] )

token_get_all() function can parse a given source string into PHP language tokens by using the Zend engine's lexical scanner. For a list of parser tokens, we can use the token_name() function to translate a token value into its string representation.

token_get_all() function can return an array of token identifiers. Each individual token identifier is either a single character (i.e.:;, ., >, ! etc...), or a three-element array containing token index in element 0, string content of an original token in element 1, and line number in element 2.

Example-1

<?php
   $tokens = token_get_all("<?php echo; ?>");

   foreach($tokens as $token) {
      if(is_array($token)) {
         echo "Line {$token[2]}: ", token_name($token[0]), " ('{$token[1]}')", PHP_EOL;
      }
   }
?>

Example-2

<?php
   $tokens = token_get_all("/* comment */");

   foreach($tokens as $token) {
      if(is_array($token)) {
         echo "Line {$token[2]}: ", token_name($token[0]), " ('{$token[1]}')", PHP_EOL;
      }
   }
?>

Example-3

<?php
   $source = <<<"code"
   <?php
   class A {
      const PUBLIC = 1;
   }
   code;

   $tokens = token_get_all($source, TOKEN_PARSE);

   foreach($tokens as $token) {
      if(is_array($token)) {
         echo token_name($token[0]) , PHP_EOL;
      }
   }
?>