It's a big state machine starting with %INITIAL% States: %INITIAL%, %ST_IN_SCRIPTING%, %ST_DOUBLE_QUOTES%, %ST_NOWDOC%… Tokens can change the state: " No *meaning* is given to these tokens PHP comes with a *tokenizer* extension