4.1 Introduction

By the time the parser starts, the scanner has completely terminated. At this point, it is not possible for any more files to be included, and special characters are no longer present to confuse things. All that remains is a list of text tokens, special tokens, and typesetter directive tokens. Text tokens consist entirely of sequences of printable characters and end of line markers. Special tokens represent the special sequences that the scanner found in the input file. Typesetter directive tokens represent the freestanding typesetter directives that the scanner encountered. The parser consumes the token list and builds a macro table that is later used to generate product files. It also constructs a document list that is used to generate the documentation file.

The syntax rules appearing in the following sections refer to the token list.