3.1 Basic Input File Processing

In order to read in an input file or include file, the scanner calls a submodule called the mapper that reads a file in and creates a contiguous copy of it in memory. The scanner then performs three checks on the file, the first (file termination) of which is performed before scanning commences, and the other two of which take place during scanning before each line is scanned.

File Termination: The first check the scanner makes is whether the file is terminated properly. A file is considered to be properly terminated if it either contains no lines, or if the last line in the file is terminated by an end-of-line marker. If the scanner detects that an input file is not properly terminated, it adds an end-of-line marker itself (to the copy in memory only).

Unprintable Characters: The second check the scanner makes is for unprintable characters (ASCII 0--31 and 127--255 (except for EOL(10))) which it flags as errors and replaces by question marks.

Line Lengths: The third check the scanner makes is input line length. When FunnelWeb starts up, a default maximum input line length of 80 is set. This can be changed dynamically during scanning using a @p maximum_input_line_length pragma. If the number of characters on a line (not including the end of line marker) exceeds this limit, FunnelWeb generates an error.