Token identifiers

Lexer and parser must agree on common set of terminal symbols.

Usually when parser is generated each terminal symbol defined in grammar is automatically assigned an integer code. If necessary user can specify explicit integer codes for some or all terminals.

User can specify explicit codes in two ways:

• by using an integer constant in terminal definition
• by using a character code (single-quoted character)

Second alternative is additionally controlled by Parser.Tokens.CharIsCode option (true by default).

Range of input codes has influence on size of parser action tables. It is best if codes come from contiguous range starting from 1 (like code assigned automatically by parser).

If custom user-defined symbol codes are used it is possible to define a second set of contiguous codes to be used internally by parser. Mapping external codes to internal codes introduces some overhead but results in smaller parser tables.

Mapping terminal symbol codes to internal representation is controlled with option:

Parser.Tokens.Mapping

With available values:

• Auto - decision to use mapping is done automatically based on sparseness of symbol range
• Yes - forces mapping
• No - disables mapping. If resulting parser table size would be large, Alpag may ignore this setting and use mapping anyway

Codes of terminal symbols are available inside parser class body as constants. Their identifiers are same as identifiers used in input grammar. Option Parser.Tokens.IdentifierPrefix can be used to prepend additional prefix to generated constants.

Terminal symbol codes are public and can be accessed from outside of parser.

Options in Parser.Code.Exports group enable placing parser public declarations in dedicated class (and file) for interfacing with external code.