Reading input
Lexer can read input stream composed of bytes or characters. Input format is controlled by option Lexer.In.Format. Possible option values are: Bytes, Chars, BytesChars. When BytesChars is set lexer can be programmatically switched between reading bytes or characters by invoking:
Switching input format is not allowed while lexer is already inside a stream.
Option Lexer.In.FormatDefault can be used to specify default format assumed when lexer is created or reset.
Lexer has separate methods for reading input in either format:
int ReadInputChars ( char[] buffer, int offset, int count )
To get a working lexer user must provide implementation for above methods.
Methods are invoked when lexer needs more data. User should read up to count elements (bytes or characters) placing them in provided buffer at given offset. Less than count elements can be provided.
Methods return number of bytes/chars read. Return value of less than zero denotes an error, which should be one of predefined error codes or custom user code defined below USER_FIRST.
If method returns zero, interpretation depends on Lexer.In.ZeroCountAction option. Possible values are:
- EOF – when method returns zero, lexer assumes that end of file has been reached, stops operation and returns EOF code to the user. This is the default behavior, compatible with typical stream implementations. User can also return EOF (-1) to indicate end of input stream.
- RepeatRead – when method returns zero, lexer attempts another read.
Method should return EOF (-1) to indicate end of file. This behavior is compatible with some non-blocking streams.
The RepeatRead flag should be used only to attempt another read from the same stream. It is not allowed to switch current input buffer from within ReadInputBytes or ReadInputChars methods.
Note that there is one more mechanism for controlling lexer behavior after EOF was reached: the ContinueOnEOF() method. This method allows switching input buffer after EOF was reached. Implementation for the method can be provided in continue_on_eof code section and is controlled by Lexer.Code.ContinueOnEOFImpl option.
Implementation of ReadInputBytes/ReadInputChars methods is controlled by Lexer.Code.ReadInputImpl option.
When Lexer.Code.ReadInputImpl is set to UserCode user provides code for body of above methods in read_input_bytes or read_input_chars.code blocks, like this:
return someIputStream.Read( buffer, offset, count );
}
When Lexer.Code.ReadInputImpl is set to Virtual above methods are declared virtual with empty body. User must subclass generated Lexer and override implementation of these methods. This approach introduces minor performance overhead, but does not require placement of custom code in Alpag input file.