Token Value
Each time lexer finds a match, a value (text) associated with this match is available in output buffer. Lexer output format can be configured to be bytes, characters or both (switchable in runtime). That's why distinct methods exist for getting matched value in either format.
Below is a summary of methods for getting matched value (text):
Returns length of matched value in bytes or two-byte characters (depending on current format selected)
char[] TokenValueCharGetCopy()
Return a copy of matched value in newly allocated array.
The convenience method is available only in Chars mode.
void TokenValueCharCopyTo( char[] dstArray, int dstOffset )
Copy current matched value into user-provided array at given offset. Array must have enough space for the value.
void TokenValueCharCopyFragmentTo(int offset, int count, char[] dstArray, int dstOffset )
Copy a portion of matched value into user-provided array at given offset. Array must have enough space for the value. Offset and count refer to positions within current match in 0..TokenValueLength range.
If lexer is configured to use single large array for entire buffer, it is also possible to access such buffer directly using:
char[] TokenValueCharBufDirect()
These methods provide direct access to buffers. Returned value is the buffer itself, not its copy. User is not allowed to modify contents of the buffers.
Value of current match does not in general start at the beginning of the buffer. Offset of current match is:
Value of current match is in the range:
TokenValueBufOffset .. TokenValueBufOffset+ TokenValueLength-1.