PerCederberg.Grammatica.Runtime
Class Token

System.Object
   |
   +--Node
         |
         +--Token

public class Token

extends Node

A token node. This class represents a token (i.e. a set of adjacent characters) in a parse tree. The tokens are created by a tokenizer, that groups characters together into tokens according to a set of token patterns.

EndColumn

public override int EndColumn;

The column number property of the last character in this node (read-only). If the node has child elements, this value will be fetched from the last child.

Since:

1.5

EndLine

public override int EndLine;

The line number property of the last character in this node (read-only). If the node has child elements, this value will be fetched from the last child.

Since:

1.5

Id

public override int Id;

The node type id property (read-only). This value is set as a unique identifier for each type of node, in order to simplify later identification.

Since:

1.5

Image

public string Image;

The token image property (read-only). The token image consists of the input characters matched to form this token.

Since:

1.5

Name

public override string Name;

The node name property (read-only).

Since:

1.5

Next

public Token Next;

The next token property. If the token list feature is used in the tokenizer, all tokens found will be chained together in a double-linked list. The next token may be a token that was ignored during the parsing, due to it's ignore flag being set. If there is no next token or if the token list feature wasn't used in the tokenizer (the default), the next token will always be null.

Since:

1.5

See Also:

Previous, Tokenizer.UseTokenList

Pattern

 internal TokenPattern Pattern;

The token pattern property (read-only).

Previous

public Token Previous;

The previous token property. If the token list feature is used in the tokenizer, all tokens found will be chained together in a double-linked list. The previous token may be a token that was ignored during the parsing, due to it's ignore flag being set. If there is no previous token or if the token list feature wasn't used in the tokenizer (the default), the previous token will always be null.

Since:

1.5

See Also:

Next, Tokenizer.UseTokenList

StartColumn

public override int StartColumn;

The column number property of the first character in this node (read-only). If the node has child elements, this value will be fetched from the first child.

Since:

1.5

StartLine

public override int StartLine;

The line number property of the first character in this node (read-only). If the node has child elements, this value will be fetched from the first child.

Since:

1.5

Token

public Token( TokenPattern pattern, string image, int line, int col );

Creates a new token.

Parameters:

pattern - the token pattern

image - the token image (i.e. characters)

line - the line number of the first character

col - the column number of the first character

GetImage

public string GetImage();

Deprecated. Use the Image property instead.

Returns the token image. The token image consists of the input characters matched to form this token.

Returns:

the token image

See Also:

Image

GetNextToken

public Token GetNextToken();

Deprecated. Use the Next property instead.

Returns the next token. The next token may be a token that has been ignored in the parsing. Note that if the token list feature hasn't been used in the tokenizer, this method will always return null. By default the token list feature is not used.

Returns:

the next token, or null if no such token is available

Since:

1.4

See Also:

Next, GetPreviousToken, Tokenizer.UseTokenList

GetPreviousToken

public Token GetPreviousToken();

Deprecated. Use the Previous property instead.

Returns the previous token. The previous token may be a token that has been ignored in the parsing. Note that if the token list feature hasn't been used in the tokenizer, this method will always return null. By default the token list feature is not used.

Returns:

the previous token, or null if no such token is available

Since:

1.4

See Also:

Previous, GetNextToken, Tokenizer.UseTokenList

ToShortString

public string ToShortString();

Returns a short string representation of this token. The string will only contain the token image and possibly the token pattern name.

Returns:

a short string representation of this token

ToString

public override string ToString();

Returns a string representation of this token.

Returns:

a string representation of this token