net.percederberg.grammatica.parser

Class Token

java.lang.Object

net.percederberg.grammatica.parser.Node

net.percederberg.grammatica.parser.Token

public class Token
extends Node

A token node. This class represents a token (i.e. a set of adjacent characters) in a parse tree. The tokens are created by a tokenizer, that groups characters together into tokens according to a set of token patterns.

Constructor Summary

Constructors

Constructor and Description
Token(TokenPattern pattern, java.lang.String image, int line, int col) Creates a new token.

Method Summary

Modifier and Type	Method and Description
int	getEndColumn() The column number of the last character in the token image.
int	getEndLine() The line number of the last character in the token image.
int	getId() Returns the token (pattern) id.
java.lang.String	getImage() Returns the token image.
java.lang.String	getName() Returns the token node name.
Token	getNextToken() Returns the next token.
Token	getPreviousToken() Returns the previuos token.
int	getStartColumn() The column number of the first character in the token image.
int	getStartLine() The line number of the first character in the token image.
java.lang.String	toShortString() Returns a short string representation of this token.
java.lang.String	toString() Returns a detailed string representation of this token.

Methods inherited from class net.percederberg.grammatica.parser.Node

addValue, addValues, addValues, getAllValues, getChildAt, getChildCount, getDescendantCount, getParent, getValue, getValueCount, printTo, printTo, removeAllValues

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

Token

public Token(TokenPattern pattern,
             java.lang.String image,
             int line,
             int col)

Creates a new token.

Parameters:

pattern - the token pattern

image - the token image (i.e. characters)

line - the line number of the first character

col - the column number of the first character

Method Detail

getId

public int getId()

Returns the token (pattern) id. This value is set as a unique identifier when creating the token pattern to simplify later identification.

Specified by:

getId in class Node

Returns:

the token id

getName

public java.lang.String getName()

Returns the token node name.

Specified by:

getName in class Node

Returns:

the token node name

getImage

public java.lang.String getImage()

Returns the token image. The token image consists of the input characters matched to form this token.

Returns:

the token image

getStartLine

public int getStartLine()

The line number of the first character in the token image.

Overrides:

getStartLine in class Node

Returns:

the line number of the first token character

getStartColumn

public int getStartColumn()

The column number of the first character in the token image.

Overrides:

getStartColumn in class Node

Returns:

the column number of the first token character

getEndLine

public int getEndLine()

The line number of the last character in the token image.

Overrides:

getEndLine in class Node

Returns:

the line number of the last token character

getEndColumn

public int getEndColumn()

The column number of the last character in the token image.

Overrides:

getEndColumn in class Node

Returns:

the column number of the last token character

getPreviousToken

public Token getPreviousToken()

Returns the previuos token. The previous token may be a token that has been ignored in the parsing. Note that if the token list feature hasn't been used in the tokenizer, this method will always return null. By default the token list feature is not used.

Returns:

the previous token, or null if no such token is available

Since:

1.4

See Also:

getNextToken(), Tokenizer.getUseTokenList(), Tokenizer.setUseTokenList(boolean)

getNextToken

public Token getNextToken()

Returns the next token. The next token may be a token that has been ignored in the parsing. Note that if the token list feature hasn't been used in the tokenizer, this method will always return null. By default the token list feature is not used.

Returns:

the next token, or null if no such token is available

Since:

1.4

See Also:

getPreviousToken(), Tokenizer.getUseTokenList(), Tokenizer.setUseTokenList(boolean)

toString

public java.lang.String toString()

Returns a detailed string representation of this token.

Overrides:

toString in class java.lang.Object

Returns:

a detailed string representation of this token

See Also:

Object.toString()

toShortString

public java.lang.String toShortString()

Returns a short string representation of this token. The string will only contain the token image and possibly the token pattern name.

Returns:

a short string representation of this token