Package jadex.common

Class ExpressionTokenizer

java.lang.Object

jadex.common.ExpressionTokenizer

public class ExpressionTokenizer
extends Object

An expression tokenizer is able to parse a string with respect to different nesting-levels (eg. brackets, quotes).

Field Summary

Fields

Modifier and Type	Field	Description
protected String	allseps	All separators.
protected String	close	The nesting level delimiters.
static final String	ESCAPE_CHARACTERS	The escape characters allow to include separators in the tokens.
protected String	open	The nesting level delimiters.
protected int	pos	The current tokenization position.
protected boolean	retsep	Return the separators.
protected String	separators	The top level separators.
protected String	source	The source string.

Constructor Summary

Constructors

Constructor	Description
ExpressionTokenizer(String string, String separators, String[] metas)	Create a new tokenizer.
ExpressionTokenizer(String string, String separators, String[] metas, boolean retsep)	Create a new tokenizer.

Method Summary

Modifier and Type	Method	Description
int	countTokens()	Count the number of tokens.
boolean	hasMoreTokens()	Test if there are more tokens available.
static void	main(String[] args)	Main for testing.
protected boolean	nestingSeparatorAt(int pos)	Check if there is a nesting level separator character at the specified position.
String	nextToken()	Get the next token from the string.
String	remainingTokens()	Get the remaining tokens as single string.
protected boolean	separatorAt(int pos)	Check if there is a separator character at the specified position.
void	setPosition(int pos)	Set the parse position manually.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

ESCAPE_CHARACTERS

public static final String ESCAPE_CHARACTERS

The escape characters allow to include separators in the tokens. E.g. with escape character '\' it is possible to parse "(\))".

See Also:

• Constant Field Values

source

protected String source

The source string.

separators

protected String separators

The top level separators.

open

protected String open

The nesting level delimiters.

close

protected String close

The nesting level delimiters.

retsep

protected boolean retsep

Return the separators.

allseps

protected String allseps

All separators.

pos

protected int pos

The current tokenization position.

Constructor Details

ExpressionTokenizer

public ExpressionTokenizer(String string,
 String separators,
 String[] metas)

Create a new tokenizer.

Parameters:

string - The string.

separators - The separator chars.

metas - The nesting level delimiters.

ExpressionTokenizer

public ExpressionTokenizer(String string,
 String separators,
 String[] metas,
 boolean retsep)

Create a new tokenizer.

Parameters:

string - The string.

separators - The separator chars.

metas - The nesting level delimiters.

retsep - Return the separator chars.

Method Details

nextToken

public String nextToken()

Get the next token from the string.

Returns:

The next token.

remainingTokens

public String remainingTokens()

Get the remaining tokens as single string. When separators are to be returned, the remaining string will start with the next separator. Otherwise, the separators at the current position are skipped.

Returns:

The remaining tokens token.

hasMoreTokens

public boolean hasMoreTokens()

Test if there are more tokens available.

countTokens

public int countTokens()

Count the number of tokens.

Returns:

The number of tokens.

setPosition

public void setPosition(int pos)

Set the parse position manually. Be careful when using this method!.

Parameters:

pos - The new position.

separatorAt

protected boolean separatorAt(int pos)

Check if there is a separator character at the specified position.

Parameters:

pos - The position to check.

Returns:

true, if the position contains a separator.

nestingSeparatorAt

protected boolean nestingSeparatorAt(int pos)

Check if there is a nesting level separator character at the specified position.

Parameters:

pos - The position to check.

Returns:

true, if the position contains a separator.

main

public static void main(String[] args)

Main for testing.