Token

Overview

Package

Class

Use

Tree

Index

Help

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

org.basex.util
Class Token

java.lang.Object
  org.basex.util.Token

public final class Token
extends java.lang.Object
extends java.lang.Object

This class provides convenience operations for handling so-called 'Tokens'. Tokens in BaseX are nothing else than UTF8 encoded strings, stored in a byte array. Note that, to guarantee a consistent string representation, all string conversions should be done via the methods of this class.

Author:: Workgroup DBIS, University of Konstanz 2005-07, ISC License, Christian Gruen

Field Summary
`static byte[]`	`DOTS` Dots.
`static byte[]`	`E_AMP` Ampersand Entity.
`static byte[]`	`E_APOS` Apostrophe Entity.
`static byte[]`	`E_CR` CarriageReturn Entity.
`static byte[]`	`E_GT` GreaterThan Entity.
`static byte[]`	`E_LT` LessThan Entity.
`static byte[]`	`E_NL` NewLine Entity.
`static byte[]`	`E_QU` Quote Entity.
`static byte[]`	`E_TAB` Tab Entity.
`static byte[]`	`EMPTY` Empty token.
`static byte[]`	`FALSE` False token.
`static byte[]`	`INF` Infinity.
`static int`	`MAXLEN` Maximum length for hash calculation and index terms.
`static byte[]`	`MININT` Minimum integer.
`static byte[]`	`MZERO` Zero token.
`static byte[]`	`NAN` Infinity.
`static byte[]`	`NINF` Infinity.
`static byte[]`	`ONE` One token.
`static byte[]`	`SPACE` Space token.
`static byte[]`	`TRUE` True token.
`static java.lang.String`	`UTF8` UTF8 encoding string.
`static java.lang.String`	`UTF82` UTF8 encoding string (variant).
`static byte[]`	`ZERO` Zero token.

Method Summary
`static byte[]`	`append(byte[] t, byte a)` Appends a character to the specified byte array.
`static byte[]`	`append(byte[] t, byte[] a)` Appends an array to the specified byte array.
`static boolean`	`ascii(byte[] text)` Checks if the specified token only consists of ASCII characters.
`static byte[]`	`chopNumber(byte[] t)` Finishes the numeric token, removing trailing zeroes.
`static int`	`cl(int v)` Returns the expected codepoint length of the specified byte.
`static boolean`	`contains(byte[] tok, byte[] sub)` Checks if the first token contains the second token.
`static boolean`	`contains(byte[] tok, int c)` Checks if the first token contains the specified character.
`static boolean`	`containslc(byte[] tok, byte[] sub)` Checks if the first token contains the second token in lowercase.
`static int`	`cp(byte[] b, int i)` Returns the codepoint of the specified bytes, starting at the specified position.
`static byte[]`	`delete(byte[] t, byte[] c)` Deletes the specified characters out of the token.
`static byte[]`	`delete(byte[] t, int c)` Deletes the specified character out of the token.
`static int`	`diff(byte[] tok, byte[] tok2)` Calculates the difference of two character arrays.
`static boolean`	`digit(int c)` Checks if the specified character is a digit.
`static boolean`	`endsWith(byte[] tok, byte[] sub)` Checks if the first token ends with the second token.
`static boolean`	`endsWith(byte[] tok, int c)` Checks if the first token starts with the specified character.
`static boolean`	`eq(byte[] tok, byte[] tok2)` Compares two character arrays for equality.
`static int`	`ft(int ch)` Returns a lowercase ASCII character version of the specified character.
`static boolean`	`ftChar(byte ch)` Checks if the specified character is a letter; special characters are converted to the standard ASCII charset.
`static boolean`	`ftcontains(byte[] tok, byte[] sub)` Checks if the first token contains the second fulltext term.
`static int`	`ftNorm(int ch)` Returns a lowercase ASCII character version of the specified character.
`static int`	`hash(byte[] tok)` Calculates a hash code for the specified token.
`static int`	`indexOf(byte[] tok, byte[] sub)` Returns the position of the specified token or -1.
`static int`	`indexOf(byte[] tok, byte[] sub, int p)` Returns the position of the specified token or -1.
`static int`	`indexOf(byte[] tok, int c)` Returns the position of the specified character or -1.
`static byte[]`	`insert(byte[] t, int i, byte[] c)` Inserts the specified characters at the specified position.
`static int`	`lastIndexOf(byte[] tok, int c)` Returns the last position of the specified character or -1.
`static byte[]`	`lc(byte[] t)` Converts the specified token to lower case.
`static int`	`lc(int ch)` Converts a character to lower case.
`static int`	`len(byte[] text)` Returns the token length.
`static boolean`	`letter(byte[] tok)` Checks if the specified token consists only of letters.
`static boolean`	`letter(int c)` Checks if the specified character is a letter.
`static boolean`	`letterOrDigit(byte[] tok)` Checks if the specified token consists only of letters and digits.
`static boolean`	`letterOrDigit(int c)` Checks if the specified character is a letter or digit.
`static byte[]`	`norm(byte[] tok)` Normalizes all whitespace occurrences from the specified token.
`static int`	`numDigits(int x)` Checks number of digits of the specified integer.
`static byte[]`	`replace(byte[] t, int s, int r)` Replaces the specified character and returns the result token.
`static byte[][]`	`split(byte[] tok, int sep)` Splits the token at all whitespaces and returns a array with all tokens.
`static boolean`	`startsWith(byte[] tok, byte[] sub)` Checks if the first token starts with the second token.
`static boolean`	`startsWith(byte[] tok, int c)` Checks if the first token starts with the specified character.
`static java.lang.String`	`string(byte[] text)` Returns the specified token as string.
`static java.lang.String`	`string(byte[] text, int s, int l)` Returns the specified token as string.
`static byte[]`	`substring(byte[] tok, int s)` Returns a subtoken of the specified token.
`static byte[]`	`substring(byte[] tok, int s, int e)` Returns a subtoken of the specified token.
`static double`	`toDouble(byte[] to)` Converts the specified token into a double value.
`static double`	`toDouble(java.lang.String to)` Converts the specified string into a double value.
`static int`	`toInt(byte[] to)` Converts the specified token into an integer value.
`static int`	`toInt(byte[] to, int ts, int te)` Converts the specified token into an integer value.
`static int`	`toInt(java.lang.String to)` Converts the specified string into an integer value.
`static byte[]`	`token(boolean b)` Creates a byte array representation of the specified boolean value.
`static byte[]`	`token(double d)` Creates a byte array representation from the specified double value; inspired by Xavier Franc's Qizx.
`static byte[]`	`token(float f)` Creates a byte array representation from the specified float value.
`static byte[]`	`token(int i)` Creates a byte array representation of the specified integer value.
`static byte[]`	`token(long i)` Creates a byte array representation from the specified long value, using Java's standard method.
`static byte[]`	`token(java.lang.String s)` Converts a string to a byte array.
`static long`	`toLong(byte[] to)` Converts the specified token into an long value.
`static long`	`toLong(byte[] to, int ts, int te)` Converts the specified token into an long value.
`static long`	`toLong(java.lang.String to)` Converts the specified string into an long value.
`static byte[]`	`translate(byte[] tok, byte[] srch, byte[] rep)` Performs a translation on the specified token.
`static byte[]`	`trim(byte[] t)` Removes leading and trailing whitespaces from the specified token.
`static byte[]`	`uc(byte[] t)` Converts the specified token to upper case.
`static int`	`uc(int ch)` Switches a character to upper case.
`static byte[]`	`utf8(byte[] s, java.lang.String enc)` Converts a token from the input encoding to UTF8.
`static boolean`	`ws(byte c)` Checks if the specified character is a whitespace.
`static boolean`	`ws(byte[] tok)` Checks if the specified token has only whitespaces.

Methods inherited from class java.lang.Object
`equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

DOTS

public static final byte[] DOTS

Dots.

E_AMP

public static final byte[] E_AMP

Ampersand Entity.

E_APOS

public static final byte[] E_APOS

Apostrophe Entity.

E_CR

public static final byte[] E_CR

CarriageReturn Entity.

E_GT

public static final byte[] E_GT

GreaterThan Entity.

E_LT

public static final byte[] E_LT

LessThan Entity.

E_NL

public static final byte[] E_NL

NewLine Entity.

E_QU

public static final byte[] E_QU

Quote Entity.

E_TAB

public static final byte[] E_TAB

Tab Entity.

EMPTY

public static final byte[] EMPTY

Empty token.

FALSE

public static final byte[] FALSE

False token.

INF

public static final byte[] INF

Infinity.

MAXLEN

public static final int MAXLEN

Maximum length for hash calculation and index terms.

See Also:: Constant Field Values

MININT

public static final byte[] MININT

Minimum integer.

MZERO

public static final byte[] MZERO

Zero token.

NAN

public static final byte[] NAN

Infinity.

NINF

public static final byte[] NINF

Infinity.

ONE

public static final byte[] ONE

One token.

SPACE

public static final byte[] SPACE

Space token.

TRUE

public static final byte[] TRUE

True token.

UTF8

public static final java.lang.String UTF8

UTF8 encoding string.

See Also:: Constant Field Values

UTF82

public static final java.lang.String UTF82

UTF8 encoding string (variant).

See Also:: Constant Field Values

ZERO

public static final byte[] ZERO

Zero token.

Method Detail

append

public static byte[] append(byte[] t,
                            byte a)

Appends a character to the specified byte array.

Parameters:: t - original token; a - token to be appended
Returns:: resulting array

append

public static byte[] append(byte[] t,
                            byte[] a)

Appends an array to the specified byte array.

Parameters:: t - original token; a - token to be appended
Returns:: resulting array

ascii

public static boolean ascii(byte[] text)

Checks if the specified token only consists of ASCII characters.

Parameters:: text - token
Returns:: result of check

chopNumber

public static byte[] chopNumber(byte[] t)

Finishes the numeric token, removing trailing zeroes.

Parameters:: t - token to be modified
Returns:: token

cl

public static int cl(int v)

Returns the expected codepoint length of the specified byte.

Parameters:: v - first character byte
Returns:: character length

contains

public static boolean contains(byte[] tok,
                               byte[] sub)

Checks if the first token contains the second token.

Parameters:: tok - first token; sub - second token
Returns:: result of test

contains

public static boolean contains(byte[] tok,
                               int c)

Checks if the first token contains the specified character.

Parameters:: tok - first token; c - character
Returns:: result of test

containslc

public static boolean containslc(byte[] tok,
                                 byte[] sub)

Checks if the first token contains the second token in lowercase.

Parameters:: tok - first token; sub - second token
Returns:: result of test

cp

public static int cp(byte[] b,
                     int i)

Returns the codepoint of the specified bytes, starting at the specified position.

Parameters:: b - byte array; i - character position
Returns:: current character

delete

public static byte[] delete(byte[] t,
                            byte[] c)

Deletes the specified characters out of the token.

Parameters:: t - token to be checked; c - characters to be removed
Returns:: new instance

delete

public static byte[] delete(byte[] t,
                            int c)

Deletes the specified character out of the token.

Parameters:: t - token to be checked; c - character to be removed
Returns:: new instance

diff

public static int diff(byte[] tok,
                       byte[] tok2)

Calculates the difference of two character arrays.

Parameters:: tok - token to be compared; tok2 - second token to be compared
Returns:: 0 if tokens are equal, negative if first token is smaller, positive if first token is bigger

digit

public static boolean digit(int c)

Checks if the specified character is a digit.

Parameters:: c - the letter to be checked
Returns:: result of comparison

endsWith

public static boolean endsWith(byte[] tok,
                               byte[] sub)

Checks if the first token ends with the second token.

Parameters:: tok - first token; sub - second token
Returns:: result of test

endsWith

public static boolean endsWith(byte[] tok,
                               int c)

Checks if the first token starts with the specified character.

Parameters:: tok - first token; c - character
Returns:: result of test

eq

public static boolean eq(byte[] tok,
                         byte[] tok2)

Compares two character arrays for equality.

Parameters:: tok - token to be compared; tok2 - second token to be compared
Returns:: true if the arrays are equal

ft

public static int ft(int ch)

Returns a lowercase ASCII character version of the specified character.

Parameters:: ch - character to be converted
Returns:: converted character

ftChar

public static boolean ftChar(byte ch)

Checks if the specified character is a letter; special characters are converted to the standard ASCII charset.

Parameters:: ch - character to be converted
Returns:: converted character

ftcontains

public static boolean ftcontains(byte[] tok,
                                 byte[] sub)

Checks if the first token contains the second fulltext term.

Parameters:: tok - first token; sub - second token
Returns:: result of test

ftNorm

public static int ftNorm(int ch)

Returns a lowercase ASCII character version of the specified character.

Parameters:: ch - character to be converted
Returns:: converted character

hash

public static int hash(byte[] tok)

Calculates a hash code for the specified token.

Parameters:: tok - specified token
Returns:: hash code

indexOf

public static int indexOf(byte[] tok,
                          byte[] sub)

Returns the position of the specified token or -1.

Parameters:: tok - first token; sub - second token
Returns:: result of test

indexOf

public static int indexOf(byte[] tok,
                          byte[] sub,
                          int p)

Returns the position of the specified token or -1.

Parameters:: tok - first token; sub - second token; p - start position
Returns:: result of test

indexOf

public static int indexOf(byte[] tok,
                          int c)

Returns the position of the specified character or -1.

Parameters:: tok - first token; c - character
Returns:: result of test

insert

public static byte[] insert(byte[] t,
                            int i,
                            byte[] c)

Inserts the specified characters at the specified position.

Parameters:: t - token to be modified; i - insert position; c - characters to be inserted
Returns:: new instance

lastIndexOf

public static int lastIndexOf(byte[] tok,
                              int c)

Returns the last position of the specified character or -1.

Parameters:: tok - first token; c - character
Returns:: result of test

lc

public static byte[] lc(byte[] t)

Converts the specified token to lower case.

Parameters:: t - token to be converted
Returns:: the converted token

lc

public static int lc(int ch)

Converts a character to lower case.

Parameters:: ch - character to be converted
Returns:: converted character

len

public static int len(byte[] text)

Returns the token length.

Parameters:: text - token
Returns:: length

letter

public static boolean letter(byte[] tok)

Checks if the specified token consists only of letters.

Parameters:: tok - token to be checked
Returns:: result of comparison

letter

public static boolean letter(int c)

Checks if the specified character is a letter.

Parameters:: c - the letter to be checked
Returns:: result of comparison

letterOrDigit

public static boolean letterOrDigit(byte[] tok)

Checks if the specified token consists only of letters and digits.

Parameters:: tok - token to be checked
Returns:: result of comparison

letterOrDigit

public static boolean letterOrDigit(int c)

Checks if the specified character is a letter or digit.

Parameters:: c - the letter to be checked
Returns:: result of comparison

norm

public static byte[] norm(byte[] tok)

Normalizes all whitespace occurrences from the specified token.

Parameters:: tok - token
Returns:: normalized token.

numDigits

public static int numDigits(int x)

Checks number of digits of the specified integer.

Parameters:: x - number to be checked
Returns:: number of digits

replace

public static byte[] replace(byte[] t,
                             int s,
                             int r)

Replaces the specified character and returns the result token.

Parameters:: t - token to be checked; s - the character to be replaced; r - the new character
Returns:: resulting token

split

public static byte[][] split(byte[] tok,
                             int sep)

Splits the token at all whitespaces and returns a array with all tokens.

Parameters:: tok - token to be split; sep - separation character
Returns:: array

startsWith

public static boolean startsWith(byte[] tok,
                                 byte[] sub)

Checks if the first token starts with the second token.

Parameters:: tok - first token; sub - second token
Returns:: result of test

startsWith

public static boolean startsWith(byte[] tok,
                                 int c)

Checks if the first token starts with the specified character.

Parameters:: tok - first token; c - character
Returns:: result of test

string

public static java.lang.String string(byte[] text)

Returns the specified token as string.

Parameters:: text - token
Returns:: string

string

public static java.lang.String string(byte[] text,
                                      int s,
                                      int l)

Returns the specified token as string.

Parameters:: text - token; s - start position; l - length
Returns:: string

substring

public static byte[] substring(byte[] tok,
                               int s)

Returns a subtoken of the specified token.

Parameters:: tok - token; s - start position
Returns:: subtoken

substring

public static byte[] substring(byte[] tok,
                               int s,
                               int e)

Returns a subtoken of the specified token.

Parameters:: tok - token; s - start position; e - end position
Returns:: subtoken

toDouble

public static double toDouble(byte[] to)

Converts the specified token into a double value. Double.NaN is returned when the input is invalid.

Parameters:: to - character array to be converted
Returns:: converted double value

toDouble

public static double toDouble(java.lang.String to)

Converts the specified string into a double value. Double.NaN is returned when the input is invalid.

Parameters:: to - character array to be converted
Returns:: converted double value

toInt

public static int toInt(byte[] to)

Converts the specified token into an integer value. Integer.MIN_VALUE is returned when the input is invalid.

Parameters:: to - character array to be converted
Returns:: converted integer value

toInt

public static int toInt(byte[] to,
                        int ts,
                        int te)

Converts the specified token into an integer value. Integer.MIN_VALUE is returned when the input is invalid.

Parameters:: to - character array to be converted; ts - first byte to be parsed; te - last byte to be parsed (exclusive)
Returns:: converted integer value

toInt

public static int toInt(java.lang.String to)

Converts the specified string into an integer value. Integer.MIN_VALUE is returned when the input is invalid.

Parameters:: to - character array to be converted
Returns:: converted integer value

token

public static byte[] token(boolean b)

Creates a byte array representation of the specified boolean value.

Parameters:: b - boolean value to be converted
Returns:: boolean value in byte array

token

public static byte[] token(double d)

Creates a byte array representation from the specified double value; inspired by Xavier Franc's Qizx.

Parameters:: d - double value to be converted
Returns:: byte array

token

public static byte[] token(float f)

Creates a byte array representation from the specified float value.

Parameters:: f - float value to be converted
Returns:: byte array

token

public static byte[] token(int i)

Creates a byte array representation of the specified integer value.

Parameters:: i - int value to be converted
Returns:: integer value in byte array

token

public static byte[] token(long i)

Creates a byte array representation from the specified long value, using Java's standard method.

Parameters:: i - int value to be converted
Returns:: byte array

token

public static byte[] token(java.lang.String s)

Converts a string to a byte array. All strings should be converted by this function to guarantee a consistent character conversion. Note that unicode characters are not supported yet.

Parameters:: s - string to be converted
Returns:: byte array

toLong

public static long toLong(byte[] to)

Converts the specified token into an long value. Long.MIN_VALUE is returned when the input is invalid.

Parameters:: to - character array to be converted
Returns:: converted long value

toLong

public static long toLong(byte[] to,
                          int ts,
                          int te)

Converts the specified token into an long value. Long.MIN_VALUE is returned when the input is invalid.

Parameters:: to - character array to be converted; ts - first byte to be parsed; te - last byte to be parsed - exclusive
Returns:: converted long value

toLong

public static long toLong(java.lang.String to)

Converts the specified string into an long value. Long.MIN_VALUE is returned when the input is invalid.

Parameters:: to - character array to be converted
Returns:: converted long value

translate

public static byte[] translate(byte[] tok,
                               byte[] srch,
                               byte[] rep)

Performs a translation on the specified token.

Parameters:: tok - token; srch - characters to be found; rep - characters to be replaced
Returns:: translated token.

trim

public static byte[] trim(byte[] t)

Removes leading and trailing whitespaces from the specified token.

Parameters:: t - token to be checked
Returns:: chopped array

uc

public static byte[] uc(byte[] t)

Converts the specified token to upper case.

Parameters:: t - token to be converted
Returns:: the converted token

uc

public static int uc(int ch)

Switches a character to upper case.

Parameters:: ch - character to be converted
Returns:: converted character

utf8

public static byte[] utf8(byte[] s,
                          java.lang.String enc)

Converts a token from the input encoding to UTF8.

Parameters:: s - token to be converted; enc - input encoding
Returns:: byte array

ws

public static boolean ws(byte c)

Checks if the specified character is a whitespace.

Parameters:: c - the letter to be checked
Returns:: result of comparison

ws

public static boolean ws(byte[] tok)

Checks if the specified token has only whitespaces.

Parameters:: tok - token
Returns:: true if all characters are whitespaces

Overview

Package

Class

Use

Tree

Index

Help

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

org.basex.util Class Token

DOTS

E_AMP

E_APOS

E_CR

E_GT

E_LT

E_NL

E_QU

E_TAB

EMPTY

FALSE

INF

MAXLEN

MININT

MZERO

NAN

NINF

ONE

SPACE

TRUE

UTF8

UTF82

ZERO

append

append

ascii

chopNumber

cl

contains

contains

containslc

cp

delete

delete

diff

digit

endsWith

endsWith

eq

ft

ftChar

ftcontains

ftNorm

hash

indexOf

indexOf

indexOf

insert

lastIndexOf

lc

lc

len

letter

letter

letterOrDigit

letterOrDigit

norm

numDigits

replace

split

startsWith

startsWith

string

string

substring

substring

toDouble

toDouble

toInt

toInt

toInt

token

token

token

token

token

token

toLong

toLong

org.basex.util
Class Token