org.basex.index
Class FTTokenizer

java.lang.Object
  org.basex.index.IndexToken
      org.basex.index.FTTokenizer

public final class FTTokenizer
extends IndexToken

Full-text tokenizer.

Author:

Workgroup DBIS, University of Konstanz 2005-09, ISC License, Christian Gruen

Nested Class Summary

Nested classes/interfaces inherited from class org.basex.index.IndexToken

IndexToken.Type

Field Summary

int	count Number of tokens.
boolean	cs Sensitivity flag.
boolean	dc Diacritics flag.
boolean	fz Fuzzy flag.
int	lastpm Backup last punctuation mark.
boolean	lc Lowercase flag.
boolean	lp Flag for loading ftposition data.
int	p Current character position.
int	para Current paragraph.
int	pos Current token.
int	s Character start position.
Map<byte[]>	sd Stemming dictionary.
int	sent Current sentence.
boolean	st Stemming flag.
boolean	uc Uppercase flag.
boolean	wc Wildcard flag.

Fields inherited from class org.basex.index.IndexToken

text, type

Constructor Summary

FTTokenizer()
Empty constructor.

FTTokenizer(byte[] txt)
Constructor.

FTTokenizer(byte[] txt, FTOpt fto)
Constructor.

Method Summary

int	count() Counts the number of tokens.
byte[]	get() Returns the current index token.
int[][]	getInfo() Gets fulltext info out of text.
TokenList	getTokenList() Converts the tokens to a TokenList.
void	init() Initializes the iterator.
void	init(byte[] txt) Sets the text.
boolean	more() Checks if more tokens are to be returned.
int	size() Returns the text size.
java.lang.String	toString()

Methods inherited from class org.basex.index.IndexToken

range

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

sd

public Map<byte[]> sd

Stemming dictionary.

st

public boolean st

Stemming flag.

dc

public boolean dc

Diacritics flag.

cs

public boolean cs

Sensitivity flag.

uc

public boolean uc

Uppercase flag.

lc

public boolean lc

Lowercase flag.

wc

public boolean wc

Wildcard flag.

fz

public boolean fz

Fuzzy flag.

lp

public boolean lp

Flag for loading ftposition data.

sent

public int sent

Current sentence.

para

public int para

Current paragraph.

pos

public int pos

Current token.

p

public int p

Current character position.

s

public int s

Character start position.

count

public int count

Number of tokens.

lastpm

public int lastpm

Backup last punctuation mark.

Constructor Detail

FTTokenizer

public FTTokenizer()

Empty constructor.

FTTokenizer

public FTTokenizer(byte[] txt)

Constructor.

Parameters:

txt - text

FTTokenizer

public FTTokenizer(byte[] txt,
                   FTOpt fto)

Constructor.

Parameters:

txt - text

fto - fulltext options

Method Detail

init

public void init(byte[] txt)

Sets the text.

Parameters:

txt - text

init

public void init()

Initializes the iterator.

more

public boolean more()

Checks if more tokens are to be returned.

Returns:

result of check

get

public byte[] get()

Description copied from class: IndexToken

Returns the current index token. Can be overwritten by an implementation to return other tokens, as is done in the FTTokenizer.

Overrides:

get in class IndexToken

Returns:

token

count

public int count()

Counts the number of tokens.

Returns:

number of tokens

getInfo

public int[][] getInfo()

Gets fulltext info out of text. int[0]: length of each token int[1]: sentence info, length of each sentence int[2]: paragraph info, length of each paragraph int[3]: each token as int[] int[4]: punctuation marks of each sentence

Returns:

int arrays

size

public int size()

Returns the text size.

Returns:

size

getTokenList

public TokenList getTokenList()

Converts the tokens to a TokenList.

Returns:

TokenList

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object