Lucene - TokenStream

TokenStream is an output of the analysis process and it comprises of a series of tokens. It is an abstract class.

Class Declaration

Following is the declaration for the org.apache.lucene.analysis.TokenStream class −

public abstract class TokenStream
   extends AttributeSource
      implements Closeable

The following table shows the different class constructors −

S.No.	Constructor & Description
1	protected TokenStream() A TokenStream that uses the default attribute factory.
2	protected TokenStream(AttributeSource.AttributeFactory factory) A TokenStream that uses the supplied AttributeFactory for creating new Attribute instances.
3	protected TokenStream(AttributeSource input) A TokenStream that uses the same attributes as the supplied one.

S.No.

Constructor & Description

protected TokenStream()

A TokenStream that uses the default attribute factory.

protected TokenStream(AttributeSource.AttributeFactory factory)

A TokenStream that uses the supplied AttributeFactory for creating new Attribute instances.

protected TokenStream(AttributeSource input)

A TokenStream that uses the same attributes as the supplied one.

The following table shows the different class methods −

S.No.	Method & Description
1	void close() Releases resources associated with this stream.
2	void end() This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API).
3	abstract boolean incrementToken() Consumers (i.e., IndexWriter) use this method to advance the stream to the next token.
4	void reset() Resets this stream to the beginning.

This class inherits methods from the following classes −

lucene_analysis.htm