Lucene - TokenStream

TokenStream is an output of the analysis process and it comprises of a series of tokens. It is an abstract class.

Class Declaration

Following is the declaration for the org.apache.lucene.analysis.TokenStream class −

public abstract class TokenStream
   extends AttributeSource
      implements Closeable

Class Constructors

The following table shows the different class constructors −

S.No. Constructor & Description

protected TokenStream()

A TokenStream that uses the default attribute factory.


protected TokenStream(AttributeSource.AttributeFactory factory)

A TokenStream that uses the supplied AttributeFactory for creating new Attribute instances.


protected TokenStream(AttributeSource input)

A TokenStream that uses the same attributes as the supplied one.

Class Methods

The following table shows the different class methods −

S.No. Method & Description

void close()

Releases resources associated with this stream.


void end()

This method is called by the consumer after the last token has been consumed, after incrementToken() returned false (using the new TokenStream API).


abstract boolean incrementToken()

Consumers (i.e., IndexWriter) use this method to advance the stream to the next token.


void reset()

Resets this stream to the beginning.

Methods Inherited

This class inherits methods from the following classes −

  • org.apache.lucene.util.AttributeSource
  • java.lang.Object