Lucene - Indexing Operations

Lucene - Query Programming

Lucene - Analysis

Lucene - Resources

Lucene - CustomAnalyzer

Quiz

We can create our own custom analyzer as per custom requirements using CustomAnalyzer.builder() method.

Class Declaration

Following is the declaration for org.apache.lucene.analysis.core.CustomAnalyzer class −

public final class CustomAnalyzer
   extends Analyzer

Class Methods

S.No.	Method & Description
1	static CustomAnalyzer.Builder builder() Returns a builder for custom analyzers that loads all resources from Lucene's classloader.
2	static CustomAnalyzer.Builder builder(Path configDir) Returns a builder for custom analyzers that loads all resources from the given file system base directory.
3	static CustomAnalyzer.Builder builder(ResourceLoader loader) Returns a builder for custom analyzers that loads all resources using the given ResourceLoader.
4	protected Analyzer.TokenStreamComponents createComponents(String fieldName)
5	List<CharFilterFactory> getCharFilterFactories() Returns the list of char filters that are used in this analyzer.
6	int getOffsetGap(String fieldName)
7	int getPositionIncrementGap(String fieldName)
8	List<TokenFilterFactory> getTokenFilterFactories() Returns the list of token filters that are used in this analyzer.
9	TokenizerFactory getTokenizerFactory() Returns the tokenizer that is used in this analyzer.
10	protected Reader initReader(String fieldName, Reader reader)
11	protected Reader initReaderForNormalization(String fieldName, Reader reader)
12	protected TokenStream normalize(String fieldName, TokenStream in)
13	String toString() String representation of the analyzer.

Methods Inherited

This class inherits methods from the following classes −

org.apache.lucene.analysis.StopwordAnalyzerBase
org.apache.lucene.analysis.Analyzer
java.lang.Object

Usage of CustomAnalyzer

private void displayTokenUsingCustomAnalyzer() throws IOException {
   String text 
      = "Lucene is simple yet powerful java based search library.";

   Analyzer analyzer = CustomAnalyzer.builder()
      .withTokenizer("standard")
      .addTokenFilter("lowercase")
      .addTokenFilter("stop")
      .addTokenFilter("capitalization")
      .build();
   TokenStream tokenStream = analyzer.tokenStream(
   LuceneConstants.CONTENTS, new StringReader(text));
   CharTermAttribute term = tokenStream.addAttribute(CharTermAttribute.class);
   tokenStream.reset();
   while(tokenStream.incrementToken()) {
      System.out.print("[" + term.toString() + "] ");
   }
   analyzer.close();
}

Example Application

To test search using CustomAnalyzer, let us create a test Lucene application.

Step	Description
1	Create a project with a name LuceneFirstApplication under a package com.tutorialspoint.lucene as explained in the Lucene - First Application chapter. You can also use the project created in Lucene - First Application chapter as such for this chapter to understand the searching process.
2	Create LuceneConstants.java as explained in the Lucene - First Application chapter. Keep the rest of the files unchanged.
3	Create LuceneTester.java as mentioned below.
4	Clean and Build the application to make sure business logic is working as per the requirements.

LuceneConstants.java

This class is used to provide various constants to be used across the sample application.

package com.tutorialspoint.lucene;

public class LuceneConstants {
   public static final String CONTENTS = "contents";
   public static final String FILE_NAME = "filename";
   public static final String FILE_PATH = "filepath";
   public static final int MAX_SEARCH = 10;
}

LuceneTester.java

This class is used to test the searching capability of the Lucene library.

package com.tutorialspoint.lucene;

import java.io.IOException;
import java.io.StringReader;

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.custom.CustomAnalyzer;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;

public class LuceneTester {
	
   public static void main(String[] args) {
      LuceneTester tester;

      tester = new LuceneTester();
   
      try {
         tester.displayTokenUsingCustomAnalyzer();
      } catch (IOException e) {
         e.printStackTrace();
      }
   }

   private void displayTokenUsingCustomAnalyzer() throws IOException {
      String text 
         = "Lucene is simple yet powerful java based search library.";
      
      Analyzer analyzer = CustomAnalyzer.builder()
    	      .withTokenizer("standard")
    	      .addTokenFilter("lowercase")
    	      .addTokenFilter("stop")
    	      .addTokenFilter("capitalization")
    	      .build();
      TokenStream tokenStream = analyzer.tokenStream(
         LuceneConstants.CONTENTS, new StringReader(text));
      CharTermAttribute term = tokenStream.addAttribute(CharTermAttribute.class);
      tokenStream.reset();
      while(tokenStream.incrementToken()) {
         System.out.print("[" + term.toString() + "] ");
      }
      analyzer.close();
   }
}

Running the Program

Once you are done with the creation of the source, the raw data, the data directory, the index directory and the indexes, you can proceed by compiling and running your program. To do this, keep the LuceneTester.Java file tab active and use either the Run option available in the Eclipse IDE or use Ctrl + F11 to compile and run your LuceneTester application. If your application runs successfully, it will print the following message in Eclipse IDE's console −

Output

[Lucene] [Simple] [Yet] [Powerful] [Java] [Based] [Search] [Library]

Print Page