Data Mining - Mining Text Data



The text databases consist most of huge collection of documents. They collect these information from several sources such as news articles, books, digital libraries, e-mail messages, and web pages etc. Due to increase amount of information, the text databases are growing rapidly. In many of the text databases the data is semi structured.

For example, a document may contain a few structured fields, such as title, author, publishing_date etc. But along with the structure data the document also contains unstructured text components, such as abstract and contents. Without knowing what could be in the documents, it is difficult to formulate effective queries for analyzing and extracting useful information from the data. To compare the documents and rank the importance and relevance of the document the users need tools.Therefore, text mining has become popular and essential theme in data mining.

Information Retrieval

Information Retrieval deals with the retrieval of information from large number of text-based documents. Some of the database systems are not usually present in information retrieval system because both handle different kinds of data. Following are the examples of information retrieval system:

  • Online Library catalogue system

  • Online Document Management Systems

  • Web Search Systems etc.

Note: The main problem in information retrieval system is to locate relevant documents in a document collection based on user's query. This kind of user's query consists of some keywords describing an information need.

In such kind of search problem the user takes initiative to pull the relevant information out from the collection. This is appropriate when the user has ad-hoc information need i.e. short term need. But if the user has long term information need then the retrieval system can also take initiative to push any newly arrived information item to the user.

This kind of access to information is called Information Filtering. And the corresponding systems are known as Filtering Systems or Recommender Systems.

Basic Measures for Text Retrieval

We need to check how accurate or correct the system is when the system retrieved a number of documents on the basis of user's input. Let the set of documents relevant to a query be denoted as {Relevant} and the set of retrieved document as {Retrieved}. The set of documents that are relevant and retrieved can be denoted as {Relevant} ∩ {Retrieved}.This can be shown in the Venn diagram as follows:


There are three fundamental measures for assessing the quality of text retrieval:

  • Precision

  • Recall

  • F-score


Precision is the percentage of retrieved documents that are in fact relevant to the query. Precision can be defined as:

Precision= |{Relevant} ∩ {Retrieved}| /  |{Retrieved}|


Recall is the percentage of documents that are relevant to the query and were in fact retrieved. Recall is defined as:

Recall = |{Relevant} ∩ {Retrieved}| /  |{Relevant}|


F-score is the commonly used trade-off. The information retrieval system often needs to trade-off for precision or vice versa. F-score is defined as harmonic mean of recall or precision as follows:

F-score = recall x precision / (recall + precision) / 2