Python Text Processing - Counting Tokens in Paragraphs

While reading the text from a source, sometimes we also need to find out some statistics about the type of words used. That makes it necessary to count the number of words as well as lines with a specific type of words in a given text. In the below example we show programs to count the words in a paragraph using two different approaches. We consider a text file for this purpose which contains the summary of a Hollywood movie.

Reading the File

main.py

fileName = "GodFather.txt"

with open(fileName, 'r') as file:
    lines_in_file = file.read()
    print(lines_in_file) 

Output

When we run the above program we get the following output −

Vito Corleone is the aging don (head) of the Corleone Mafia Family. 
His youngest son Michael has returned from WWII just in time to see 
the wedding of Connie Corleone (Michael's sister) to Carlo Rizzi. 
...

Counting Words Using nltk

Next we use the nltk module to count the words in the text. Please note the word '(head)' is counted as 3 words and not one.

main.py

import nltk

fileName = "GodFather.txt"

with open(fileName, 'r') as file:
    lines_in_file = file.read()
    
    nltk_tokens = nltk.word_tokenize(lines_in_file)
    print(nltk_tokens)
    print("\n")
    print("Number of Words: " , len(nltk_tokens))

Output

When we run the above program we get the following output −

['Vito', 'Corleone', 'is', 'the', 'aging', 'don',
...
]

Number of Words:  167

Counting Words Using Split

Next we count the words using Split function and here the word '(head)' is counted as a single word and not 3 words as in case of using nltk.

fileName = "GodFather.txt"

with open(fileName, 'r') as file:
    lines_in_file = file.read()

    print(lines_in_file.split())
    print("\n")
    print("Number of Words: ", len(lines_in_file.split()))

Output

When we run the above program we get the following output −

['Vito', 'Corleone', 'is', 'the', 'aging', 'don', 
...
]

Number of Words:  146