Implementing of strtok() function in C++

The strtok() function is one of the most utilised functions in C++. Using a delimiter as a guide, this function can divide a text into smaller chunks or tokens. It is simple to work with strings in C++ thanks to the strtok() function. The strtok() function will be thoroughly examined in this article, along with its definition, syntax, algorithm, and various implementation strategies. It is crucial to remember that the strtok function has several restrictions and potential downsides. It cannot be used on const or read-only strings, for instance, because it changes the original string in place. Edge situations and unexpected input, such as empty strings or delimiters that appear repeatedly in a sequence, can also be challenging to manage.

This function continues to be a useful tool for many programmers despite these drawbacks and is frequently used in a wide range of applications, including text processing, data parsing, and network protocols. It is a flexible and strong function that may significantly streamline many routine programming tasks.

Characteristics of strtok() function

To split a string into smaller chunks or tokens based on a delimiter, use the strtok() function in C++. A pointer to the string that has to be tokenized and a string with the delimiter characters are the two inputs for the function. A pointer to the string's very first token is returned by the function. As long as there are tokens left in the string, successive calls to the function with the same string parameter will return additional tokens and this happens with the help of NULL pointer in while loop. When calling the'strtok()' function again, supplying NULL as the first argument instructs the function to pick up where it left off with the previous tokenization of the same string.

When the function'strtok()' is first called with the input string as the first argument, it starts looking for the first token in the string at the beginning. When it locates the initial token, it terminates it by replacing the delimiter that comes after it with the null character (''0''). When'strtok()' is used again with a NULL pointer as the first argument, the function picks up where it left off with the previous tokenization of the string, that is, directly after the previous token. It keeps looking for the delimiter's next appearance.

Syntax

Char* strtok(char* str, const char* delimiters);  

Where

- `str` is a pointer to the string to be tokenized.

- `delimiters` is a string containing the delimiter characters.

Algorithm

• Step 1 − The `strtok()` function takes two arguments - the first argument is the input string to be tokenized and the second argument is a string containing all the delimiters that should be used to split the input string into tokens.

• Step 2 − When the function is called for the first time, the input string is passed as the first argument, and the delimiter string is passed as the second argument.

• Step 3 − The function searches for the first occurrence of any of the delimiter characters in the input string.

• Step 4 − When it finds a delimiter character, it replaces it with a null terminator (`'\0'`) and returns a pointer to the beginning of the first token.

• Step 5 − The next time the function is called, the first argument should be set to `NULL` instead of the original input string. This tells the function to continue where it left off and search for the next token in the string.

• Step 6 − The function continues to search for delimiter characters and returns pointers to the beginning of subsequent tokens until it reaches the end of the string, at which point it returns `NULL`.

Approaches to follow

Approach 1 − Program to show code using a loop with strtok() function and single delimeter.

Approach 2 − Program to show code using a loop with strtok() function and multiple delimeter

Approach 1

Below is the program to show code using a loop with strtok() function and single delimeter

This example demonstrates how to tokenize a string by space using strtok. The input string str is first passed to strtok along with the delimiter " ". The first call to strtok returns a pointer to the first token ("Hi,"), which is printed to the console. strtok continues to tokenize the string until there are no more tokens, and each token is printed to the console. Note that for subsequent calls to strtok, we pass a null pointer as the first argument, indicating that we want to continue tokenizing the original string

Example 1

#include <iostream>
#include <cstring> // Include the header for strtok()
using namespace std;

int main() {
   char str[] = "Hi, this topic tells about strtok() function";
   char* ptr;
   ptr = strtok(str, " ");
   while (ptr != NULL) {
      cout << ptr << endl;
      ptr = strtok(NULL, " ");
   }
   return 0;
}

Output

Hi, 
this  
topic 
tells  
about 
strtok()  
function 

Approach 2

Below is the program to show code using a loop with strtok() function with multiple delimeters.Below is the program code for the same.

The '|' and'' (space) delimiters are used to tokenize the string "The boy|is|standing|with his pet|lazy dog." in this example. Once with the string as the first argument and the characters '|' and'' as the second argument, the strtok() method is called twice. Following calls return the remaining tokens, the initial call returns the first token, "The." Until all tokens have been extracted, the loop keeps running.

Example-2

#include <iostream> 
#include <cstring> 
 
int main() { 
   char str[] = "The boy|is|standing|with his pet|lazy dog."; 
   char* token = strtok(str, "| "); 
   while (token != NULL) { 
      std::cout << token << std::endl; 
      token = strtok(NULL, "| "); 
   } 
   return 0; 
} 

Output

The 
boy 
is 
standing 
with 
his 
pet 
lazy 
dog.

Conclusion

In conclusion, the C++ strtok function is a helpful and effective tool for manipulating strings. Even though it has significant restrictions and potential downsides, it is nevertheless a common option for parsing and processing text data and is a useful tool for any coder.