Tokenize a string in C++?

First way is using a stringstream to read words seperated by spaces. This is a little limited but does the task fairly well if you provide the proper checks. 

example

#include <vector>
#include <string>
#include <sstream>

using namespace std;

int main() {
   string str("Hello from the dark side");
   string tmp; // A string to store the word on each iteration.
   stringstream str_strm(str);

   vector<string> words; // Create vector to hold our words

   while (str_strm >> tmp) {
      // Provide proper checks here for tmp like if empty
      // Also strip down symbols like !, ., ?, etc.
      // Finally push it.
      words.push_back(tmp);
   }
}

Example

Another way is to provide a custom delimiter to split the string by using the getline function −

#include <vector>
#include <string>
#include <sstream>

using namespace std;

int main() {
   std::stringstream str_strm("Hello from the dark side");
   std::string tmp;
   vector<string> words;
   char delim = ' '; // Ddefine the delimiter to split by

   while (std::getline(str_strm, tmp, delim)) {
      // Provide proper checks here for tmp like if empty
      // Also strip down symbols like !, ., ?, etc.
      // Finally push it.
      words.push_back(tmp);
   }
}

Ramu Prasad

Updated on 11-Feb-2020 11:03:12

• Related Questions & Answers
• Tokenize text using NLTK in python
• How can Tensorflow and Tensorflow text be used to tokenize string data?
• Interchanging a string to a binary string in JavaScript
• Left pad a String with a specified String in Java
• Check if a string contains a sub-string in C++
• String slicing in C# to rotate a string
• String slicing in Python to rotate a string
• Insert a String into another String in Java
• How to split a string with a string delimiter in C#?
• Reverse a string in C#
• Sorting a String in C#
• Tokenizing a String in Java
• Split a string in Java
• Tokenizing a string in C++
• A unique string in Python