Tokenizer Builder - Problem

Build a tokenizer that breaks a math expression string into tokens: numbers, operators, and parentheses.

A math expression contains:

Numbers: Integer sequences like 123, 45
Operators: +, -, *, /
Parentheses: ( and )

Return an array of tokens in the order they appear. Ignore spaces between tokens.

Input & Output

Example 1 — Basic Expression

$ Input: expression = "3 + 45 * (2)"

› Output: ["3", "+", "45", "*", "(", "2", ")"]

💡 Note: Numbers (3, 45, 2), operators (+, *), and parentheses are separated into individual tokens. Spaces are ignored.

Example 2 — No Spaces

$ Input: expression = "123+456"

› Output: ["123", "+", "456"]

💡 Note: Multi-digit numbers 123 and 456 are correctly identified as single tokens even without spaces.

Example 3 — Complex Expression

$ Input: expression = "(1+2)*3/4"

› Output: ["(", "1", "+", "2", ")", "*", "3", "/", "4"]

💡 Note: All token types: parentheses, single-digit numbers, and all four basic operators.

Constraints

1 ≤ expression.length ≤ 1000
expression contains only digits 0-9, operators +, -, *, /, parentheses (, ), and spaces
Numbers are positive integers

Visualization

Tap to expand

Asked in

G Google 35 M Microsoft 28 a Amazon 22 Apple 18

The key insight is recognizing token boundaries while parsing. The optimal approach uses a single-pass state machine to efficiently identify and extract numbers, operators, and parentheses. Time: O(n), Space: O(n)

Common Approaches

✓ Character-by-Character Processing

⏱️ Time: O(n) Space: O(n)

Process the expression character by character, manually checking each character type and building tokens by appending characters until we hit a boundary.

Single Pass with State Machine

⏱️ Time: O(n) Space: O(n)

Implement a state machine approach where we track current parsing state (reading number, operator, etc.) and efficiently build tokens without redundant string operations.

Character-by-Character Processing — Algorithm Steps

Iterate through each character in the expression
Check if character is digit, operator, parenthesis, or space
Build current token by appending characters of same type
When type changes, add completed token to result

Visualization

Tap to expand

Step-by-Step Walkthrough

Start

Begin at first character, empty token list

Process

Check character type, build current token

Complete

Add finished tokens to result array

Code -

solution.c — C

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <ctype.h>

char** solution(char* expression, int* returnSize) {
    char** tokens = malloc(100 * sizeof(char*));
    *returnSize = 0;
    int i = 0;
    int len = strlen(expression);
    
    while (i < len) {
        if (expression[i] == ' ') {
            i++;
        } else if (isdigit(expression[i])) {
            char* num = malloc(20 * sizeof(char));
            int numIdx = 0;
            while (i < len && isdigit(expression[i])) {
                num[numIdx++] = expression[i];
                i++;
            }
            num[numIdx] = '\0';
            tokens[*returnSize] = num;
            (*returnSize)++;
        } else if (strchr("+-*/()", expression[i]) != NULL) {
            char* op = malloc(2 * sizeof(char));
            op[0] = expression[i];
            op[1] = '\0';
            tokens[*returnSize] = op;
            (*returnSize)++;
            i++;
        } else {
            i++;
        }
    }
    return tokens;
}

int main() {
    char expression[1000];
    fgets(expression, sizeof(expression), stdin);
    expression[strcspn(expression, "\n")] = 0;
    
    int returnSize;
    char** result = solution(expression, &returnSize);
    
    printf("[");
    for (int i = 0; i < returnSize; i++) {
        if (i > 0) printf(",");
        printf("\"%s\"", result[i]);
    }
    printf("]\n");
    
    return 0;
}

Time & Space Complexity

Time Complexity

⏱️

O(n)

Single pass through the string, each character examined once

✓ Linear Growth

Space Complexity

O(n)

Result array stores all tokens, worst case n tokens for n characters

⚡ Linearithmic Space

28.5K Views

Medium Frequency

~15 min Avg. Time

890 Likes

Ln 1, Col 1

Smart Actions

💡 Explanation

AI Ready

💡 Suggestion Tab to accept Esc to dismiss

// Output will appear here after running code

Code Editor Closed

Click the red button to reopen

Tokenizer Builder - Problem

Input & Output

Constraints

Visualization

Related Problems

Common Approaches

Character-by-Character Processing — Algorithm Steps

Visualization

Code -

Time & Space Complexity

Select Compiler