Web Crawler Multithreaded - Practice Coding Problems

Web Crawler Multithreaded - Problem

Imagine you're building a web crawler that needs to explore all pages within a specific website domain as quickly as possible. You have a starting URL and need to discover all linked pages that belong to the same hostname, but there's a catch - single-threaded crawling is too slow!

Given a startUrl and an HtmlParser interface, implement a multi-threaded web crawler that:

🌐 Starts crawling from startUrl
📄 Uses HtmlParser.getUrls(url) to extract all URLs from each page
🚫 Never crawls the same URL twice (avoid infinite loops)
🏠 Only explores URLs with the same hostname as the starting URL
⚡ Utilizes multiple threads for concurrent crawling

Hostname Rules: URLs http://leetcode.com/problems and http://leetcode.com/contest share the same hostname (leetcode.com), but http://example.org/test and http://example.com/abc have different hostnames.

The HtmlParser interface is:

interface HtmlParser {
    // Returns all URLs found on the given webpage
    // This is a blocking HTTP request (takes ~15ms max)
    public List<String> getUrls(String url);
}

Challenge: Single-threaded solutions will exceed the time limit. Can your multi-threaded approach crawl faster by processing multiple pages simultaneously?

Input & Output

example_1.py — Basic Tree Structure

$ Input: startUrl = "http://news.yahoo.com/news/topics/" urls = ["http://news.yahoo.com/news/topics/", "http://news.yahoo.com/news/topics/1", "http://news.yahoo.com/news/topics/2"] edges = [[0,1],[0,2]] (URLs 0->1, 0->2 means URL 0 links to URLs 1 and 2)

› Output: ["http://news.yahoo.com/news/topics/", "http://news.yahoo.com/news/topics/1", "http://news.yahoo.com/news/topics/2"]

💡 Note: Starting from the root URL, we discover and crawl all linked pages within the same hostname (news.yahoo.com). The multi-threaded approach processes multiple URLs concurrently.

example_2.py — Complex Network

$ Input: startUrl = "http://news.yahoo.com/news/topics/" urls = ["http://news.yahoo.com/news/topics/", "http://news.yahoo.com/news/topics/1", "http://news.yahoo.com/news/topics/2", "http://news.google.com"] edges = [[0,1],[0,2],[1,3],[2,3]] (Cross-links between pages, including external domain)

› Output: ["http://news.yahoo.com/news/topics/", "http://news.yahoo.com/news/topics/1", "http://news.yahoo.com/news/topics/2"]

💡 Note: Even though URLs 1 and 2 link to google.com, we only crawl URLs with the same hostname (yahoo.com). External links are filtered out.

example_3.py — Circular References

$ Input: startUrl = "http://example.com/page1" urls = ["http://example.com/page1", "http://example.com/page2", "http://example.com/page3"] edges = [[0,1],[1,2],[2,0]] (Circular reference: page1->page2->page3->page1)

› Output: ["http://example.com/page1", "http://example.com/page2", "http://example.com/page3"]

💡 Note: Despite circular references, each URL is crawled exactly once due to the visited set. Multi-threading safely handles concurrent access to shared data structures.

Visualization

Tap to expand

Understanding the Visualization

Initialize Resources

Create shared queue, visited set, and launch worker threads

Parallel Processing

Each thread takes URLs from queue and processes them concurrently

Coordinate Updates

Threads safely add discovered URLs to queue and update shared state

Detect Completion

Workers coordinate to detect when all URLs have been processed

Key Takeaway

🎯 Key Insight: Multi-threading dramatically improves crawling performance by processing multiple URLs concurrently while using proper synchronization to prevent race conditions and ensure correctness.

Time & Space Complexity

Time Complexity

⏱️

O((V + E) / T)

V URLs and E links divided by T threads, assuming good load distribution and no bottlenecks

✓ Linear Growth

Space Complexity

O(V + T)

Space for visited set (V URLs) plus thread stacks and synchronization structures (T threads)

✓ Linear Space

Constraints

1 ≤ urls.length ≤ 1000
1 ≤ urls[i].length ≤ 300
startUrl is one of the urls
All URLs follow the format: http://hostname/path
HtmlParser.getUrls(url) returns URLs within 15ms
Single-threaded solutions will exceed time limit

Asked in

G Google 45 a Amazon 38 f Meta 32 ⊞ Microsoft 28

The optimal solution uses a thread pool with concurrent data structures. Create multiple worker threads that process URLs from a shared queue, using thread-safe visited set and proper synchronization. Key insights: concurrent queue operations, hostname filtering, and worker coordination for completion detection.

Common Approaches

Approach	Time	Space	Notes
✓ Single-Threaded BFS	O(V + E)	O(V)	Sequential breadth-first crawling using a simple queue
Multi-Threaded with Thread Pool (Optimal)	O((V + E) / T)	O(V + T)	Use thread pool with concurrent queue and thread-safe visited set

Single-Threaded BFS — Algorithm Steps

Extract hostname from startUrl
Initialize queue with startUrl and visited set
While queue is not empty, dequeue a URL
Call HtmlParser.getUrls() to get all links
Filter links to same hostname and not yet visited
Add new URLs to queue and mark as visited

Visualization

Tap to expand

Step-by-Step Walkthrough

Start with initial URL

Add startUrl to queue and visited set

Process URL sequentially

Dequeue URL, fetch its links, filter by hostname

Add new URLs

Add unvisited URLs to queue for later processing

Repeat until done

Continue until queue is empty

Code -

solution.c — C

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define MAX_URLS 1000
#define MAX_URL_LEN 256

char* getHostname(const char* url) {
    // Simple hostname extraction for http://hostname/path
    const char* start = strstr(url, "://");
    if (!start) return NULL;
    start += 3;
    
    const char* end = strchr(start, '/');
    if (!end) end = start + strlen(start);
    
    int len = end - start;
    char* hostname = malloc(len + 1);
    strncpy(hostname, start, len);
    hostname[len] = '\0';
    return hostname;
}

char** crawl(char* startUrl, HtmlParser* htmlParser, int* returnSize) {
    char* hostname = getHostname(startUrl);
    char** result = malloc(MAX_URLS * sizeof(char*));
    char visited[MAX_URLS][MAX_URL_LEN];
    char queue[MAX_URLS][MAX_URL_LEN];
    int visitedCount = 0, queueSize = 1, resultSize = 0;
    
    strcpy(queue[0], startUrl);
    strcpy(visited[0], startUrl);
    visitedCount = 1;
    
    while (queueSize > 0) {
        char currentUrl[MAX_URL_LEN];
        strcpy(currentUrl, queue[0]);
        
        // Shift queue
        for (int i = 0; i < queueSize - 1; i++) {
            strcpy(queue[i], queue[i + 1]);
        }
        queueSize--;
        
        result[resultSize] = malloc(strlen(currentUrl) + 1);
        strcpy(result[resultSize], currentUrl);
        resultSize++;
        
        // Process URLs (simplified)
        // In real implementation, would call htmlParser->getUrls
    }
    
    *returnSize = resultSize;
    free(hostname);
    return result;
}

Time & Space Complexity

Time Complexity

⏱️

O(V + E)

V is number of URLs, E is number of links. Each URL is processed once, each link examined once.

✓ Linear Growth

Space Complexity

O(V)

Space for visited set and queue to store all URLs

✓ Linear Space

Constraints

1 ≤ urls.length ≤ 1000
1 ≤ urls[i].length ≤ 300
startUrl is one of the urls
All URLs follow the format: http://hostname/path
HtmlParser.getUrls(url) returns URLs within 15ms
Single-threaded solutions will exceed time limit

58.0K Views

High Frequency

~25 min Avg. Time

1.7K Likes

Ln 1, Col 1

Smart Actions

💡 Explanation

AI Ready

💡 Suggestion Tab to accept Esc to dismiss

// Output will appear here after running code

Code Editor Closed

Click the red button to reopen

Input & Output

Visualization

Time & Space Complexity

Related Problems

Constraints

Common Approaches

Single-Threaded BFS — Algorithm Steps

Visualization

Code -

Time & Space Complexity

Constraints

Select Compiler