How to use Boto3 to get the metrics of one/manyspecified crawler from AWS Glue Data Catalog?

Boto3 is the AWS SDK for Python that allows you to interact with AWS services. The AWS Glue Data Catalog stores metadata about your data sources, and you can retrieve crawler metrics to monitor performance and status.

Problem Statement

Use the boto3 library in Python to retrieve the metrics of one or more specified crawlers from AWS Glue Data Catalog.

Approach

Step 1 ? Import boto3 and botocore exceptions to handle errors.

Step 2 ? Define crawler_names as a list parameter containing the names of crawlers whose metrics you want to retrieve.

Step 3 ? Create an AWS session using boto3. Ensure your AWS credentials and region are properly configured.

Step 4 ? Create an AWS Glue client using the session.

Step 5 ? Use the get_crawler_metrics method with the CrawlerNameList parameter.

Step 6 ? Handle exceptions that may occur during the operation.

Single Crawler Metrics

Here's how to retrieve metrics for a single crawler ?

import boto3
from botocore.exceptions import ClientError

def get_single_crawler_metrics(crawler_name):
    """Retrieve metrics for a single crawler"""
    session = boto3.session.Session()
    glue_client = session.client('glue')
    
    try:
        response = glue_client.get_crawler_metrics(
            CrawlerNameList=[crawler_name]
        )
        return response
    except ClientError as e:
        raise Exception(f"boto3 client error in get_single_crawler_metrics: {e}")
    except Exception as e:
        raise Exception(f"Unexpected error in get_single_crawler_metrics: {e}")

# Example usage
result = get_single_crawler_metrics("my-s3-crawler")
print(result)

Multiple Crawler Metrics

To retrieve metrics for multiple crawlers at once ?

import boto3
from botocore.exceptions import ClientError

def get_multiple_crawler_metrics(crawler_names):
    """Retrieve metrics for multiple crawlers"""
    session = boto3.session.Session()
    glue_client = session.client('glue')
    
    try:
        response = glue_client.get_crawler_metrics(
            CrawlerNameList=crawler_names
        )
        return response
    except ClientError as e:
        raise Exception(f"boto3 client error: {e}")
    except Exception as e:
        raise Exception(f"Unexpected error: {e}")

# Example usage with multiple crawlers
crawler_list = ["crawler-1", "crawler-2", "data-lake-crawler"]
metrics = get_multiple_crawler_metrics(crawler_list)

# Display metrics for each crawler
for crawler_metric in metrics['CrawlerMetricsList']:
    print(f"Crawler: {crawler_metric['CrawlerName']}")
    print(f"Last Runtime: {crawler_metric['LastRuntimeSeconds']} seconds")
    print(f"Tables Created: {crawler_metric['TablesCreated']}")
    print("---")

Key Metrics Explained

Example Output

{
    'CrawlerMetricsList': [
        {
            'CrawlerName': 'my-s3-crawler',
            'TimeLeftSeconds': 0.0,
            'StillEstimating': False,
            'LastRuntimeSeconds': 79.673,
            'MedianRuntimeSeconds': 79.673,
            'TablesCreated': 1,
            'TablesUpdated': 0,
            'TablesDeleted': 0
        }
    ],
    'ResponseMetadata': {
        'RequestId': '680cf4ca-****-****-****-**********0abe',
        'HTTPStatusCode': 200
    }
}

Conclusion

Use boto3's get_crawler_metrics method to monitor AWS Glue crawler performance. This helps track runtime, table operations, and overall crawler health for data pipeline optimization.