How to use Boto3 to get the details of multiple glue jobs at a time?

This article demonstrates how to retrieve detailed metadata for multiple AWS Glue jobs simultaneously using the boto3 library. We'll use the list_jobs() and batch_get_jobs() methods to efficiently fetch job information.

Problem Statement

Use boto3 library in Python to get comprehensive details of all Glue jobs available in your AWS account, including their configurations, roles, and execution properties.

Step-by-Step Approach

Step 1 ? Import boto3 and botocore exceptions to handle potential errors.

Step 2 ? Create an AWS session and Glue client with proper region configuration.

Step 3 ? Use list_jobs() to retrieve all job names from your account.

Step 4 ? Call batch_get_jobs() with the retrieved job names to get detailed metadata.

Step 5 ? Handle exceptions appropriately for robust error management.

Implementation

Here's the complete code to fetch details of all Glue jobs in your account ?

import boto3
from botocore.exceptions import ClientError

def get_resource_metadata_of_glue_jobs():
    session = boto3.session.Session()
    glue_client = session.client('glue')
    
    try:
        # Get list of all job names
        job_list = glue_client.list_jobs()
        
        # Get detailed metadata for all jobs
        job_details = glue_client.batch_get_jobs(JobNames=job_list['JobNames'])
        
        return job_list, job_details
        
    except ClientError as e:
        raise Exception("boto3 client error in get_resource_metadata_of_glue_jobs: " + str(e))
    except Exception as e:
        raise Exception("Unexpected error in get_resource_metadata_of_glue_jobs: " + str(e))

# Execute the function
job_names, job_metadata = get_resource_metadata_of_glue_jobs()

# Display results
print("List of Jobs:")
print(job_names)
print("\nDetailed Job Metadata:")
print(job_metadata)

Understanding the Output

The function returns two dictionaries ?

Job Names List: Contains job names and pagination token for large result sets.

Job Metadata: Contains detailed information including:

• Name ? Job identifier
• Role ? IAM role ARN used by the job
• Command ? Script location and execution details
• ExecutionProperty ? Concurrency settings
• MaxRetries ? Retry configuration
• Timeout ? Maximum execution time
• GlueVersion ? AWS Glue version being used

Key Features

Conclusion

Using batch_get_jobs() with list_jobs() provides an efficient way to retrieve comprehensive metadata for all Glue jobs. This approach handles multiple jobs in a single API call, making it ideal for monitoring and management tasks.