How to use Boto3 to get the list of workflows present an in AWS account

In this article, we will see how to use Boto3 to get the list of all workflows present in an AWS account using AWS Glue service.

AWS Glue workflows help orchestrate ETL jobs and crawlers. The list_workflows() method retrieves all workflows from the AWS Glue Data Catalog.

Prerequisites

Before running this code, ensure you have ?

• AWS credentials configured (via AWS CLI, environment variables, or IAM roles)

• Boto3 library installed: pip install boto3

• Appropriate permissions to access AWS Glue workflows

Approach

The solution involves the following steps ?

• Step 1: Import boto3 and botocore exceptions to handle errors

• Step 2: Create an AWS session using boto3

• Step 3: Create an AWS Glue client

• Step 4: Use list_workflows() method to fetch all workflows

• Step 5: Handle exceptions appropriately

Example

The following code demonstrates how to fetch all workflows from AWS Glue ?

import boto3
from botocore.exceptions import ClientError

def list_of_workflows():
    session = boto3.session.Session()
    glue_client = session.client('glue')
    try:
        workflows = glue_client.list_workflows()
        return workflows
    except ClientError as e:
        raise Exception("boto3 client error in list_of_workflows: " + str(e))
    except Exception as e:
        raise Exception("Unexpected error in list_of_workflows: " + str(e))

# Call the function and print results
result = list_of_workflows()
print("Workflows found:", result['Workflows'])
print("Total workflows:", len(result['Workflows']))

Output

Workflows found: ['tick-data-etl', 'test-wf-daily-jobs']
Total workflows: 2

Response Structure

The list_workflows() method returns a dictionary containing ?

• Workflows: List of workflow names

• ResponseMetadata: AWS response metadata including RequestId and HTTP status

Error Handling

The code handles two types of exceptions ?

• ClientError: Boto3 specific errors (authentication, permissions, etc.)

• Generic Exception: Any other unexpected errors

Additional Options

The list_workflows() method supports pagination for large numbers of workflows ?

# For paginated results
def list_workflows_paginated():
    session = boto3.session.Session()
    glue_client = session.client('glue')
    
    workflows = []
    paginator = glue_client.get_paginator('list_workflows')
    
    for page in paginator.paginate():
        workflows.extend(page['Workflows'])
    
    return workflows

Conclusion

Using Boto3's list_workflows() method provides a straightforward way to retrieve all AWS Glue workflows in your account. Always implement proper error handling and consider pagination for large datasets.