How to use Boto3 to get the workflows that are created in your account?

AWS Glue workflows help orchestrate ETL jobs and crawlers. Using Boto3, you can retrieve all workflows in your AWS account along with their detailed metadata. This tutorial shows how to use list_workflows() and batch_get_workflows() functions.

Algorithm

Step 1 − Import boto3 and botocore exceptions to handle exceptions.

Step 2 − Create an AWS session using boto3 library. Make sure region_name is mentioned in default profile.

Step 3 − Create an AWS client for Glue service.

Step 4 − Use list_workflows() function to get all workflow names in your account.

Step 5 − Call batch_get_workflows() and pass the workflow names to get detailed metadata.

Step 6 − Handle exceptions for any errors during the operation.

Example

Use the following code to fetch details of each workflow created in your account ?

import boto3
from botocore.exceptions import ClientError

def get_resource_metadata_of_workflows():
    session = boto3.session.Session()
    glue_client = session.client('glue')
    try:
        list_of_workflows = glue_client.list_workflows()
        response = glue_client.batch_get_workflows(Names=list_of_workflows['Workflows'])
        
        return list_of_workflows, response
    except ClientError as e:
        raise Exception("boto3 client error in get_resource_metadata_of_workflows: " + str(e))
    except Exception as e:
        raise Exception("Unexpected error in get_resource_metadata_of_workflows: " + str(e))

# Get workflows data
workflows_list, workflows_metadata = get_resource_metadata_of_workflows()

# Display list of workflows
print("List of Workflows:")
print(workflows_list)

# Display resource metadata of each workflow
print("\nResource metadata of each Workflow:")
print(workflows_metadata)

Output

List of Workflows:
{'Workflows': ['dev-aiml-naviga-ods-load'], 'ResponseMetadata': 
{'RequestId': '556890ce-bcd1-4bb0-9c33-3ae1db13f3a9', 'HTTPStatusCode': 
200, 'HTTPHeaders': {'date': 'Sat, 27 Feb 2021 13:57:37 GMT', 'content-type': 'application/x-amz-json-1.1', 'content-length': '91', 
'connection': 'keep-alive', 'x-amzn-requestid': '556890ce-bcd1-4bb0-
9c33-3ae1db13f3a9'}, 'RetryAttempts': 0}}

Resource metadata of each Workflow:
{'Workflows': [{'Name': 'dev-aiml-naviga-ods-load', 
'DefaultRunProperties': {}, 'CreatedOn': datetime.datetime(2020, 5, 27, 
3, 10, 57, 967000, tzinfo=tzlocal()), 'LastModifiedOn': 
datetime.datetime(2020, 5, 27, 3, 10, 57, 967000, tzinfo=tzlocal()), 
'StartedOn': datetime.datetime(2021, 2, 3, 16, 14, 48, 795000, 
tzinfo=tzlocal()), 'CompletedOn': datetime.datetime(2021, 2, 3, 16, 28, 
6, 207000, tzinfo=tzlocal()), 'Status': 'COMPLETED', 'Statistics': 
{'TotalActions': 3, 'TimeoutActions': 0, 'FailedActions': 0, 
'StoppedActions': 0, 'SucceededActions': 3, 'RunningActions': 0}}}, 
'MissingWorkflows': [], 'ResponseMetadata': {'RequestId': 'b328d064-
24ab-48c4-b058-852387a3d474', 'HTTPStatusCode': 200, 'HTTPHeaders': 
{'date': 'Sat, 27 Feb 2021 13:57:37 GMT', 'content-type': 
'application/x-amz-json-1.1', 'content-length': '2655', 'connection': 
'keep-alive', 'x-amzn-requestid': 'b328d064-24ab-48c4-b058-
852387a3d474'}, 'RetryAttempts': 0}}

Key Points

The response contains workflow details including:

• Name − Workflow identifier
• Status − Current workflow status (COMPLETED, RUNNING, etc.)
• Statistics − Action counts (total, succeeded, failed)
• CreatedOn/LastModifiedOn − Timestamps for workflow lifecycle

Conclusion

Use list_workflows() to get workflow names and batch_get_workflows() for detailed metadata. This combination provides complete visibility into your AWS Glue workflows and their execution status.