How to use Boto3 to check the statusof all the runsof a given Glue Job?

Boto3PythonServer Side ProgrammingProgramming

Problem Statement − Use boto3 library in Python to check statuses of all runs of a given job.

Example − Get the status of all runs of a glue job named as ‘run_s3_file_job’.

Approach/Algorithm to solve this problem

Step 1 − Import boto3 and botocore exceptions to handle exceptions.

Step 2job_name is the mandatory parameters. The function will fetch the details of a given job_name.

Step 3 − Create an AWS session using boto3 library. Make sure region_name is mentioned in default profile. If it is not mentioned, then explicitly pass the region_name while creating the session.

Step 4 − Create an AWS client for glue.

Step 5 − Now use get_job_runs function and pass the job_name as JobName parameter.

Step 6 − It fetches the details of all past job runs of the given job.

Step 7 − Use for loop to get details of specific job runs one by one.

Step 8 − Now, get the specific status of the job and corresponding job run id. Status could be Running if the job is not completed, else SUCCEEDED/FAILED.

Step 9 − Handle the generic exception if something went wrong while checking the job.

Example

Use the following code to check statuses of all runs of a given job −

import boto3
from botocore.exceptions import ClientError

def get_status_of_job_all_runs(job_name):
   session = boto3.session.Session()
   glue_client = session.client('glue')
   try:
      response = glue_client.get_job_runs(JobName=job_name)
      for res in response['JobRuns']:
         print("Job Run id is:"+res.get("Id"))
         print("status is:"+res.get("JobRunState"))
   except ClientError as e:
      raise Exception("boto3 client error in get_status_of_job_all_runs: " + e.__str__())
   except Exception as e:
      raise Exception("Unexpected error in get_status_of_job_all_runs: " + e.__str__())
get_status_of_job_all_runs("run_s3_file_job"))

Output

job id
is:jr_6ef92e90ad66b1a6c7abb1c2659d114a34962b8c6ae4bf9b328ac90b99a33b7d
status is:FAILED
job id
is:jr_9fef13265036406e03e7cae79257305353203ab20b5f400e0c429e10a4999dba
status is:FAILED
job id
is:jr_f9d715a33e83460fc2ef6dee0840a98ef52c06c2ff569627633c4505fda7d835
status is:FAILED
job id
is:jr_71b57633ac4d8c24f904f0ae01f613f6d54baee440d0ede23f6030cffb0bf4d7
status is:FAILED
job id
is:jr_b7ead6b6ae43da2580888c73d6896c177510df73bd77c843d3e77b4dc5f22e2f
status is:FAILED
job id
is:jr_6e757509d51066648d49c22a47c26e728d6f842a1c5d2fd4f41941ca868460e6
status is:FAILED
job id
is:jr_89c1a7b8ea045fac36d25733d7fc657d3560eb159e7e122a8960981dd225a9d0
status is:FAILED
job id
is:jr_87522bcb924e41a6cf0294185b0a09e46a4ff4c67db5007ff7c10f959836a44b
status is:FAILED
job id
is:jr_08bb54854c8e5c60e96a3e2bade1184963973c1ea7a1e760029e740d4c4c5d7d
status is:FAILED
job id
is:jr_540a90407bb7fbde72b3e1d6cbc98c3b246c21c87d836ff389491bf95520bb41
status is:FAILED
job id
is:jr_f27227cb16ec9d3df665d8753b09c2e2d24d5b5e5263f3d3f17a9bd4669bee67
status is:FAILED
job id
is:jr_a1ae4ad2edcbdac5c948b8be92a79a37c27517113364ddc88c8a93bc712fe6c1
status is:SUCCEEDED
raja
Published on 23-Mar-2021 06:44:25
Advertisements