Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
How to use Boto3 to start the scheduler of a crawler in AWS Glue Data Catalog
In this article, we will see how a user can start the scheduler of a crawler in AWS Glue Data Catalog.
Example
Start the scheduler of a crawler available in AWS Glue Data Catalog.
Problem Statement: Use boto3 library in Python to start the scheduler of a crawler.
Approach/Algorithm to solve this problem
Step 1: Import boto3 and botocore exceptions to handle exceptions.
Step 2: crawler_name is the required parameter in this function.
Step 3: Create an AWS session using boto3 lib. Make sure region_name is mentioned in the default profile. If it is not mentioned, then explicitly pass the region_name while creating the session.
Step 4: Create an AWS client for glue.
Step 5: Now use the start_crawler_schedule function and pass the parameter crawler_name as CrawlerName.
Step 6: It returns the response metadata and sets the schedule state of the crawler as SCHEDULED. If the status of the crawler is running or the schedule state is already SCHEDULED, then it throws the exception - SchedulerRunningException.
Step 7: Handle the generic exception if something went wrong while starting the scheduler of a crawler.
Example Code
The following code starts the scheduler of a crawler −
import boto3
from botocore.exceptions import ClientError
def start_scheduler_of_a_crawler(crawler_name)
session = boto3.session.Session()
glue_client = session.client('glue')
try:
response = glue_client.start_crawler_schedule(CrawlerName=crawler_name)
return response
except ClientError as e:
raise Exception("boto3 client error in start_scheduler_of_a_crawler: " + e.__str__())
except Exception as e:
raise Exception("Unexpected error in start_scheduler_of_a_crawler: " + e.__str__())
print(start_scheduler_of_a_crawler("Data Dimension"))Output
{'ResponseMetadata': {'RequestId': '73e50130-*****************8e', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Sun, 28 Mar 2021 07:26:55 GMT', 'content-type': 'application/x-amz-json-1.1', 'content-length': '2', 'connection': 'keep-alive', 'x-amzn-requestid': '73e50130-***************8e'}, 'RetryAttempts': 0}}
149 Views
