AWS Glue - Introduction

---

---

AWS Glue is a fully managed serverless data integration cloud service from Amazon Web Services (AWS). It is designed to help users prepare and transform data for analytics, machine learning, and application development. With AWS Glue, you can connect with more than 70 diverse data sources and manage your data in a centralized data catalog.

As a serverless data integration service, AWS Glue automates much of the work associated with ETL (Extract, Transform, Load) processes. It simplifies the extraction, cleaning, enrichment, and movement of data between various sources and destinations.

AWS Glue also integrates very easily with other AWS services like Amazon S3, RDS, Redshift, and Athena. This feature makes it a perfect choice for organizations who want to build data lakes or data warehouses.

Key Components of AWS Glue

The key components of AWS Glue are described below −

Glue Data Catalog

The Glue Data Catalog is a central repository that stores metadata information about your data. It automatically scans and organizes the data so that the user can easily search, query, and manage datasets. It also connects well with AWS tools like Redshift and Athena, allowing user to smoothly access the data.

Crawlers

ETL jobs that we define in AWS Glue use Data Catalog tables as sources and targets. These data catalog tables should be updated always.

The role of Crawlers in AWS Glue is to automatically discover new data, identify its schema, and update the Data Catalog accordingly. They ensure that the metadata is always up to date.

Glue Jobs

Glue Jobs is used to define and manage the ETL workflows. They extract data, transform it using Apache Spark, and load it into target systems. You can run jobs on-demand or schedule them to run at specified intervals. Glue Jobs are the core of the data transformation process.

Triggers

With the help of Triggers users can automate job execution based on a schedule or specific event. The use of triggers is helpful for automating repetitive tasks or for building complex data pipelines.

Job Notebooks

AWS Glue provides IDE (interactive development environment) using Jupyter Notebooks. You can run queries, analyze data, and develop Glue Jobs interactively.

Glue Studio

As the name implies, the Glue Studio is a visual interface for creating, running, and monitoring ETL workflows without writing code. It is useful for non-technical users or for those who are not familiar with Apache Spark.

Features of AWS Glue

We can divide the important features of AWS Glue into following three categories −

Discover and Organize Data

AWS Glue enables you to organize metadata in a structured way so that you can easily store, search, and manage all the data in one place.

AWS Glue crawlers automatically discover the data and integrate it into your Data Catalog. It validates and controls access to your databases and tables.

Transform, Prepare, and Clean Data for Analysis

You can define your ETL process in Glue studio and it automatically generates code for that process. The Job Notebooks of AWS Glue provide serverless notebooks that require minimal setup. Using these notebooks, you can start working on your project quickly.

AWS Glue has the feature of sensitive data detection which allows you to define, identify, and process sensitive data in your data lake and pipeline. AWS Glue allows users to interactively explore and prepare data.

Build and Monitor Data Pipelines

You can automate jobs using Crawlers or AWS Glue jobs with event-based triggers. It allows you to run the jobs with your choice of engine, Apache Spark or Ray.

You can organize and manage ETL processes and integration activities for different crawlers, jobs, and triggers.