Data Structure
Networking
RDBMS
Operating System
Java
MS Excel
iOS
HTML
CSS
Android
Python
C Programming
C++
C#
MongoDB
MySQL
Javascript
PHP
Physics
Chemistry
Biology
Mathematics
English
Economics
Psychology
Social Studies
Fashion Studies
Legal Studies
- Selected Reading
- UPSC IAS Exams Notes
- Developer's Best Practices
- Questions and Answers
- Effective Resume Writing
- HR Interview Questions
- Computer Glossary
- Who is Who
What is ETL?
ETL stands for Extract, transform, and load. It is the process data-driven organizations use to gather data from multiple sources and then bring it together to support discovery, reporting, analysis, and decision-making.
The data sources can be divergent in type, format, volume, and reliability, hence the data required to be processed to be helpful when delivered together. The target data stores can be databases, data warehouses, or data lakes, based on the objectives and technical execution. There are the following steps of ETL which are as follows −
Extract − During extraction, ETL recognizes the data and duplicate it from its sources, therefore it can transport the data to the target datastore. The data can appear from structured and unstructured sources, including files, emails, business software, databases, equipment, sensors, third parties, etc.
There are different ways to perform the extraction such as
Partial Extraction − The simplest way to access the information is if the source system notifies us when a record has been modified.
Partial Extraction (with update notification) − It is not all systems can provide an announcement in case an update has taken place; but, they can mark to those records that have been transformed and support an extract of such records.
Full extract − Certain systems cannot identify which data has been changed at all. In this case, a full extract is the only feasibility to extract the record out of the system. This approach needed having a copy of the final extract in the same format so it can identify the changes that have been created.
Transform − The second step includes transforming the raw information that has been extracted from the sources into a format that can be used by several applications. In this phase, data have cleansed, mapped, and transformed, providing to a definite schema, so it finds operational needs.
This process requires several types of transformation that provide the quality and integrity of data. Data is not generally loaded precisely into the target data source, but alternatively, it is frequent to have it uploaded into a staging database.
This step ensures a quick rollback in case something does not go as planned. During this phase, it can create audit documents for regulatory compliance, or diagnose and repair some data problems.
Load − ETL moves the transformed information into the target datastore. This step can require the original loading of all the source information, or it can be the loading of incremental changes in the source information. It can load the data in real-time or in scheduled batches.
279 Views
