Beginning Azure Data Factory - Build Data Ingestion

The main objective of this course is to help you to learn Data Engineering techniques of building Metadata-Driven frameworks with Azure Data Engineering tools such as Data Factory, Azure SQL, and others.

Building Frameworks are now an industry norm and it has become an important skill to know how to visualize, design, plan and implement data frameworks.

The framework that we are going to build together is referred to as the Metadata-Driven Ingestion Framework. 

Data ingestion into the data lake from the disparate source systems is a key requirement for a company that aspires to be data-driven, and finding a common way to ingest the data is a desirable and necessary requirement. 

Metadata-Driven Frameworks allow a company to develop the system just once and it can be adopted and reused by various business clusters without the need for additional development, thus saving the business time and costs. Think of it as a plug-and-play system.

The objective of the course is to onboard you onto the Azure Data Factory platform to help you assemble your first Azure Data Factory Pipeline. Once you get a good grip of the Azure Data Factory development pattern, then it becomes easier to adopt the same pattern to onboard other sources and data sinks.

Once you are comfortable with building a basic azure data factory pipeline,  as a second objective we then move on to building a fully-fledged and working metadata-driven framework to make the ingestion more dynamic. Furthermore, we will build the framework in such a way that you can audit every batch orchestration and individual pipeline runs for business intelligence and operational monitoring.