Azure Data Engineer Course | Azure Data Engineer Training Hyderabad

1. What is Azure Data Factory architecture? +91-9989971070 www.visualpath.in

2. Azure Data Factory architecture • Azure Data Factory is a cloud-based data integration service on Microsoft Azure that allows you to create data-driven workflows for orchestrating and automating data movement and data transformation. The architecture of Azure Data Factory involves several key components and concepts. Let's explore the main elements of the Azure Data Factory architecture: www.visualpath.in

3. 1. Azure Data Factory Components: • a. Data Pipelines: • A data pipeline is a logical grouping of activities that together perform a task. Activities within a pipeline can include data movement, data transformation, and data orchestration. • b. Activities: • Activities are the processing steps within a pipeline. Examples include copying data from one source to another, transforming data using Azure HDInsight, executing a stored procedure in Azure SQL Database, etc. www.visualpath.in

4. c. Datasets: • Datasets represent the input and output data structures within Azure Data Factory. They define the structure of the data to be processed by activities. • d. Linked Services: • Linked Services define the connection information to external data sources or destinations. They store the information needed to connect to different data stores or compute services. www.visualpath.in

5. 2. Azure Data Factory Architecture: • a. Control Flow: • Control Flow is responsible for orchestrating the execution of activities within a pipeline. It defines the sequence and conditions for the execution of activities. • b. Data Flow: • Data Flow defines the data transformation logic within a pipeline. It includes activities like data cleaning, aggregation, and transformation using Azure Data Flow or other computing services. www.visualpath.in

6. c. Integration Runtimes: • Integration Runtimes define the compute infrastructure where data movement and data transformation activities are executed. Azure Data Factory supports four types of integration runtimes: • Azure: • Data movement activities run on Azure compute resources. • Self-hosted: • Data movement activities run on an on-premises network or in a virtual network. www.visualpath.in

7. Azure-SSIS: • For running SQL Server Integration Services (SSIS) packages in Azure Data Factory. • Azure Synapse Analytics (formerly SQL Data Warehouse): • For running data transformation activities using Azure Synapse Analytics. • d. Triggers: • Triggers define when a pipeline should be executed. There are different types of triggers, including: www.visualpath.in

8. Schedule Triggers: Based on a specified schedule (e.g., hourly, daily). • Event-based Triggers: Triggered by an event, such as a new file arriving in Azure Storage. • e. Pipeline Execution: • When a trigger is activated, the Azure Data Factory service manages the execution of the pipeline. The execution involves the allocation of the necessary resources and the orchestration of activities. www.visualpath.in

9. f. Monitoring and Management: • Azure Data Factory provides monitoring and management capabilities through the Azure Portal, Azure Monitor, and Azure PowerShell. • 3. Data Movement and Transformation: • a. Data Movement Activities: • Azure Data Factory supports various data movement activities for copying data between different data stores. This includes Azure Blob Storage, Azure SQL Database, Azure Data Lake Storage, on-premises databases, and more. www.visualpath.in

10. b. Data Transformation Activities: • Data transformation activities can be performed using Azure Data Flow, which allows for the visual design of data transformation logic using a code-free interface. Additionally, other compute services like Azure HDInsight, Azure Machine Learning, or Azure Synapse Analytics can be used for complex transformations. • 4. Security and Authentication: • a. Azure Active Directory (AAD): • Azure Data Factory integrates with Azure Active Directory for authentication and access control. Users and services need appropriate permissions to access data stores and perform data operations. www.visualpath.in

11. 5. Data Store Integration: • a. Supported Data Stores: • Azure Data Factory supports a wide range of data stores, including Azure Blob Storage, Azure SQL Database, Azure Data Lake Storage, Azure Cosmos DB, on-premises SQL Server, and more. • b. Linked Services: • Linked Services are used to define the connection details and authentication information for different data stores. • 6. Error Handling and Logging: www.visualpath.in

12. a. Monitoring and Logging: • Azure Data Factory provides monitoring capabilities through Azure Monitor and logging through Azure Monitor Logs. It allows users to track pipeline runs, and activity runs, and diagnose issues. • This architecture allows Azure Data Factory to efficiently handle various data integration scenarios, from simple data movement to complex data transformations, across a diverse range of data sources and destinations. The flexibility and scalability of Azure Data Factory make it suitable for building end-to-end data workflows in the cloud. www.visualpath.in

13. CONTACT For More Information About Azure Data Engineer Training Address:- Flat no: 205, 2nd Floor NilagiriBlock, Aditya Enclave, Ameerpet, Hyderabad-16 Ph No : +91-9989971070 Visit : www.visualpath.in E-Mail : online@visualpath.in

14. THANK YOU Visit: www.visualpath.in