Azure Data Factory
This article outlines the integration with the Azure Data Factory connector, enabling streamlined metadata management through features such as crawling and lineage building (both automatic and manual).
The connector connects to Azure Data Factory through REST APIs. Authentication uses Client ID, Client Secret, and Tenant ID to authenticate the application and establish access to Azure Data Factory metadata, with Subscription ID and Resource Group Name used to identify the Azure resources.
Overview
Connector Details
Connector Category
ETL
OvalEdge Release Supported
Release5.1 and later
Connectivity
[How the connection is established with Azure Data Factory]
REST APIs
Verified Azure Data Factory Version
V2
Connector Features
Crawling
✅
Profiling
NA
Query Sheet
NA
Data Preview
NA
Auto Lineage
✅ To learn more, click here.
Manual Lineage
✅
Secure Authentication via Credential Manager
✅
Data Quality
❌
DAM (Data Access Management)
❌
Bridge
✅
Metadata Mapping
The following objects are crawled from Azure Data Factory and mapped to the corresponding UI assets.
Schemas
ADF_Schema
Note: its a static name provided in ovaledge not a real schema
Schema
Schema
Activities
Dataset Name
Datasets
datasets
datasets
Activities
Pipelines
Pipelines
Pipelines
Pipelines
Activities
Dataflows
Dataflows
Dataflows
Dataflows
Activities
PipelineRuns
PipelineRuns
PipelineRuns
PipelineRuns
Activities
PipelineRunParams
PipelineRunParams
PipelineRunParams
PipelineRunParams
Activities
Static params
Static params
Static params
Static params
Activities
Global Params
Global Params
Global Params
Global Params
Activities
Linked services
Linked services
Linked services
Linked services
Activities
Activity Datasets
Activity Datasets
Activity Datasets
Activity Datasets
Set up a Connection
Prerequisites
The following are the prerequisites to establish a connection:
Service Account User Permissions
It is recommended to use a separate service account to establish the connection to the data source, configured with the following minimum set of permissions.
Crawling
Datasets
Read
Crawling
Pipelines
Read
Crawling
Dataflows
Read
Crawling
Linkedservices
Read
Connection Configuration Steps
Users are required to have the Connector Creator role in order to configure a new connection.
Log into OvalEdge, go to Administration > Connectors, click + (New Connector), search for Azure Data Factory, and complete the required parameters.
Connector Type
By default, "Azure Data Factory" is displayed as the selected connector type.
Credential Manager*
Select the desired credentials manager from the drop-down list. Relevant parameters will be displayed based on the selection.
Supported Credential Managers:
OE Credential Manager
AWS Secrets Manager
HashiCorp
Azure Key Vault
For more details, click here.
License Add Ons
Select the checkbox for the Auto Lineage Add-On to build data lineage automatically. For more details, click here.
Connector Name*
Enter a unique name for the Azure Data Factory connection
(Example: "AzureDataFactorydb").
Connector Description
Enter a brief description of the connector.
Connector Environment
Select the environment (Example: PROD, STG) configured for the connector. For more details, click here.
Client Id*
Enter the Application ID (also known as Client ID) of the registered application. This is a unique identifier for the app in Azure.
Client Secret*
Enter the client secret generated for the application. This is utilized to authenticate the application's identity during the token request process.
Tenant Id*
Enter the Tenant ID of the Azure Active Directory. This is used for authentication with the Azure AD tenant.
Subscriber Id*
Enter the Subscription ID for the Azure account. This is a unique identifier for the Azure subscription.
Resource Group Name*
Enter the name of the Resource Group. The name should match the project or deployment requirements.
API Version*
Enter the API version.
Pipeline Types to crawl
Select the pipeline type from the drop-down list.
Proxy Enabled*
Select Yes to route the connection through a configured proxy, or No to connect directly without a proxy.
Default Governance Roles
Default Governance Roles*
Select the appropriate users or teams for each governance role from the drop-down list. All users configured in the security settings are available for selection.
Admin Roles
Admin Roles*
Select one or more users from the dropdown list for Integration Admin and Security & Governance Admin. All users configured in the security settings are available for selection.
No of Archive Objects
No Of Archive Objects*
This shows the number of recent metadata changes to a dataset at the source. By default, it is off. To enable it, toggle the Archive button and specify the number of objects to archive.
Example: Setting it to 4 retrieves the last four changes, displayed in the 'Version' column of the 'Metadata Changes' module.
Bridge
Select Bridge*
If applicable, select the bridge from the drop-down list.
The drop-down list displays all active bridges that have been configured. These bridges facilitate communication between data sources and the system without requiring changes to firewall rules.
After entering all connection details, the following actions can be performed:
Click Validate to verify the connection.
Click Save to store the connection for future use.
Click Save & Configure to apply additional settings before saving.
The saved connection will appear on the Connectors home page.
Manage Connector Operations
Crawl
The Crawl/Profile button allows users to select one or more schemas for crawling.
Navigate to the Connectors page and click Crawl/Profile.
Select the schemas to be crawled.
The Crawl option is selected by default.
Click Run to collect metadata from the connected source and load it into the Data Catalog.
After a successful crawl, the information appears in the Data Catalog > Databases/<>Codes tab.
The Schedule checkbox allows automated crawling at defined intervals, from a minute to a year.
Click the Schedule checkbox to enable the Select Period drop-down.
Select a time period for the operation from the drop-down menu.
Click Schedule to initiate metadata collection from the connected source.
The system will automatically execute the crawl operation at the scheduled time.
Other Operations
The Connectors page provides a centralized view of all configured connectors, along with their health status.
Managing connectors includes:
Connector Health: Displays the current status of each connector using a green icon for active connections and a red icon for inactive connections, helping to monitor connectivity with data sources.
Viewing: Click the Eye icon next to the connector name to view connector details.
Nine Dots Menu Options:
To view, edit, validate, build lineage, configure, or delete connectors, click on the Nine Dots menu.
Edit Connector: Update and revalidate the data source.
Validate Connector: Check the connection's integrity.
Settings: Modify connector settings.
Query Policies: Define query execution rules based on roles.
Connection Pooling: Configure database connection pool parameters to optimize database connectivity and resource utilization. For more details, click here.
Lineage: Configure Server Dialects for source code parsing and Connector Priority for table lineage connection. For more details, click here.
Build Lineage: Automatically build data lineage using source code parsing.
Delete Connector: Remove a connector with confirmation.
For more details on connector settings, click here.
Troubleshooting
If incorrect parameters are entered, error messages may appear. Ensure all inputs are accurate to resolve these issues. If issues persist, contact the assigned support team.
1
Error while validating connection: authentication failed or invalid credentials
The Client ID, Secret ID, or Tenant ID entered does not match the values registered in the Azure portal for the application registration.
Resolution:
Confirm the Client ID, Secret ID, and Tenant ID exactly match the values shown in the Azure application registration.
Confirm the secret key linked to the application registration is still active and has not expired.
Retry validation after updating the credentials.
2
Validation successful but factory list returned is empty
Validation can complete successfully even when the Subscription ID or Resource Group Name entered does not correctly point to the intended data factory.
Resolution:
Confirm the Subscription ID matches the subscription that actually holds the data factory.
Confirm the Resource Group Name matches the resource group containing the data factory.
Confirm the reader role has been granted to the service principal at the subscription or resource group level.
3
Validation successful but no pipelines, datasets, or linked services are catalogued
The service principal used for the connection may not belong to the correct tenant associated with the data factory, even though the credentials themselves are valid.
Resolution:
Confirm the service principal belongs to the correct tenant linked to the data factory.
Confirm the reader role assigned covers the correct scope, either the subscription or the resource group.
Involve an Azure administrator to verify the application registration scope if the mismatch continues.
4
Certain factories, pipelines, or datasets do not appear after crawling
This typically occurs when the scope assigned to the credentials does not extend to the workspace containing the missing objects, or a proxy/bridge path is limiting visibility.
Resolution:
Confirm the scope assigned to the credentials includes the workspace holding the missing objects.
Confirm the proxy or bridge path in use is not restricting visibility to certain factories.
Verify with an Azure administrator that reader access extends to all required pipelines and datasets.
1
Pipelines are catalogued, but no pipeline run information is available
Run-based metadata is only collected for successful triggered runs that occurred within the configured historic run window, which defaults to 30 days. Debug runs are not retrieved by the ADF API.
Resolution:
Confirm the pipeline has at least one successful triggered run within the configured historic run window.
If runs are available but are debug runs, trigger the pipeline through an ADF trigger and verify that a successful triggered run is available.
Adjust the historic run length setting if the relevant triggered runs are older than the default 30-day window.
Confirm the pipeline contains a copy activity, since run-based metadata is collected only for pipelines that include one.
2
Trigger information is not visible in the catalog
Trigger objects are not currently supported for crawling as part of the Azure Data Factory connector.
Resolution:
Note that trigger crawling is not a supported capability at this time.
Rely on pipeline and pipeline run information instead, as no configuration change enables trigger crawling.
Continue monitoring release documentation for future support of this capability.
3
Older API version fields do not return expected results
The connector supports only the latest officially published Azure Data Factory API version. Deprecated versions are not supported.
Resolution:
Confirm the API version field is set to the latest version referenced in the official Azure documentation.
Update the API version field if an older, deprecated version is currently configured.
Reach out to the standard support channel if the required version is unclear.
1
Lineage graph is empty or incomplete for a pipeline
Lineage depends on the pipeline containing a copy activity with at least one successful run, and on both the source and target systems already being catalogued.
Resolution:
Confirm the source and target systems referenced by the pipeline are already catalogued.
Confirm the pipeline contains a copy activity, as this activity establishes the source and target mapping.
Confirm the copy activity has at least one successful run within the configured historic run window.
2
Crawl or lineage build works intermittently after previously functioning correctly
This is commonly related to secret rotation or expiry on the application registration used for the connection.
Resolution:
Confirm the secret associated with the application registration has not expired.
Revalidate the connection once the credential is confirmed to be active.
Retry the crawl or lineage build job after revalidation is successful.
FAQs
1
What details are needed to set up the Azure data factory connector?
The following details are required to establish a connection:
Client ID
Client Secret
Tenant ID
Subscription ID
Resource Group Name
The Client ID, Client Secret, and Tenant ID are obtained from an application registration in the Azure portal. These details are used to generate a bearer token, which allows the connector to access the data factory and its objects, including pipelines, linked services, datasets, and data flows.
2
Why does a connection validate successfully but show zero factories or pipelines after crawling?
Connection validation only confirms that the credentials are valid and Azure is reachable. It does not verify that the connection points to the correct data factory.
Verify that:
Subscription ID matches the subscription containing the data factory.
Resource Group Name matches the resource group containing the data factory.
These values can be verified in the Azure portal.
3
What level of permission is required for this connector?
The reader role is the minimum required permission.
The role can be assigned at either:
Subscription level, or
Resource group level containing the data factory.
The connector only reads metadata, such as pipeline definitions, linked service details, and run history. It does not modify or write data to Azure. Therefore, a contributor or writer role is not required.
4
Why might a connection that worked previously start failing or behave inconsistently?
The most common cause is an expired or changed Client Secret associated with the application registration.
If crawling or lineage previously worked and has started failing:
Verify that the Client Secret is still active in the Azure portal.
Update the Client Secret if required.
Revalidate the connection.
5
Can one connector instance cover multiple subscriptions or resource groups?
No. Each connector instance is scoped to one Subscription ID and one Resource Group Name.
If data factories are located across multiple subscriptions or resource groups, configure a separate connector instance for each one.
6
Is a bridge deployment supported for isolated network environments?
Yes. Bridge deployment is supported for this connector.
As the connector is a cloud-based, REST API driven connector, bridge deployment can be used for environments without direct outbound access.
1
What kind of objects are catalogued when Azure data factory is crawled?
A single Azure data factory schema is created with the following objects:
Azure data factory
Pipelines
Linked services
Datasets
Data flows
Global parameters
Pipeline run details
These are the primary objects collected during a standard crawl.
2
Are triggers included when Azure data factory is crawled?
No. Trigger objects are not currently catalogued by this connector.
Although Azure data factory supports triggers for pipeline scheduling, cataloguing trigger definitions is not currently supported.
3
Why does a pipeline appear in the catalog without run history?
Run history is collected only for successful runs within the configured historic run window, which defaults to the most recent 30 days.
Run history is not shown if:
The pipeline has no successful run within the configured window.
The pipeline has never had a copy activity run.
The historic run window can be adjusted in the system settings.
4
Does this connector support delta crawling?
No. Delta crawling is not currently supported.
Each crawl processes the full set of selected objects rather than identifying only objects changed since the previous crawl. Therefore, a complete recrawl is required when refreshed data is needed.
5
Which Azure API version does the connector use? Are older versions supported?
The connector uses the latest officially published Azure data factory API version specified in Azure's documentation.
Older API versions that are no longer actively published or supported by Microsoft are not maintained by this connector. Therefore, only the current supported API version should be expected to work.
6
Does the Azure data factory connector support profiling or previewing sample data?
No. The connector does not support:
Profiling
Sample data preview
Data quality checks
The Azure data factory connector is an ETL and orchestration connector rather than a direct data storage connector. It supports metadata such as object structure, pipeline definitions, and lineage relationships.
1
What needs to be in place for lineage to be built successfully for a pipeline?
The following three conditions must be met:
The pipeline must contain a copy activity to map the source to the target.
The copy activity must have at least one successful run within the configured historic run window.
Both the source system and target system must already be catalogued separately.
If any of these conditions are not met, lineage may not be created.
2
Why can a lineage graph look incomplete even though the crawl completed successfully?
A successful crawl only confirms that the pipeline objects were read successfully. It does not guarantee that lineage can be built.
Check the following:
The pipeline contains a copy activity.
The copy activity has a successful run within the configured historic run window.
The source system and target system are already catalogued.
These conditions determine whether lineage can be created successfully.
3
Does the connector capture actual runtime values, such as dynamic parameters?
No. The connector captures only the static values defined in the pipeline's JSON definition at the time of crawling.
Dynamic parameters set or changed at runtime are not captured because the connector reads the stored pipeline definition rather than individual execution values.
Copyright © 2026, OvalEdge LLC, Peachtree Corners, GA, USA.
Last updated
Was this helpful?