Microsoft Fabric OneLake

This article outlines the integration with the Microsoft Fabric OneLake connector, enabling streamlined metadata management through features such as crawling, sample profiling, data preview, manual lineage, and Data Quality.

This connector establishes connectivity to Microsoft Fabric OneLake using the OneLake SDK to retrieve metadata from workspaces, Lakehouses, folders, and supported file formats. It authenticates using a Microsoft Entra ID Service Principal (Tenant ID, Client ID, and Client Secret) and connects through the configured OneLake Blob endpoint to access supported Microsoft Fabric resources.

Overview

Connector Details

Connector Category

File Connectors

OvalEdge Releases Supported

Release8.2 and later

Connectivity

[How the connection is established with Microsoft Fabric OneLake]

OneLake SDK

Connector Features

Feature

Availability

Crawling

✅

Delta Crawling

❌

Profiling*

❌

Sample Profiling

✅

Query Sheet

❌

Data Preview

✅

Auto Lineage

❌

Manual Lineage

✅

Secure Authentication via Credential Manager

✅

Data Quality

❌

DAM (Data Access Management)

❌

Bridge

✅

*Full profiling is supported through DuckDB. To enable this capability, configure the system setting (enable.duckdb) to True. When enabled, the File Data tab displays only the first 10000 rows from the source file.

Metadata Mapping

The following objects are crawled from Microsoft Fabric OneLake and mapped to the corresponding UI assets.

OneLake Object

OneLake Attribute

OvalEdge Attribute

OvalEdge Category

OvalEdge Type

File/ Folder

Workspace/Lakehouse/Folder

Folder

File/ Folder

Workspace/Lakehouse/Folder/File

File

XLSX(sheets)

File(subfile)

File

XLS(sheets)

File(subfile)

File

CSV

File

TXT

File

PARQUET

File

ORC

File

JSON

File

YAML

File

PIP

File

YML

File

AVRO

File

SQL

File

XML

File

Set up a Connection

Prerequisites

The following are the prerequisites to establish a connection:

Ensure that the CSV files follow the required formatting standards for proper data processing and visibility. Refer to CSV Format Requirements.

Service Account User Permissions

It is recommended to use a dedicated service account to establish the connection to the data source, configured with the following minimum set of permissions.

👨‍💻 Who can provide these permissions? These permissions are typically granted by the Microsoft Fabric OneLake administrator, as users may not have the required access to assign them independently.

Operation

Objects

Access Permission

Connector Validation

Fabric Workspace

Member or Contributor workspace role

Crawling

Folders

Read permission with Member or Contributor workspace role

Crawling

Files

Read permission with Member or Contributor workspace role

View Data

Files and Folders

Read permission with Member or Contributor workspace role

Enable Service principals to use Fabric APIs in the Microsoft Fabric tenant before configuring the connector.

Add the Microsoft Entra ID application to each target Microsoft Fabric workspace with the Member or Contributor role.
When using Service Principal authentication, assign the required Azure Storage permissions to the Microsoft Entra ID application.

Connection Configuration Steps

Users are required to have the Connector Creator role in order to configure a new connection.

Log in to OvalEdge, go to Administration > Connectors, click + (New Connector), search for Microsoft Fabric OneLake, and complete the required parameters.

Fields marked with an asterisk (*) are mandatory for establishing a connection.

Field Name

Description

Connector Type

By default, "Microsoft Fabric OneLake" is displayed as the selected connector type.

Credential Manager*

Select the desired credentials manager from the drop-down list. Relevant parameters will be displayed based on your selection.

Supported Credential Managers:

OE Credential Manager
AWS Secrets Manager
HashiCorp Vault
Azure Key Vault

For more details, click here.

Connector Environment

Select the environment (Example: PROD, STG) configured for the connector. For more details, click here.

Connector Name*

Enter a unique name for the Microsoft Fabric OneLake connection

(Example: "Fabric_Onelake").

Connector description

Enter a brief description of the connector.

Tenant Id*

Enter the Microsoft Entra ID Tenant ID (Directory ID) associated with the registered application. This identifier is used to authenticate the connector with Microsoft Entra ID.

Client Id*

Enter the Microsoft Entra ID Client ID (Application ID) of the registered application. This identifier uniquely identifies the application used for authentication.

Client Secret*

Enter the Client Secret associated with the registered application. This secret is used to authenticate the application and obtain access tokens from Microsoft Entra ID.

OneLake Blob endpoint*

Enter the OneLake Blob endpoint URL of the Microsoft Fabric workspace or Lakehouse. This endpoint is used to establish connectivity and access supported files and folders stored in OneLake.

Default Governance Roles

Default Governance Roles*

Select the appropriate users or teams for each governance role from the drop-down list. All users and teams configured in OvalEdge Security are displayed for selection.

Admin Roles

Admin Roles*

Select one or more users from the dropdown list for Integration Admin and Security & Governance Admin. All users configured in OvalEdge Security are available for selection.

No of Archive Objects

No Of Archive Objects*

This shows the number of recent metadata changes to a dataset at the source. By default, it is off. To enable it, toggle the Archive button and specify the number of objects to archive.

Example: Setting it to 4 retrieves the last four changes, displayed in the 'Version' column of the 'Metadata Changes' module.

Bridge

Select Bridge*

If applicable, select the bridge from the drop-down list.

The drop-down list displays all active bridges configured in OvalEdge. These bridges enable communication between data sources and OvalEdge without altering firewall rules.

After entering all connection details, the following actions can be performed:
1. Click Validate to verify the connection.
2. Click Save to store the connection for future use.
3. Click Save & Configure to apply additional settings before saving.
The saved connection will appear on the Connectors home page.

Manage Connector Operations

Crawl/Profile

To perform crawl operations, users must be assigned the Integration Admin role.

Navigate to the Connectors page and click Crawl/Profile.
This action initiates the metadata collection process from the data source and loads the retrieved metadata into the File Manager > File Explorer.
After a successful crawl, only the first-level folders and files are displayed in the File Manager tab under the specified connection name. To access inner-level folders or files, navigate through the folder hierarchy and perform the required operations.
In the File Manager, click the connector name, select the specific folder(s) or file(s), then click Catalog / Catalog and Profile from the Nine Dots menu. For more details, click here.
The selected files or folders will be added to the Data Catalog > Files/File Columns tab.

Profiling is supported only at the individual file level. To perform profiling, select the required file in the File Manager and use the Nine Dots menu to run the profiling operation. File columns become available only after the profiling process has been successfully completed.

Other Operations

The Connectors page in OvalEdge provides a centralized view of all configured connectors, including their health status.

Managing connectors includes:

Connector Health: Displays the current status of each connector using a green icon for active connections and a red icon for inactive connections, helping to monitor connectivity with data sources.
Viewing: Click the Eye icon next to the connector name to view connector details.

Nine Dots Menu Options:

To view, edit, validate, configure, or delete connectors, click on the Nine Dots menu.

Edit Connector: Update and revalidate the data source.
Validate Connector: Check the connection's integrity.
Settings: Modify connector settings.
- Crawler: Configure data extraction.
- Access Instructions: Add notes on how data can be accessed.
- Business Glossary Settings: Manage term associations at the connector level.
- Anomaly Detection Settings: Configure anomaly detection preferences at the connector level.
Delete Connector: Remove a connector with confirmation.

For more details on connector settings, click here.

Limitations

Sl. No

Scenario

Limitation

Bridge-enabled Connection

When Bridge is configured, the fileconnector.datatab.rowlimit System Setting must be configured in the OvalEdge application to ensure that Sample Data is retrieved and displayed correctly. If the setting is not configured, the Sample Data tab may display "No Data Exists" even when the source file contains data.

Troubleshooting

If incorrect parameters are entered, error messages may appear. Ensure all inputs are accurate to resolve these issues. If issues persist, contact the assigned support team.

S. No.

Error Message

Error Description & Resolution

Connection Validation Failed

Resolution:

Verify the Tenant ID, Client ID, Client Secret, and OneLake endpoint.
Confirm the Service Principal can access Microsoft Fabric OneLake.
Verify outbound connectivity to Microsoft Entra ID and OneLake endpoints.

Authentication Failed / Invalid Client Secret

Authentication fails because the Client Secret is invalid or expired, the Client ID or Tenant ID is incorrect, or the Microsoft Entra application configuration is invalid.

Resolution:

Verify the Client ID, Client Secret, and Tenant ID.
Regenerate the Client Secret if it has expired.
Ensure the Service Principal is active and properly configured.

Authorization Failure (Access Denied)

Authentication succeeds, but the Service Principal does not have sufficient permissions to list workspaces, browse folders, or read files in Microsoft Fabric OneLake.

Resolution:

Assign the required Azure Storage Blob Data Reader role for read operations.
Verify the Service Principal has the required Microsoft Fabric workspace access.
Confirm resource-level permissions are correctly configured.

Workspace or File Not Found

The specified workspace, lakehouse, folder, or file cannot be located because the configured path is incorrect or the resource no longer exists.

Resolution:

Verify the workspace, lakehouse, folder, and file path.
Confirm the resource exists in Microsoft Fabric.
Ensure the correct Files or Tables path is selected.

No Workspaces or Files Found

No Microsoft Fabric workspaces or files are returned because the Service Principal cannot list available workspaces, the tenant contains no accessible workspaces, or the configured location is empty.

Resolution:

Verify Microsoft Fabric workspaces exist.
Confirm the Service Principal has permission to list workspaces.
Validate the configured workspace and folder path.

Operation Timed Out

File browsing, preview, or profiling exceeds the configured timeout because of large datasets, network latency, or temporary Microsoft service delays.

Resolution:

Retry the operation after some time.
Limit operations to a smaller folder or dataset.
Verify network stability and timeout configuration.

File Download or Preview Failed

File download or preview fails because the file is unavailable, access is restricted, the configured path is incorrect, or network connectivity is interrupted.

Resolution:

Verify the file exists.
Confirm the Service Principal has read permission.
Validate the file path and retry the operation.

Unsupported or Invalid File Format

File preview or profiling fails because the file format, structure, encoding, or content is unsupported or invalid.

Resolution:

Verify the file format is supported.
Ensure CSV files contain a valid header row and supported encoding.
Open the file to confirm it is not corrupted.

Data Detection Failed

Column names or data types cannot be accurately detected because of inconsistent data, mixed data types, or insufficient sample records.

Resolution:

Ensure files contain sufficient sample data.
Maintain consistent values within each column.
Use supported data formats and encoding.

Network or Proxy Error

Communication with Microsoft Entra ID or Microsoft Fabric OneLake fails because of network restrictions, proxy configuration issues, or firewall rules.

Resolution:

Verify proxy configuration, if applicable.
Allow outbound HTTPS access to Microsoft Entra ID and OneLake endpoints.
Confirm DNS resolution and firewall settings.

AADSTS Authentication Error

Microsoft Entra ID returns an AADSTS authentication error because of invalid credentials, incorrect tenant configuration, expired secrets, or application configuration issues.

Resolution:

Verify the Tenant ID, Client ID, Client Secret, and endpoint configuration.
Confirm the Service Principal is correctly configured and has the required permissions.
Review the returned AADSTS error code to identify the specific authentication issue.

FAQs

Why does connection validation fail?

Connection validation fails when the Tenant ID, Client ID, Client Secret, or OneLake endpoint is incorrect, the Service Principal cannot authenticate, required permissions are missing, Microsoft Fabric workspaces are inaccessible, or the application cannot reach Microsoft Entra ID or OneLake services. Verify the authentication details, assigned permissions, and network connectivity.

Why are workspaces, folders, or files not displayed?

Workspaces, folders, or files may not appear when the Service Principal does not have permission to list Microsoft Fabric workspaces, the configured path is incorrect, the selected location is empty, or the resource is unavailable. Verify the workspace path, lakehouse location, and assigned permissions.

Which permissions are required?

Assign the Storage Blob Data Reader role for read operations such as browsing, previewing, downloading, and profiling files. Assign Storage Blob Data Contributor only when write operations are required. The Service Principal must also have an appropriate Microsoft Fabric workspace role to access workspace content.

Why are file operations slow?

File browsing, preview, download, or profiling may take longer when processing large lakehouses, numerous files, large Parquet datasets, or when network latency is high. Limiting operations to a specific folder or smaller dataset and ensuring stable network connectivity can improve performance.

Why does Service Principal authentication fail?

Service Principal authentication fails when the Tenant ID, Client ID, or Client Secret is incorrect or expired, the Microsoft Entra application is improperly configured, or the Service Principal does not have the required permissions. Verify the authentication details, regenerate the Client Secret if necessary, and ensure the Service Principal is active.

How can AADSTS authentication errors be resolved?

AADSTS errors indicate Microsoft Entra ID authentication problems such as invalid credentials, expired Client Secrets, incorrect Tenant IDs, application configuration issues, or missing permissions. Review the returned AADSTS error code, verify the Microsoft Entra application configuration, and update the connection details if required.

Why does connection validation succeed but file access fail?

Connection validation only verifies authentication and workspace accessibility. File operations require additional read permissions on Microsoft Fabric OneLake resources. Verify that the Service Principal has the required Azure Storage and Microsoft Fabric workspace permissions to access the target files.

Why does View Data work without Bridge but fail when using Bridge?

View Data operations executed through Bridge may fail when the required Bridge configuration or system properties are unavailable. Verify that the Bridge deployment is up to date, the required system properties are configured, and the Bridge can access Microsoft Fabric OneLake successfully.

PreviousAzure DevOps Repo

NextETL Tool

Last updated 4 hours ago

Was this helpful?