> For the complete documentation index, see [llms.txt](https://docs.ovaledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.ovaledge.com/connectors/connector-repositories/etl-tool/dbt.md).

# DBT

This article outlines the integration with the DBT (Data Build Tool) connector, enabling streamlined metadata management through features such as crawling and lineage building (both automatic and manual).

The connector establishes connectivity to DBT through REST APIs. Authentication is performed using a service token with the required permissions to access DBT APIs and retrieve metadata for crawling, validation, and lineage generation.

<figure><img src="https://content.gitbook.com/content/gnJyYKHzsnoFOdZJtHsc/blobs/t2UMxEPKVD50FPanGy5D/unknown.png" alt=""><figcaption></figcaption></figure>

## Overview

### Connector Details

| Connector Category                                                      | ETL Tool     |
| ----------------------------------------------------------------------- | ------------ |
| Connector Version                                                       | Release7.2.3 |
| Releases Supported (Available from)                                     | Release6.1   |
| <p>Connectivity</p><p>\[How the connection is established with DBT]</p> | REST APIs    |
| Verified DBT Version                                                    | 1.7.6        |

{% hint style="info" %}
The DBT connector has been validated with the mentioned "Verified DBT Versions" and is expected to be compatible with other supported DBT versions. If there are any issues with validation or metadata crawling, please submit a support ticket for investigation and feedback.
{% endhint %}

### Connector Features

| Feature                                      | Availability |
| -------------------------------------------- | :----------: |
| Crawling                                     |       ✅      |
| Delta Crawling                               |       ❌      |
| Data Preview                                 |       ❌      |
| Auto Lineage                                 |       ✅      |
| Manual Lineage                               |       ✅      |
| Secure Authentication via Credential Manager |       ✅      |
| Data Quality                                 |       ❌      |
| DAM (Data Access Management)                 |       ❌      |
| Bridge                                       |       ✅      |

### Metadata Mapping

The following objects are crawled from DBT and mapped to the corresponding UI assets.

<table><thead><tr><th width="142.6666259765625">DBT Object</th><th width="151">DBT Attribute</th><th width="172">OvalEdge Attribute</th><th width="169">OvaEdge Category</th><th>OvalEdge Type</th></tr></thead><tbody><tr><td>Manifest.json</td><td>Manifest.json</td><td>Code Name</td><td>Codes</td><td>DBT_JOB</td></tr><tr><td>SQL's inside Manifest file</td><td>Raw_code</td><td>Code Name</td><td>Codes</td><td>DBT_RUN_SQL</td></tr></tbody></table>

## Set up a Connection

### Prerequisites

The following are the prerequisites to establish a connection:

#### **Service Account User Permissions**

{% hint style="warning" %}
It is recommended to use a dedicated service Token to establish the connection to the data source, configured with the following minimum set of permissions.
{% endhint %}

{% hint style="info" %}
👨‍💻 **Who can provide these permissions?** These permissions are typically granted by the DBT administrator, as users may not have the required access to assign them independently.
{% endhint %}

| Operations                  | Objects       | Access Permission |
| --------------------------- | ------------- | ----------------- |
| Crawling                    | Projects      | Read-only         |
| Crawling & Lineage Building | Manifest.json | Read-only         |

### Connection Configuration Steps

{% hint style="warning" %}
Users are required to have the Connector Creator role in order to configure a new connection.
{% endhint %}

1. Log into OvalEdge, go to **Administration > Connectors**, click **+ (New Connector)**, search for **DBT**, and complete the required parameters.

{% hint style="info" %}
Fields marked with an asterisk (\*) are mandatory for establishing a connection.
{% endhint %}

<table><thead><tr><th width="220.666748046875">Field Name</th><th>Description</th></tr></thead><tbody><tr><td>Connector Type</td><td>By default, "DBT" is displayed as the selected connector type.</td></tr><tr><td>Credential Manager*</td><td><p>Select the desired credentials manager from the drop-down list. Relevant parameters will be displayed based on the selection.</p><p>Supported Credential Managers:</p><ul><li>OE Credential Manager</li><li>AWS Secrets Manager</li><li>HashiCorp</li><li>Azure Key Vault</li></ul><p>For more details, click <a href="https://docs.ovaledge.com/connectors/additional-requirements/credential-manager-configuration">here</a>.</p></td></tr><tr><td>License Add Ons</td><td>Select the checkbox for Auto Lineage Add-On to build data lineage automatically.<br>For more details, click <a href="https://docs.ovaledge.com/connectors/introduction-to-connectors/setup-and-connectivity/license-types-and-add-ons">here</a>.</td></tr><tr><td>Connector Name*</td><td><p>Enter a unique name for the DBT connection              </p><p>(Example: "DBT").</p></td></tr><tr><td>Connector Environment</td><td>Select the environment (Example: PROD, STG) configured for the connector.<br>For more details, click <a href="https://docs.ovaledge.com/connectors/introduction-to-connectors/setup-and-connectivity/prerequisites#connector-environment">here</a>.</td></tr><tr><td>Connector Description</td><td>Enter a brief description of the connector.</td></tr><tr><td>Server*</td><td>Enter the DBT server name or IP address.</td></tr><tr><td>Account ID*</td><td>Enter the Account ID associated with the DBT workspace to establish the connection.</td></tr><tr><td>Service account token*</td><td>Enter the service account token generated for the DBT workspace.</td></tr><tr><td>Excluded Jobs</td><td>Enter comma-separated job names that must be excluded from crawling.</td></tr><tr><td>Proxy Enabled*</td><td>Select <strong>Yes</strong> to route the connection through a configured proxy, or <strong>No</strong> to connect directly without a proxy.</td></tr><tr><td>Plugin Server</td><td>Enter the server’s name when running as a plugin server.</td></tr><tr><td>Plugin Port</td><td>Enter the port number on which the plugin is running.</td></tr></tbody></table>

**Default Governance Roles**

<table data-header-hidden><thead><tr><th width="219.333251953125"></th><th></th></tr></thead><tbody><tr><td>Default Governance Roles*</td><td>Select the appropriate users or teams for each governance role from the drop-down list. All users configured in the security settings are available for selection.</td></tr></tbody></table>

**Admin Roles**

<table data-header-hidden><thead><tr><th width="220.666748046875"></th><th></th></tr></thead><tbody><tr><td>Admin Roles*</td><td>Select one or more users from the drop-down list for Integration Admin and Security &#x26; Governance Admin. All users configured in the security settings are available for selection.</td></tr></tbody></table>

**No of Archive Objects**

<table data-header-hidden><thead><tr><th width="220.6666259765625"></th><th></th></tr></thead><tbody><tr><td>No Of Archive Objects*</td><td><p>This shows the number of recent metadata changes to a dataset at the source. By default, it is off. To enable it, toggle the Archive button and specify the number of objects to archive.</p><p><strong>Example</strong>: Setting it to 4 retrieves the last four changes, displayed in the 'Version' column of the 'Metadata Changes' module.</p></td></tr></tbody></table>

**Bridge**

<table data-header-hidden><thead><tr><th width="220.6666259765625"></th><th></th></tr></thead><tbody><tr><td>Select Bridge*</td><td><p>If applicable, select the bridge from the drop-down list.</p><p>The drop-down list displays all active bridges that have been configured. These bridges facilitate communication between data sources and the system without requiring changes to firewall rules.</p></td></tr></tbody></table>

2. After entering all connection details, the following actions can be performed:
   1. Click **Validate** to verify the connection.
   2. Click **Save** to store the connection for future use.
   3. Click **Save & Configure** to apply additional settings before saving.
3. The saved connection will appear on the Connectors home page.

## Manage Connector Operations

### Crawl

{% hint style="warning" %}
To perform crawl operations, users must be assigned the Integration Admin role.
{% endhint %}

The **Crawl/Profile** button allows users to select one or more Manifest files for crawling.&#x20;

1. Navigate to the Connectors page and click **Crawl/Profile**.
2. Select the Manifest files to crawl.
3. The **Crawl** option is selected by default.
4. Click **Run** to collect metadata from the connected source and load it into the **Data Catalog**.
5. After a successful crawl, the information appears in the **Data Catalog** > **Databases/<>Codes** tab.

The **Schedule** checkbox allows automated crawling at defined intervals, from a minute to a year.

1. Click the **Schedule** checkbox to enable the **Select** Period drop-down.
2. Select a time period for the operation from the drop-down menu.
3. Click **Schedule** to initiate metadata collection from the connected source.
4. The system will automatically execute the **crawl** operation at the scheduled time.

### Other Operations

The **Connectors** page provides a centralized view of all configured connectors, along with the health status.

**Managing connectors includes:**

* **Connector Health**: Displays the current status of each connector using a **green** icon for active connections and a **red** icon for inactive connections, helping to monitor connectivity with data sources.
* **Viewing**: Click the **Eye** icon next to the connector name to view connector details.

**Nine Dots Menu Options:**

To view, edit, validate, build lineage, configure, or delete connectors, click on the **Nine Dots** menu.

* **Edit Connector**: Update and revalidate the data source.
* **Validate Connector**: Check the connection's integrity.
* **Settings**: Modify connector settings.
  * **Lineage**: Select server dialects, connector priority, and parsing method (Regular or AI-Augmented) for generating lineage, and set a confidence threshold to control when AI-generated lineage is accepted or held for review. For more details, click [here](https://docs.ovaledge.com/release8.2/connectors/introduction-to-connectors/setup-and-connectivity/connector-settings#lineage-applicable-for-reports-and-etls-etl-connectors-reports).
* **Build Lineage**: Automatically build data lineage using source code parsing.
* **Delete Connector**: Remove a connector with confirmation.

For more details on connector settings, click [here](https://docs.ovaledge.com/connectors/introduction-to-connectors/setup-and-connectivity/connector-settings).

## Troubleshooting

If incorrect parameters are entered, error messages may appear. Ensure all inputs are accurate to resolve these issues. If issues persist, contact the assigned **support team**.

{% tabs %}
{% tab title="Connectivity" %}

<table data-search="true"><thead><tr><th width="59.2421875" align="right">S. No</th><th width="189.85546875">Error Message</th><th>Error Description &#x26; Resolution</th></tr></thead><tbody><tr><td align="right">1</td><td>Error while validating connection: invalid or expired service token</td><td><p>The service account token used for the DBT connector has expired, been revoked, or does not carry read-only permission.</p><p><strong>Resolution:</strong></p><ul><li>Confirm the account ID and API access URL entered match the values shown in the DBT account settings.</li><li>Confirm the service token is still active and generate a new token if it has expired or been revoked.</li><li>Confirm the token has at least read-only permission assigned before revalidating.</li></ul></td></tr><tr><td align="right">2</td><td>Unauthorized Access (401)</td><td><p>The authentication token or credentials are invalid or expired.</p><p><strong>Resolution</strong>:</p><ul><li>Validate the token or credentials used for the request.</li><li>Regenerate the token if expired.</li><li>Ensure the correct authentication method is configured.</li></ul></td></tr></tbody></table>
{% endtab %}

{% tab title="Crawling" %}

<table data-search="true"><thead><tr><th width="59" align="right">S. No</th><th width="190.23828125">Error Message</th><th width="500.21484375">Error Description &#x26; Resolution</th></tr></thead><tbody><tr><td align="right">1</td><td>Project listed during crawl selection but no dataset gets catalogued</td><td><p>The selected project has not yet been compiled, so no target folder or manifest file exists for it to be read.</p><p><strong>Resolution:</strong></p><ul><li>Confirm the project has been compiled at least once so that a target folder is generated.</li><li>Confirm a manifest.json file is present inside that target folder.</li><li>Request that the project be run in compile mode again if the target folder is missing.</li></ul></td></tr><tr><td align="right">2</td><td>Manifest file present but not recognized during crawl</td><td><p>The generated manifest file or its containing folder has been renamed away from the expected default names.</p><p><strong>Resolution:</strong></p><ul><li>Confirm the manifest file retains its default name, manifest.json, without manual renaming.</li><li>Confirm the folder containing it is named target, matching the structure DBT generates automatically.</li><li>Recrawl once the file and folder names are restored to their default values.</li></ul></td></tr><tr><td align="right">3</td><td>Descriptions are not getting copied from the DBT Manifest JSON file to Delta Lake tables and table columns after executing the DBT AdvanceJob.</td><td><ul><li>The DBT AdvanceJob cannot copy business descriptions because the manifest JSON file exceeds the configured manifest crawl size limit.</li><li>In this case, the 17 MB manifest JSON file was not cataloged because the <code>dbt.crawl.manifest.size.limit</code> setting was set to the default value of <strong>5 MB</strong>.</li><li>As a result, descriptions were not transferred to the corresponding Delta Lake tables and columns.<br><strong>Resolution:</strong></li><li>Verify the size of the DBT manifest JSON file.</li><li>Set <code>dbt.crawl.manifest.size.limit</code> to a value greater than or equal to the manifest file size.</li><li>Re-crawl the manifest JSON file.</li><li>Rerun the DBT AdvanceJob.</li><li>Verify that the descriptions are copied to the corresponding Delta Lake tables and columns.</li></ul></td></tr></tbody></table>
{% endtab %}

{% tab title="Lineage" %}

<table data-search="true"><thead><tr><th width="58.89453125" align="right">S. No</th><th width="189.609375">Error Message</th><th width="466.76953125">Error Description &#x26; Resolution</th></tr></thead><tbody><tr><td align="right">1</td><td>Table level lineage is present but column level lineage is missing for a model</td><td><p>The manifest file does not contain compiled code for that model, which happens when the project was run rather than compiled.</p><p><strong>Resolution:</strong></p><ul><li>Confirm the project has been executed in compile mode rather than only run mode.</li><li>Confirm every model intended for lineage was included in the compile step.</li><li>Request an updated manifest export after a full project compile.</li></ul></td></tr><tr><td align="right">2</td><td>A table appears in the source list but is missing from the lineage mapping, or the reverse</td><td><p>This occurs when a table is referenced using a hardcoded name in the raw query instead of the standard reference syntax, causing a mismatch between the recorded sources and the compiled query content.</p><p><strong>Resolution:</strong></p><ul><li>Confirm standard reference or source functions are used in the model queries instead of hardcoded table names.</li><li>Review the specific model query for any hardcoded references.</li><li>Escalate to the development team if the mismatch continues, as a longer-term enhancement is planned for this scenario.</li></ul></td></tr><tr><td align="right">3</td><td>Lineage displays a temporary table name instead of the expected system name</td><td><p>The target or source system referenced by the model has not yet been catalogued separately in the application.</p><p><strong>Resolution:</strong></p><ul><li>Confirm the relevant source or target system connector has already been catalogued.</li><li>Recrawl the lineage once that system is catalogued so the actual object name resolves correctly.</li><li>Expect a temporary placeholder name to continue appearing until the related system is added.</li></ul></td></tr><tr><td align="right">4</td><td>No clear indication of why lineage failed for a specific model</td><td><p>Detailed logging for lineage build failures is limited at present, making the root cause harder to identify directly from job logs.</p><p><strong>Resolution:</strong></p><ul><li>Check whether the dataset associated with the model shows an empty query or SQL content.</li><li>Manually verify whether the model was included in a full project compile.</li><li>Escalate the specific model for review, since enhanced logging for this scenario is still pending.</li></ul></td></tr></tbody></table>
{% endtab %}
{% endtabs %}

## FAQs

{% tabs %}
{% tab title="General" %}

<table data-search="true"><thead><tr><th width="59.2890625" align="right">S. No</th><th width="189.54296875">Question</th><th width="479.6484375">Answer</th></tr></thead><tbody><tr><td align="right">1</td><td>When should the dbt core connector be used instead of the dbt cloud connector?</td><td><p>The choice depends on where the dbt projects are hosted:</p><ul><li><strong>DBT cloud connector:</strong> Use when the dbt projects are hosted in a <strong>dbt cloud instance</strong>. It uses REST API access to retrieve the metadata.</li><li><strong>DBT core connector:</strong> Use when the dbt projects are stored in a <strong>local directory, NFS location, or S3 bucket</strong> without a dbt cloud instance. It reads the project files directly from the storage location.</li></ul><p>The project structure and lineage process remain the same for both connectors. Only the method used to access the metadata differs.</p></td></tr><tr><td align="right">2</td><td>What access is required to set up each type of dbt connector?</td><td><p><strong>DBT connector:</strong></p><ul><li><strong>Account ID</strong></li><li><strong>API access URL</strong></li><li><strong>Service account token</strong></li></ul><p>These details are generated from the <strong>dbt instance</strong> account settings. The <strong>service account token</strong> should have <strong>read-only</strong> permission because write access is not required for cataloguing metadata.</p></td></tr><tr><td align="right">3</td><td>Can a single dbt project transform data between completely different source platforms?</td><td>No. A single dbt project is expected to work within <strong>one source platform</strong> at a time.<br>Different databases or connection details within the same platform can be used. However, transformations between completely different platforms generally require <strong>separate dbt projects</strong>, with each project aligned to its respective platform</td></tr></tbody></table>
{% endtab %}

{% tab title="Crawling" %}

<table data-search="true"><thead><tr><th width="59.015625" align="right">S. No</th><th width="190.44140625">Question</th><th>Answer</th></tr></thead><tbody><tr><td align="right">1</td><td>What exactly gets crawled when a dbt project is catalogued?</td><td><p>The primary artifact collected from a dbt project is <strong>manifest.json</strong>.The <strong>manifest.json</strong> file is automatically generated in the <strong>target</strong> folder when the project is compiled. It contains information such as:</p><ul><li>Model definitions</li><li>Test information</li><li>Documentation details</li><li>Source and target relationships used for lineage</li></ul><p>The <strong>dbt connector</strong> collects this manifest.json file, regardless of where the project is hosted.</p></td></tr><tr><td align="right">2</td><td>What happens if a dbt project has not been compiled yet?</td><td><p>If the project has not been compiled:</p><ul><li>The <strong>target</strong> folder and <strong>manifest.json</strong> file will not exist.</li><li>The project can still appear as a selectable schema during crawl selection because its folder structure is available.</li><li>No dataset will be catalogued because there is no manifest.json file to read.</li></ul><p>Compile the project in <strong>compile mode</strong> to generate the target folder and manifest.json file.</p></td></tr><tr><td align="right">3</td><td>Does each dbt project generate more than one manifest file?</td><td><p>No. Each dbt project generates <strong>one manifest.json file per compile action</strong>.</p><p>The manifest.json contains information for all models in the project. If only selected models are included in the compile command, the manifest.json is still generated for the entire project, but <strong>compiled code</strong> is populated only for the models included in that compile step.</p></td></tr></tbody></table>
{% endtab %}

{% tab title="Lineage" %}

<table data-search="true"><thead><tr><th width="58.82421875" align="right">S. No</th><th width="190.1484375">Question</th><th width="468.89453125">Answer</th></tr></thead><tbody><tr><td align="right">1</td><td>How is table-level lineage built for a dbt project?</td><td><p><strong>Table-level lineage</strong> is built using the <strong>depends-on</strong> information in the manifest.json file.</p><p>The depends on field identifies the source objects used by each model. The connector uses this information across the models to link source tables to their target tables and build table-level lineage.</p></td></tr><tr><td align="right">2</td><td>How is column-level lineage built for a dbt project?</td><td><p><strong>Column-level lineage</strong> is built using the <strong>compiled code</strong> in the manifest.json file.</p><p>The compiled code contains the processed query for each model. The query is parsed to identify the source and target column mappings, which are then used to build column-level lineage.</p><p>If <strong>compiled code</strong> is not available for a model, column-level lineage cannot be generated for that model.</p></td></tr><tr><td align="right">3</td><td>Why might column lineage be missing for certain models even though the project appears to be fine?</td><td><p>Column lineage may be missing when the required <strong>compiled code</strong> is not available.</p><p>This can happen when:</p><ul><li>The project was <strong>run</strong> instead of properly compiled.</li><li>Only a subset of models was included in the compile command.</li><li>The required model was not included in the compile step.</li></ul><p>Run the full project in <strong>compile mode</strong> to generate compiled code for all models.</p></td></tr><tr><td align="right">4</td><td>What do the temporary table names shown in some lineage diagrams mean?</td><td><p>Temporary table names are displayed when a referenced <strong>source or target system</strong> has not yet been catalogued separately.</p><p>In this case:</p><ul><li>The table and column details remain accurate.</li><li>The originating connector information is displayed as temporary.</li><li>Once the source or target system is catalogued, recrawling resolves the reference to the correct name.</li></ul></td></tr><tr><td align="right">5</td><td>Can lineage exist for a model even when there is no column-level mapping?</td><td><p>Yes. <strong>Table-level object lineage</strong> can be available even when <strong>column-level mapping</strong> is empty.</p><p>This can occur when:</p><ul><li><strong>Compiled code</strong> is not available for the model.</li><li>A table is referenced through a join, but none of its individual columns are selected in the query.</li></ul><p>In these cases, the table relationship can still be identified even though column mappings cannot be generated.</p></td></tr><tr><td align="right">6</td><td>Is detailed logging currently available when a lineage build fails for a specific model?</td><td><p>No. Detailed logging for lineage build failures is currently limited. Troubleshooting may require manual checks, such as:</p><ul><li>Checking whether the associated dataset has an empty query.</li><li>Verifying that the project was correctly compiled for the affected model.</li></ul><p>Enhanced logging for easier diagnosis is being considered for a future update.</p></td></tr></tbody></table>
{% endtab %}
{% endtabs %}

***

Copyright © 2026, OvalEdge LLC, Peachtree Corners GA USA


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.ovaledge.com/connectors/connector-repositories/etl-tool/dbt.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.