AWS AppFlow

Amazon AppFlow is a fully managed integration service that enables you to securely transfer data between Software-as-a-Service (SaaS) applications (like Salesforce, Marketo, Slack, and ServiceNow) and AWS services (like Amazon S3 and Amazon Redshift) in just a few clicks.

OvalEdge uses the APPFlow SDK to connect to the AWS AppFlow, which allows the user to crawl various object types (Flows, Tasks, Mapping, Transformations, etc) and build Lineage.

Connector Capabilities

The following lists the objects supported by the AWS AppFlow connector:

Functionality	Supported Objects
Crawling	Flows Tasks Mappings Transformations

Prerequisites

The following are the prerequisites required to establish a connection between the AWS AppFlow Connector and the OvalEdge application.

Establish Environment Variables (Optional)

This section describes the settings or instructions that you should be aware of prior to establishing a connection. If your environments have been previously configured, skip this step.

Configure Environment Names

The Environment Names allow you to select the environment configured for the specific connector from the drop-down list in the Add Connector pop-up window.

You might want to consider crawling the same schema in both the stage and production environments for consistency. The typical environments for crawling are PROD, STG, or Temporary, and may also include QA or other environments. Additionally, crawling a temporary environment can be useful for schema comparisons, which can later be deleted, especially during application upgrade assistance.

Steps to Configure the Environment

1. Navigate to Administration > System Settings.

2. Select the Connector tab.

3. Find the Key name “connector.environment”.

4. Enter the desired environment values (PROD, STG) in the value column.

5. Click ✔ to save.

Service Account with Minimum Permissions

A service account with read-only privileges on AppFlow SDK/APIs is required for crawling.

Establish a Connection

To connect to AWS AppFlow using the OvalEdge application, complete the following steps:

1. Log in to the OvalEdge application.

2. Navigate to Administration > Connectors module.

3. Click on the "+” (New Connector) button enabled at the top right of the page.

4. Add Connector pop-up window is displayed where you can search for the AWS AppFlow connector.

5. The Add Connector with connector type-specific details pop-up window is displayed. Enter the relevant information to configure the AWS AppFlow connection.

Note: The asterisk (*) denotes mandatory fields required for establishing a connection.

Field Name	Description
Connector Type	By default the selected connection type is displayed as the AWS AppFlow. Additionally, you can select the connector type from the drop-down list. Based on the selection, the relevant details are displayed.
Connector Settings
License Add-Ons	All the connectors will have a Base Connector License by default that allows you to crawl and profile to obtain the metadata and statistical information from a data source. OvalEdge supports various License Add-Ons based on the connector’s functionality requirements. Select the Auto Lineage Add-On license that enables the automatic construction of the Lineage of data objects for a connector with the Lineage feature. Select the Data Quality Add-On license to identify, report, and resolve the data quality issues for a connector whose data supports data quality, using DQ Rules/functions, Anomaly detection, Reports, and more. Select the Data Access Add-On license that will enforce connector access via OvalEdge with the Remote Data Access Management (RDAM) feature enabled.
Connector Name*	Select a connection name for AWS AppFlow. You can specify a connection name to identify the AWS AppFlow connection in OvalEdge. Example: Sample_AWSAppFlow
Connector Environment	The environment drop-down list allows you to select the environment configured for the connector from the drop-down list. For example, PROD, or STG (based on the configured items in the OvalEdge configuration for the connector.environment). The purpose of the environment field is to help you identify which connector is connecting what type of system environment (Production, STG, or QA). Note: The steps to set up environment variables are explained in the Prerequisites section.
Aws AppFlow Accesskey*	The Access Key ID is a unique identifier for an access key. Example: AKIAW7P****S45E3B**
Aws AppFlow Secret Access key*	The Secret Access Key is the secret password that pairs with the Access Key ID. Example: WcI***Yw+Ik+dIpjr****FRAesxP****Lh+m
Default Governance Roles
Default Governance Roles*	Users can select a specific user or a team from the governance roles (Steward, Custodian, Owner) that get assigned for managing the data asset. Note: The drop-down list displays all the configurable roles (single user or a team) as per the configurations made in the OvalEdge Administration > Security > Governance Roles section.
Admin Roles
Admin Roles*	Select the required admin roles for this connector. To add Integration Admin Roles, search for or select one or more roles from the Integration Admin options, and then click on the Apply button. The responsibility of the Integration Admin includes configuring crawling and profiling settings for the connector, as well as deleting connectors, schemas, or data objects. To add Security and Governance Admin roles, search for or select one or more roles from the list, and then click on the Apply button. The security and Governance Admin is responsible for: Configure role permissions for the connector and its associated data objects. Add admins to set permissions for roles on the connector and its associated data objects. Update governance roles. Create custom fields. Develop Service Request templates for the connector. Create Approval workflows for the templates.
Select Bridge	A solution is required to circumnavigate the customer firewall when OvalEdge is deployed as a SaaS application. That solution is OvalEdge Bridge. A bridge is a type of firewall that operates at the network layer. When a bridge has been set up, it will get displayed here in a drop-down list. Users can select the required Bridge ID. The user can select "NO BRIDGE" when it is not configured. For more information, refer to Bridge Overview

1. After entering all the connection details, select the appropriate button based on your preferences:

2. Validate: Click on the Validate button to verify the connection details. This ensures that the provided information is accurate and enables successful connection establishment.

3. Save: Click on the Save button to store the connection details. Once saved, the connection will be added to the Connectors home page for easy access.

4. Save & Configure: For certain Connectors that require additional configuration settings, click on the Save & Configure button. This will open the Connection Settings pop-up window, allowing you to configure the necessary settings before saving the connection.

5. Once the connection is validated and saved, it will be displayed on the Connectors home page.

Note: It is up to the user's wish, you can save the connection details first, or you can validate the connection first and then save it.

Connection Validation Errors

Error Messages	Description
Failed to establish a connection, please check the credentials (AWS AppFlow Accesskey, Aws AppFlow Secret Access key)	In case of an invalid access key and secret access key

Note: If you have any issues creating a connection, please contact your assigned OvalEdge Customer Success Management (CSM) team.

Connector Settings

Once the connection is validated successfully, various settings are provided to retrieve and display information from the data source.

To view the Connector Settings page,

1. Go to the Connectors page.

2. From the nine dots select the Settings option.

3. The Connector Settings page is displayed where you can view the connector setting options. For the AWS AppFlow Connector, only the Lineage setting option is available. Click Save Changes. All the settings will be applied to the metadata.

The following is a list of connection settings along with their corresponding descriptions:

Connection Settings

Description

Lineage

The purpose of the lineage setting is to serve the option of changing the source/server connection to build the lineage. The following are the two options that are displayed: Select server dialects to use for parsing source codes*: While parsing the source codes from the connector, the dialects selected will be used in the specified order. If the first server dialect fails to parse the code, the parsing will be tried using the next server dialects and so forth until the code is parsed successfully. If all the dialects fail to parse the code, then lineage will not be built, and the status will be shown as parse failed. At least one server dialect should be specified for building lineage for the connector codes. Connector priority to specify the connection for table Lineage: When a table with the same name or signature is present in more than one connector cataloged table, these objects will be resolved to the above-listed connectors in the specified order.

Crawling of Schema(s)

A Crawl/Profile option allows you to select the specific schemas for the following operations: Crawl and Schedule. For any scheduled crawlers, the defined run date and time are displayed to set.

1. Navigate to the Connectors page, and click on the Crawl/Profile button.

2. Select Schema For Crawling and Profiling pop-up window is displayed.

3. The below list of actions is displayed.

4. Crawl: It allows to crawling of the metadata of the selected schemas.

5. Schedule: Connectors can also be scheduled for crawling at prescribed times and selected intervals. Note: For more information on Scheduling, refer to the Scheduling Connector.

6. Click on the Run button that gathers all metadata from the connected source into the OvalEdge Data Catalog.