# ECS Variant-1: New Customers

This article outlines the end-to-end process for enabling AskEdgi components for newly onboarded SaaS customers on ECS-Variant-1. The implementation leverages AWS Control Tower automation for creating and managing core AWS resources, incorporating manual verification steps to ensure correct configuration and functionality.

AskEdgi components rely on a combination of AWS services such as ECS, RDS, EFS, S3, SQS, IAM, and Secrets Manager. These components are configured to interact seamlessly, enabling secure and scalable operations for the SaaS environment. This document provides a structured, step-by-step guide for setting up network security, IAM roles, ECS tasks, S3 buckets, SQS queues, event notifications, Bridge server and client setup, and feature validation for AskEdgi.

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FVrIuBlzWDt839CPQJ0DG%2Fimage.png?alt=media&#x26;token=8ae8175a-8508-4408-810b-d522042c7707" alt=""><figcaption></figcaption></figure>

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FMFgusluRkqSQrC28UCcx%2Fimage.png?alt=media&#x26;token=a146e06f-c34b-46a4-8246-e02cfc0085a4" alt=""><figcaption></figcaption></figure>

## Purpose of the document

The purpose of this document is to provide a detailed, professional, and structured process for implementing AskEdgi components on ECS-Variant-1 for new SaaS customers. It ensures all AWS resources, ECS tasks, S3 buckets, SQS queues, IAM configurations, and application services are configured correctly. It also provides verification steps to validate operational readiness and integration of AskEdgi within the Ovaledge environment.

## Prerequisites

<table><thead><tr><th width="128.9166259765625">Component</th><th width="251.5">Requirement</th><th>Notes / Important</th></tr></thead><tbody><tr><td>ECS</td><td>ECS cluster and service access</td><td>Required for deploying Edgi Data and Insight services</td></tr><tr><td>SQS</td><td>Access to create and manage queues</td><td>Needed for data sync between services</td></tr><tr><td>IAM</td><td>Permissions to create roles, users, and attach policies</td><td>Ensure IAM policies include required permissions for ECS, SQS, and S3</td></tr><tr><td>RDS</td><td>Database instance</td><td>Used for storing Edgi data and metadata</td></tr><tr><td>EFS</td><td>File system access</td><td>Required for persistent storage for the Insight service</td></tr><tr><td>Secrets Manager</td><td>Create and manage secrets</td><td>Used to store database credentials and other sensitive keys</td></tr><tr><td>EC2</td><td>EC2 instances for ECS tasks</td><td>Ensure instance type and network configurations are compatible</td></tr></tbody></table>

{% hint style="warning" %}

* Ensure AWS Control Tower automation is active and operational before starting.
* Manual verification is required for ECS task roles, IAM user credentials, S3 bucket permissions, and event notifications.
* All services should comply with corporate security policies regarding VPCs, subnets, and access controls.
  {% endhint %}

## Implementation Steps

### Step 1: Network Security

* Create two private subnets for the Edgi Insight Service.
* Create a Security Group for the Edgi ECS service with the following rules:

| Protocol | Port | Source/Destination                   |
| -------- | ---- | ------------------------------------ |
| TCP      | 8000 | Destination VPC range                |
| TCP      | 9200 | Private Elasticsearch Security Group |
| TCP      | 3306 | Private RDS Security Group           |
| All      | All  | Private EFS Security Group           |

{% hint style="info" %}

* Ensure that ports 8000, 9200, and 3306 are allowed only from specified sources.
* Security Group configuration ensures proper communication between Edgi ECS, RDS, EFS, and Elasticsearch.
  {% endhint %}

### Step 2: EFS Access Points

* Create EFS Access Points for askEdgi Insight Service:

| Access Point Path                   | Notes                       |
| ----------------------------------- | --------------------------- |
| /oe\_edgi\_insights\_dev\_logs      | Stores Insight service logs |
| /oe\_edgi\_insights\_dev\_documents | Stores uploaded documents   |
| /oe\_edgi\_insights\_dev\_store     | Stores processed data       |

{% hint style="warning" %}
Assign correct mount permissions to ECS tasks to ensure persistent storage access.
{% endhint %}

### Step 3: S3 Bucket Creation

* Create a new S3 bucket for storing Edgi-related data.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2Fvg4LECDhWaRlKb1MFCES%2Fimage.png?alt=media&#x26;token=013811ee-d411-4228-bbde-175aca793125" alt=""><figcaption></figcaption></figure>
* Configure CORS permissions:

```
[
  {
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["PUT"],
    "AllowedOrigins": ["*"],
    "ExposeHeaders": []
  }
]
```

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FrxhIQQRm18F5ceWuP5LJ%2Fimage.png?alt=media&#x26;token=db2e45cd-cce7-4433-9daf-550041b7098a" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}

* The bucket will store all data uploaded via Edgi Data Service and processed by the Insight Service.
* Ensure correct permissions to allow S3 to send events to SQS.
  {% endhint %}

### Step 4: IAM User Configuration

#### IAM User for AskEdgi Python

* Create IAM user (example: edgi\_user) with:
  * AmazonSQSFullAccess
  * AskEdgi\_policy\_Saas from: AWS SAAS ECS Policy JSON

    * AmazonSQSFullAccess: <https://oe-saas-distributions-bridgeclient.s3.ap-south-1.amazonaws.com/Askedgi/AWS_SAAS_ECS_Policy.json>

    <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FEVG4nMElFWLRrbqEL80c%2Fimage.png?alt=media&#x26;token=e6690db8-9bab-4dc8-9bca-3396467ea9b8" alt=""><figcaption></figcaption></figure>

    <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FWVpINV4jyMXXPNBf9wGX%2Fimage.png?alt=media&#x26;token=6e346781-f43c-4bce-bfe7-d5570ce8db5e" alt=""><figcaption></figcaption></figure>
* Generate Access Key and Secret Key for CLI access.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FgyHLcEv84zdHOYXMVEj3%2Fimage.png?alt=media&#x26;token=3735ca84-3d78-4a7c-824a-28d4dba704d5" alt=""><figcaption></figcaption></figure>
* Use credentials in Ovaledge Task Definition environment variables:

| Environment Variable   | Value         |
| ---------------------- | ------------- |
| EDGI\_AWS\_ACCESS\_ID  | \<access key> |
| EDGI\_AWS\_SECRET\_KEY | \<secret key> |

#### IAM User for Bridge Server and Client

* Create another IAM user with S3 PutObject permissions only.<br>

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2Fen993HdPArklBuq40Io8%2Fimage.png?alt=media&#x26;token=61c753a5-bc81-4475-9b86-3a0454bb1905" alt=""><figcaption></figcaption></figure>
* Use credentials in Ovaledge Task Definition environment variables:

| Environment Variable           | Value                    |
| ------------------------------ | ------------------------ |
| BRIDGE\_AWS\_USER\_ACCESS\_KEY | \<bridge IAM access key> |
| BRIDGE\_AWS\_USER\_SECRET\_KEY | \<bridge IAM secret key> |

{% hint style="info" %}

* Bridge user credentials are required for secure S3 uploads.
* Validate IAM policies before ECS task deployment.
  {% endhint %}

### Step 5: ECS Task Role Configuration

* Create IAM Role with ECS task execution permissions.
* Attach the custom policy: [AskEdgi ECS Policy JSON.](https://oe-saas-distributions-bridgeclient.s3.ap-south-1.amazonaws.com/Askedgi/AWS_SAAS_ECS_Policy.json)

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FUFa6KeVaTO02GfOkU0UD%2Fimage.png?alt=media&#x26;token=b615f53a-75cf-4313-b80e-e4ca218da567" alt=""><figcaption></figcaption></figure>
* Modify the Trust Relationship for ECS.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FjBhTIjtGFXJz2IEGrYBA%2Fimage.png?alt=media&#x26;token=992afe7f-80ad-479d-8c7a-1886d243189c" alt=""><figcaption></figcaption></figure>

{% hint style="warning" %}

* This IAM role is shared across all ECS Task Definitions to ensure consistent and secure access.
* Automation scripts handle permission setup automatically, but verification is recommended.
  {% endhint %}

### Step 6: AWS SQS Configuration

Amazon SQS (Simple Queue Service) is a fully managed message queuing service provided by AWS. It allows decoupling of application components so that they can communicate and scale independently.

* Create SQS queue: data\_sync\_queue using naming convention \<prefix>\_data\_sync\_queue (example: askedgi\_data\_sync\_queue).

**Configure Access Policy – Send Message Permissions**

* Go to Advanced settings and note the Resource ARN.
* Modify the JSON below to include the AWS Account ID, Region, IAM Role Name, and S3 Bucket ARN.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FtvfKqHlSQE2bmnF7RE8r%2Fimage.png?alt=media&#x26;token=241effe0-3e4f-4086-82cd-315f8a1f9951" alt=""><figcaption></figcaption></figure>

**Access Policy – add Send Message Permissions:**

* Modify the below JSON according to the account ID  and S3 ARN
* AWS: SourceArn: The S3 bucket ARN created at Step 3.

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "__owner_statement_1",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::${AWS_ACCOUNT_ID}:root"},
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:{AWS_REGION}:${AWS_ACCOUNT_ID}:{PREFIX}_data_sync_queue"
    },
    {
      "Sid": "__receiver_statement",
      "Effect": "Allow",
      "Principal": {"AWS": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/{IAM_ROLE_NAME}"},
      "Action": ["SQS:ChangeMessageVisibility","SQS:DeleteMessage","SQS:ReceiveMessage"],
      "Resource": "arn:aws:sqs:{AWS_REGION}:${AWS_ACCOUNT_ID}:{PREFIX}_data_sync_queue"
    },
    {
      "Sid": "__s3_sender_statement",
      "Effect": "Allow",
      "Principal": {"Service": "s3.amazonaws.com"},
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:{AWS_REGION}:${AWS_ACCOUNT_ID}:{PREFIX}_data_sync_queue",
      "Condition": {"ArnLike": {"aws:SourceArn": "arn:aws:s3:::${BUCKET_NAME}"}}
    }
  ]
}

```

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FYQFuWsCIPPp7zqaiuUNv%2Fimage.png?alt=media&#x26;token=a9a10302-2e62-4d55-942e-b9204b791dd1" alt=""><figcaption></figcaption></figure>

**Access Policy – Receive Message Permissions:**

* Principal: IAM role ARN
* Resource: edgi\_galaxy\_data\_sync\_queue ARN

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FtPMCpHOTalb4VmrhOgSU%2Fimage.png?alt=media&#x26;token=9861e190-a94d-4966-85e9-150cab850184" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}

* Ensure aws: SourceArn points to the correct S3 bucket.
* Verify SQS ARN for both send and receive permissions.
  {% endhint %}

### Step 7: S3 Event Notifications

Amazon S3 Event Notifications allow automatic triggering of actions when certain events happen in an S3 bucket - like when an object is uploaded, deleted, or restored.

To enable automated processing of uploaded files, configure S3 Event Notifications to trigger the SQS queue:

* Navigate to the S3 bucket > Properties > Event Notifications > Click Create Event Notification.
* Create Notification for CSV Files

  * Name: csv\_event\_to\_sqs (customizable)
  * File Suffix: .csv
  * Event Type: All object creation events
  * Destination: SQS
  * SQS ARN: {PREFIX}\_data\_sync\_queue

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FDdimOiwvog4zSpafLDlc%2Fimage.png?alt=media&#x26;token=bca2b125-4061-40ab-a417-9a14f3e81842" alt=""><figcaption></figcaption></figure>

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FHs1aNfWLY5ZjkwAKvKWX%2Fimage.png?alt=media&#x26;token=4d4bd0d5-f9b0-4fe6-9a2d-9e019c232821" alt=""><figcaption></figcaption></figure>

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2Fok6AG8zvXboeZp1Nnv9u%2Fimage.png?alt=media&#x26;token=ee5ddf3e-c8df-4cf0-b5f9-591c7e76c9cb" alt=""><figcaption></figcaption></figure>

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FWic8yX9bgoAlBmovT3DQ%2Fimage.png?alt=media&#x26;token=1b6150b8-fb0e-40d4-882f-0bd56c06c85c" alt=""><figcaption></figcaption></figure>
* **Create Additional Notifications**\
  Repeat the above steps for the following file types, ensuring each notification points to the same SQS queue:

<table><thead><tr><th width="165.6666259765625">Notification Name</th><th width="134">File Suffix</th><th>Event Type</th><th>Destination</th></tr></thead><tbody><tr><td>avro_event</td><td>.avro</td><td>All object creation events</td><td>data_sync_queue</td></tr><tr><td>json_event</td><td>.json</td><td>All object creation events</td><td>data_sync_queue</td></tr><tr><td>parquet_event</td><td>.parquet</td><td>All object creation events</td><td>data_sync_queue</td></tr><tr><td>xlsx_event</td><td>.xlsx</td><td>All object creation events</td><td>data_sync_queue</td></tr></tbody></table>

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FsrouIygGE5JiGUMKA2Mn%2Fimage.png?alt=media&#x26;token=a8899717-3c3d-4f76-8562-b73a6e1f706c" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}

* These notifications enable real-time processing of files uploaded to the S3 bucket.
* Verify that the correct SQS queue ARN is used for each notification.
  {% endhint %}

### Step 8: ECS Task Definition – Data Service

Create an ECS Task Definition for the Edgi Data Service with the following configuration details.

| Configuration   | Value             |
| --------------- | ----------------- |
| CPU             | 0.5 vCPU          |
| Memory          | 2 GB              |
| Container Name  | edgi-data-service |
| Port            | 8000              |
| Storage         | 200 GB            |
| CloudWatch Logs | Enabled           |

#### Environment Variables

| Key              | Value                   |
| ---------------- | ----------------------- |
| AWS\_REGION      | \<Region>               |
| AWS\_SCTNAME     | \<Database secret name> |
| BUCKET\_NAME     | \<Bucket Name>          |
| CREDENTIAL\_TYPE | container               |
| DEPLOYMENT\_TYPE | ecs                     |
| ENV              | production              |

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2Fdngp2edHw1tm7ANIuVKe%2Fimage.png?alt=media&#x26;token=900e79c4-d2de-4433-8240-0f28d2974d63" alt=""><figcaption></figcaption></figure>

* Enable CloudWatch Logs for the container to capture logs.
* Under Storage configuration, allocate a maximum of 200 GB.

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FBLqOMnW9AnzeOuUHb7Hp%2Fimage.png?alt=media&#x26;token=64da0535-5dd9-4d23-bc68-57eed66c2e1c" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
CloudWatch logging is essential for monitoring Data Service activity.
{% endhint %}

### Step 9: ECS Task Definition – Insight Service

* Create an ECS Task Definition for the Edgi Insight Service using the details below:

| Configuration   | Value                       |
| --------------- | --------------------------- |
| CPU             | 1 vCPU                      |
| Memory          | 2 GB                        |
| Container Name  | oe\_edgi\_insights\_service |
| Port            | 8000                        |
| Storage         | 200 GB                      |
| CloudWatch Logs | Enabled                     |

#### Environment Variables

| Key                          | Value                                   |
| ---------------------------- | --------------------------------------- |
| AI\_FUNCTION\_QUEUE\_NAME    | ai\_functions\_queue                    |
| AWS\_REGION                  | \<region>                               |
| AWS\_SCTNAME                 | \<db\_secretname>                       |
| BUCKET\_NAME                 | \<S3 bucket name>                       |
| CLUSTER\_NAME                | \<ecs cluster name>                     |
| CREDENTIAL\_TYPE             | container                               |
| DATA\_SYNC\_QUEUE\_NAME      | data\_sync\_queue                       |
| DEPLOYMENT\_TYPE             | ecs                                     |
| DNS\_NAME\_SPACE\_ID         | \<ecs namespace ID>                     |
| DOCS\_FOLDER                 | /app/documents                          |
| EDGI\_WORKSPACE\_DISCOVERY   | dynamic                                 |
| EDGI\_WORKSPACE\_HOST        | edgi-data-service                       |
| EDGI\_WORKSPACE\_NAME\_SPACE | \<ecs namespace name>                   |
| ENV                          | prod                                    |
| LOG\_PATH                    | /app/logs                               |
| SECURITY\_GROUPS             | \<Edgi insight>                         |
| SQS\_QUEUE\_PREFIX           | \<sqs\_prefix\_name>                    |
| STORE\_FOLDER                | /app/store                              |
| SUBNETS                      | sub1, sub2                              |
| TASK\_DEFINITION             | \<Edgi data task definition latest ARN> |

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FmmvqSEJ5QtREELrDbH7B%2Fimage.png?alt=media&#x26;token=d6c5ec7f-23e8-4e87-951c-3fb718ec6c61" alt=""><figcaption></figcaption></figure>

* Elastic File System Configuration
  * Add three EFS mounts for the Insight service.

    <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2F5UvGNOF6pTP82Aly9qFR%2Fimage.png?alt=media&#x26;token=7411c5df-0e90-46aa-a684-3eae9a6378fa" alt=""><figcaption></figcaption></figure>
* Enable CloudWatch Logs for the container to capture logs
* Under Storage configuration, allocate a maximum of 200 GB

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FqikjU7JGJ81HeIfudNPL%2Fimage.png?alt=media&#x26;token=bc165ac8-5b0a-4020-bc57-5979191a0ddf" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}

* Add three EFS mounts for persistent storage.
* CloudWatch logging must be enabled for troubleshooting.
  {% endhint %}

### Step 10: ECS Service Creation – Insight Service

* Deploy ECS Service using the Insight Task Definition.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FLCEQ6mid00EDoUmleE1h%2Fimage.png?alt=media&#x26;token=8c7ecbc1-d2da-46d5-a5a0-c884ecdfd679" alt=""><figcaption></figcaption></figure>
* **Network Security**

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2F2DDW3dWm4Fk9TlFyWJsw%2Fimage.png?alt=media&#x26;token=c2b160b4-1d82-44e1-a5b2-b236629afae2" alt=""><figcaption></figcaption></figure>
* Enable Service Discovery during service creation.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FJGgJvaxiIi25ezIyJrGk%2Fimage.png?alt=media&#x26;token=198df93b-97d5-4314-b8c4-3bb7d3c09e37" alt=""><figcaption></figcaption></figure>

**Once the service is deployed:**

* Go to **Configuration and Networking** > **Service Discovery.**
* Note down the Service Discovery Endpoint, which will be used in Ovaledge configuration.

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FCuvdqwO92Mg6Zkk3BTY1%2Fimage.png?alt=media&#x26;token=3ac44578-f82f-4ebe-b926-4a385b80f243" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
Service Discovery endpoint is required in Ovaledge task definitions.
{% endhint %}

### Step 11: Update Ovaledge Task Definition and Job

To enable the AskEdgi feature in the Ovaledge environment, update the UI and Job ECS Task Definitions by adding the following environment variables.

#### Environment Variables for UI and Job Tasks

| Key                              | Value                                    |
| -------------------------------- | ---------------------------------------- |
| BRIDGE\_AWS\_USER\_ACCESS\_KEY   | \<Bridge user access key>                |
| BRIDGE\_AWS\_USER\_SECRET\_KEY   | \<Bridge user secret key>                |
| EDGI\_AI\_FUNCTION\_QUEUE        | ai\_functions\_queue                     |
| EDGI\_AWS\_ACCESS\_ID            | \<edgi user access key>                  |
| EDGI\_AWS\_SECRET\_KEY           | \<edgi user secret key>                  |
| EDGI\_AWS\_REGION                | \<region>                                |
| EDGI\_BUCKET\_NAME               | \<s3 bucket name>                        |
| EDGI\_CLIENT\_PREFIX             | \<sqs prefix name>                       |
| EDGI\_CONSUMER\_CREDENTIAL\_TYPE | static                                   |
| EDGI\_ENABLE                     | true                                     |
| EDGI\_INSIGHT\_SERVICE\_HOST     | \<insight service endpoint>              |
| EDGI\_WORKSPACE\_DISCOVERY       | dynamic                                  |
| EDGI\_WORKSPACE\_HOST            | \<edgi data task definition family name> |
| EDGI\_WORKSPACE\_NAMESPACE       | \<ECS cluster namespace>                 |

#### Optional OAUTH2 Configuration

| Key                          |                         |
| ---------------------------- | ----------------------- |
| OAUTH2\_BASE\_URL            | \<provided by dev team> |
| OAUTH2\_CLIENTID             | \<provided by dev team> |
| OAUTH2\_CLIENT\_REGISTRATION | auth0                   |
| OAUTH2\_ROLE\_ATTRIBUTE      | https\:///roles         |
| OAUTH2\_SECRET               | \<provided by dev team> |
| OVALEDGE\_SECURITY\_TYPE     | oauth2                  |

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FHHP82tzylFrmO8B9sYZ2%2Fimage.png?alt=media&#x26;token=18e3029c-af80-4011-b45e-a18cf7d3d48f" alt=""><figcaption></figcaption></figure>

Both the UI and Job task definitions must be updated, and then new ECS services must be deployed using the latest task definition revisions.

### Step 12: AskEdgi Feature Checks

* Once the application is up and running.
* Log in with the admin user.
* Verify Workspace creation with a valid project ID.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2F2durirlA2f9M1yukguID%2Fimage.png?alt=media&#x26;token=8e70e490-fecc-40d0-a1a0-92dde000e674" alt=""><figcaption></figcaption></figure>
* Navigate to **Application > Settings > AI**.
* Under AI Config, select OpenAI.
* Enter the API Key provided by the development team.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FvzxHbJJMPEChKNmPbam4%2Fimage.png?alt=media&#x26;token=43987ef4-6d98-4e07-815d-601e5c4be5b3" alt=""><figcaption></figcaption></figure>

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FfLX4vKjMQXY8J7DLnBNI%2Funknown.png?alt=media&#x26;token=ea35bf2f-1e74-4822-b6dd-cde7f9930193" alt=""><figcaption></figcaption></figure>
* Save the configuration to enable AI functionality within AskEdgi.
* By default, AskEdgi Edition provides Data Analytics functionality (upload and catalog data).
* If configured for Metadata Analytics, only data cataloging will be available.
* After saving, log out and log in again.
* Go to the AskEdgi module and wait for the workspace to initialize.
* Then, try uploading a file from the application.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FW33szu03MlCWNmwJ6jrL%2Fimage.png?alt=media&#x26;token=90bb5cb0-983a-4be5-b9c6-bd6e82fd0d5f" alt=""><figcaption></figcaption></figure>
* Navigate to **Application > Settings > AI**.
* Ensure that all Marketplace details are correctly populated in the settings.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2Fi8R2xNv7e14dfaqsosjg%2Fimage.png?alt=media&#x26;token=9d7861b1-dbcb-4cca-aaf8-fe55adef166e" alt=""><figcaption></figcaption></figure>
* Save the configuration if any updates are made.

If Workspace is not created:

* Check Insight Service logs.
* Validate IAM Role permissions.
* Verify user entry in the database:

```
select * from edgi_data_service_stack edss ;
delete * from edgi_data_service_stack edss;
delete from edgi_data_service_uptime_history ;
```

{% hint style="warning" %}
Do not run delete queries unless confirmed by the development team.
{% endhint %}

**Remove User Entry from Service Discovery**

* Navigate to **AWS Cloud Map** > **Service Discovery Resource**.
* Locate and delete the user entry associated with the application.
* Log back into the application to verify that the changes have taken effect.
* Check if the files have been uploaded or not.

### Step 13: NiFi Bridge Server

#### AWS Access Key Secret Key for Bridge Server and Client

* These credentials have S3 PutObject permissions, allowing secure uploads to the S3 bucket (created in Step **4B**).
* During Ovaledge application installation, AWS Credentials (Access Key & Secret Key) are securely stored in **/mnt/tmp** and also provided as environment variables in the **run.sh** file.
* During Bridge Client installation, the credentials, along with S3 bucket details (name and region), are packaged into the Bridge Client certificate ZIP file. This enables the Bridge Client to authenticate with S3 seamlessly.<br>

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FrQdUMU6lpd6FRSs8q8x6%2Funknown.png?alt=media&#x26;token=4b359b49-d763-456b-9fce-5ba4851c0ce4" alt=""><figcaption></figcaption></figure>

**ECS Task Definition and Credential Management**

During installation, these values are passed into the ECS Task Definition.

* Credentials are temporarily stored in a file during installation.
* After installation, the credentials file remains in the EFS mount directory at /mnt/tmp.

{% hint style="warning" %}
Ensure that the EFS mount has proper access controls to prevent unauthorized access to AWS credentials.
{% endhint %}

#### S3 Bucket Configuration

* During Bridge Server installation, the **nifi.properties** file is updated with the following S3 details:
  * S3 Bucket Name
  * AWS Region
* These values are also passed as **environment variables** in the Bridge Server Docker **run.sh** file and defined in the ECS Task Definition. The **run.sh** script handles these updates.

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FegrGtK1C4TdBCRbNw0kL%2Funknown.png?alt=media&#x26;token=f74350da-d317-460a-a2d2-bbe22714c1a4" alt=""><figcaption></figcaption></figure>

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FAAuTkmOeF3PbL9yKaQmw%2Funknown.png?alt=media&#x26;token=34f6720a-d063-4632-b162-e702bc901c1c" alt=""><figcaption></figcaption></figure>
* After installation, the script ensures that the **nifi.properties** file contains the correct S3 bucket name and region.<br>

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FafmwYDIBxdSgkyag9pEj%2Fimage.png?alt=media&#x26;token=5c53a03f-d4c4-4ac9-850a-bb06322b416d" alt=""><figcaption></figcaption></figure>

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FaiJmVOyvXo10NJfkXcUU%2Funknown.png?alt=media&#x26;token=0bbea135-5251-475f-b9a6-6b304dbe1834" alt=""><figcaption></figcaption></figure>

**Bridge Server UI Changes**

* To configure S3 access through the UI, update the following parameters under S3PutObject > View Configuration:
  * S3 Bucket Name
  * AWS Region
  * AWS Access Key and Secret Key

{% hint style="info" %}

* Always verify that the credentials used have S3 PutObject permissions.
* Changes made via the UI should match the configuration in nifi.properties to prevent synchronization issues.
  {% endhint %}

<div align="left"><figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2F7Ng1GvXRVK1FtIgupN4t%2Funknown.png?alt=media&#x26;token=a22dec41-e65d-467f-bf79-84f593313a3a" alt=""><figcaption></figcaption></figure></div>

**Bridge Client Installation**

* During Bridge Client installation, the S3 bucket name and region are retrieved from the Bridge Server’s **nifi.properties** file.
* The script **generate\_bridge\_client.sh** is responsible for applying these changes.<br>

  <figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FIAOBgR8IjkFzTaFvMnQz%2Funknown.png?alt=media&#x26;token=68d52789-5ce7-4ca6-b00e-e0e517b48ff0" alt=""><figcaption></figcaption></figure>

The Bridge Client’s **nifi.properties** file is updated with the same S3 configuration.

<figure><img src="https://1813356899-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FhTnkoJQml0pok9awFDhx%2Fuploads%2FTOPZKCFBYdmZ2uUYPQGm%2Funknown.png?alt=media&#x26;token=f4b710d8-7159-4374-907f-a4ca1c076d0f" alt=""><figcaption></figcaption></figure>

**Key Change Summary:**

* **Before**: Bridge Client certificate ZIP contained only certificates.
* **After**: Bridge Client certificate ZIP now includes:
  * AWS Credentials (Access Key & Secret Key)
  * S3 Bucket Name
  * AWS Region

This ensures the Bridge Client can directly connect to S3 using the provided credentials and configuration.

**Bridge Client UI Changes**

To configure S3 access through the UI, update the following parameters under **S3PutObject > View Configuration**:

* S3 Bucket Name
* AWS Region
* AWS Access Key and Secret Key

{% hint style="info" %}

* Ensure that the Bridge Client certificate ZIP file is stored securely.
* Do not share AWS credentials externally; they are sensitive and allow direct access to S3.
* Always confirm that the S3 bucket exists and the region is correct before starting the Bridge Client.
  {% endhint %}

***

Copyright © 2025, OvalEdge LLC, Peachtree Corners, GA, USA.