Upload File or Folder

OvalEdge supports secure file and folder transfers from local devices using the built-in upload utility. The utility integrates with Network File System (NFS) to enable centralized file management and support collaborative workflows. It allows uploading of individual files or entire folders as part of the data onboarding process.

Upload Files or Folders

Users can transfer files from their devices to OvalEdge using the built-in Upload File or Folder tool, available under Advanced Tools. This tool leverages a Network File System (NFS) connection to enable seamless file management and collaboration.

Uploaded files and folders are stored securely and appear in the File Manager module. Administrators can further organize and catalog these assets within the Data Catalog for improved accessibility and governance.

The upload feature also creates secure backups on a remote server, helping protect against device failures and data loss. Additionally, it supports collaboration through network-based file sharing.

Upload Process

The upload process consists of three steps:

1. Select Data Lake: Choose the destination Data Lake connection for the upload.

2. Select Your Directory: Browse and specify the file or folder on your device to upload.

3. Upload: Initiate the upload to transfer the selected files or folders to the chosen destination.

Select Data Lake

OvalEdge's Data Lake is a central hub for all structured and unstructured data. Users can explore available file connections for upload on the "Select Data Lake" page. The Connection Type allows users to easily find specific connection types and quickly locate the appropriate Data Lake for their upload.

Choosing the Storage Location

To initiate the upload, users select a connection name, such as "NFS - A," which indicates the storage location. This selection uses the Network File System (NFS) protocol.

Upon selecting the Data Lake, users proceed to the Select Your Directory tab to specify the target directory for the file upload. This step ensures accurate placement and efficient data transfer.

Select Your Directory

The Data Lake offers organized directories for file uploads. Users can:

• Choose an Existing Directory: Select a directory linked to the chosen NFS connection for a streamlined upload that aligns with the specific data needs.

• Create a New Directory: Use the "Create Directory" option in the 9 Dots menu to create a new directory with a unique name.

  • Users can create a new directory at either the first level or within an existing directory (at the second level).

    • To create a first-level directory: Enter the directory name in the Enter Directory Name text box and click OK. The new directory appears under the Select Your Directory tab.

    • To create a second-level directory: Select an existing folder, click Create New Directory, enter the directory name, and click OK. The new directory will be created under the selected folder.

Once the directory is selected, users proceed to the final step: Upload File or Folder.

Upload File or Folder

The final tab lets users choose:

• Single File: Upload a single file for specific needs.

• Entire Folder: Upload a whole folder for efficient data transfer.

• Maximum file size: Uploads are limited to 5 MB for optimal performance.

• File types: Users should ensure compatibility by uploading supported types (e.g., CSV, CONF, ENV).

Selecting Files or Folders:

Users can select files or folders from their device using the Select from your computer button. Clicking this button opens a File Explorer window, where users browse, select the required file or folder, and click the Open button.

The selected file or folder name is then displayed on the Upload File or Folder tab. Clicking the Finish button completes the upload step and returns the user to the Select Your Directory page.

To view the uploaded file or folder, users can navigate to the File Manager module. The List View displays a detailed summary organized by directory.

Supported File Formats for Upload

The Upload File or Folder tool supports a predefined set of file types. Users can configure the allowed file formats by setting the config.file.types.to.be.cataloged option, available under the File Manager tab in System Settings. Ensure that only valid and supported file formats are selected for upload.

Supported File Formats for Profiling

Once files are uploaded and cataloged in the Data Catalog, the following file formats are supported for profiling:

• CSV (.csv, .txt): Comma-Separated Values format used to store tabular data, where each line represents a record and fields are separated by commas.

• TXT (.txt): Plain text file containing unformatted textual data, often used for notes, logs, or documentation.

• JSON (.json): JavaScript Object Notation format used for representing structured data, enabling easy data interchange between applications and servers.

• Parquet (.parquet): Apache Parquet is a columnar storage file format optimized for efficient storage and processing of large-scale datasets.

• ORC (.orc): Optimized Row Columnar format used in the Hadoop ecosystem for high-performance storage of structured data.

• XLSX (.xlsx): Microsoft Excel Open XML Spreadsheet format that supports advanced features like formulas, charts, and formatting.

• XLS (.xls): Microsoft Excel binary file format used for storing spreadsheet data in rows and columns.

• Avro (.avro): Apache Avro is a data serialization system that supports efficient data exchange and schema evolution.

• Gzip (.gz): Compressed file format using the Gzip compression algorithm to reduce file size and improve transmission speed.

• CONF (.conf): Configuration file used to define parameters and settings for software applications or systems.

• ENV (.env): Environment variable file that stores key-value pairs for configuring application environments.

• SH (.sh): Shell script file used to automate command-line tasks in Unix/Linux environments.

• PROPERTIES (.properties): Java properties file used to store configuration settings as key-value pairs, typically in Java-based applications.

• YAML (.yaml): Human-readable data serialization format used for configuration files and data exchange, often in DevOps and cloud-native environments.

• DDL (.ddl): Data Definition Language script used to define or modify database structures such as tables, schemas, or indexes.

• SQL (.sql): Structured Query Language file containing commands for querying or manipulating relational databases.

• HQL (.hql): Hive Query Language file used for querying data in Apache Hive environments.

• PDF (.pdf): Portable Document Format used to present documents consistently across different platforms, preserving layout and formatting.

• PNG (.png): Portable Network Graphics format for lossless image compression, commonly used for graphics and screenshots.

• JPG / JPEG (.jpg, .jpeg): Common compressed image file formats used for photographs and digital images.

• SCREENSHOT (.screenshot): Image file representing a captured view of the screen; typically saved in standard image formats like PNG or JPEG.

Upload File or Folder System Settings

System settings for the Upload File or Folder tool (Administration > System Settings > File Manager tab) allow administrators and users to control the behavior and display. These settings allow users to:

• Control Maximum Files per Upload (ovaledge.fileupload.maxfiles): Set the maximum number of files allowed for upload (default: 10).

• Specify Maximum File Size (ovaledge.filesize.limit): Define the largest file size (in bytes) permitted for upload (default: 5242880 bytes, equivalent to 5 MB).