Job Workflow Configuration

Job Workflows lets users automate and organize tasks like crawling, profiling, and data quality checks. They consist of sequences that can run one after another (sequentially), simultaneously (in parallel), or a combination of both.

Functionality:

• Multiple job types can be used to configure and create workflows.

• OE_Admin users can create, edit, and execute workflows.

• Viewers can only select, design, and save workflows.

OE_Admin and Author users can create Job Workflows.

Steps:

1. Navigate to Administration > Job Workflow.

2. Click the "Add New Job Workflow" button (plus icon).

3. Select a Job Type from the available options.

4. Design the Workflow: Drag and drop job types to define the execution sequence (sequential, parallel, or combination).

5. Save the Workflow Template.

6. (For OE_Admin only) Execute the workflow to run the automated tasks.

Job Type Selection

When designing a Job Workflow, select job types from the dropdown menu on the left.

OvalEdge supports various job types for building workflows:

• Data Acquisition:

  • Crawler Connector: Connect to external data sources.

  • SQL: Execute SQL queries.

  • Crawler: Extract data from web sources.

• Data Processing:

  • Profile: Analyze data characteristics.

  • Profile Unprofiled: Profile new or unprofiled data.

• Data Quality:

  • Anomaly Detection: Identify data anomalies.

  • Data Quality Rules: Enforce data quality standards.

• User Management:

  • Crawl Users, Roles, and Permissions: Extract user and permission data.

• Other:

  • Job Workflow: Integrate other workflows into the design.

  • Advanced Job: Execute custom scripts or programs.

  • Alert: Generate notifications upon completion or errors.

  • Process New/Changed Queries: Process newly added or modified queries.

Design Workflow

The Job Workflow design area allows users to define the execution sequence:

• Sequential: Jobs run one after another, with each job starting only after the previous one finishes.

  
• Parallel: Jobs run simultaneously, independent of each other.

  
• Hybrid: Combine sequential and parallel execution for flexible workflows.

  

Design Tips:

• Drag and drop job types in the desired order to visually arrange them.

• The options on the top right of the screen can further assist users in designing workflows.

  • Adding/Removing Rows/Columns: Use the "+" or "-" icons to adjust the number of Rows or Columns within the workflow.

  • Saving Workflows: Once the workflow design is configured and a name is provided, click "Save" to save the Job Workflow.

  • Scheduling Workflows: Job Workflows let users automate task execution at specific times or intervals. This reduces manual intervention and streamlines repetitive processes. Here's how:

• Scheduling: OE_Admin users can schedule Job Workflows using the Cron job scheduler. If no schedule is set, a default one applies.

  • Benefits:

    • Automation: Scheduled workflows run automatically, saving time and effort.

    • Notifications: Users receive alerts upon job completion (success, failure, partial success).

    • Example: Schedule AI Term Recommendations to run every month on the 5th at 2:10 AM (format: Month/Day/Hour/Minutes).

• Job Workflow Experts: Users who create, understand, and have business context for a Job Workflow's tasks.

  • Notifications: Receive updates on the workflow's execution status.

  • Collaboration: Multiple Author and Viewer license users can be added as experts.

  • Viewer Limitations: The user with a viewer license can only receive Job Workflow notifications in their Inbox.

• Run-on Error Option: Allows experts to restart the entire workflow if an error occurs within any job.

Job Workflow Design Options

The 9 Dots menu on the Job Workflow design page provides various actions:

• New Job Workflow (✚): Create a new workflow. Before creating a new one, confirm whether the user wants to save the current Workflow.

• Job Workflow History: View the execution history of jobs within the current Workflow. Choosing the History option directs users to the Jobs page, enabling them to track job statuses within a workflow.

• Job Workflow Notifications: Job Workflow creators and experts can set notification preferences for errors, success, or partial success of the workflow execution. Notifications are delivered to Inboxes of OE_Admin users and Job Workflow experts. There are three options to configure:

  • Error: Receive alerts whenever an error occurs during any job within the workflow.

  • Success: Get notified upon successful completion of the entire workflow execution.

  • Partial Success: Be informed if the workflow execution achieves partial success, meaning some jobs might have failed.

  • Notification Details: When viewed in the inbox, these notifications provide essential details like:

    • Executed Job Status

    • Job ID

    • Job Name

    • Job Execution Timestamp

• Job Workflow Executions: The 'Job Workflow Executions' option allows users to fetch and review all historical executions associated with a specific Job ID. This feature provides a comprehensive log of each run instance, including execution status, timestamps, duration, executed user details, and any errors encountered during processing. The Job Workflow Executions associated with the specific Job ID have the following columns:

  • Job Workflow Execution ID: It refers to a unique identifier assigned to each individual run of a job.

  • Job ID: A unique identifier assigned to each job to track and manage its executions.

  • Total Job Steps Count: It indicates the total number of individual steps executed within a specific job run.

  • Executed Jobs Steps Count: This indicates the total number of steps executed during a specific job run.

  • Status: It indicates the current or final state of a job execution, such as Success, Failed, or In Progress.

  • Success: It displays the count of Jobs that fall under the ‘Success’ category.

  • Partial Success: It displays the count of Jobs that fall under the ‘Partial Success’ category.

  • Error: It displays the count of Jobs that fall under the ‘Error’ category.

  • Killed: It displays the count of Jobs that are killed.

  • Skipped: It displays the count of Jobs that are skipped.

  • Execution Method: This indicates how the job was triggered—either manually or scheduled.

  • Start Time: It displays the exact timestamp when a specific job execution began.

  • End Time: It displays the exact timestamp when a specific job execution was completed.

  • Duration: It displays the total time taken to complete each job execution.

  • Executed By: It indicates the user who initiated the execution of the job.

  • Re-Execute: It allows users to manually rerun a specific past job execution directly from the list.