Ingestion Control Center

System Standby

Total Records Ingested

0

Live ingestion counter

Throughput Speed

0/s

Records per second

Ingestion Latency

0ms

Network hop response

Throughput Realtime Waveform

Active Data Packets

No active ingestion packets. Start adapters under the "Data Ingestion" tab.

Syncoriq Enterprise Connectors & Automated Pipelines Hub

Ingestion Engine Online

Provision pre-built ingestion connectors to sync datasets from cloud databases, storage buckets, or SaaS platforms. Ingested data is monitored for CortIQ-style schema evolution and incremental Change Data Capture (CDC).

Active Sync Pipelines (CDC & Schema Evolution)

Connector Target Silver Table CDC Tracking Cursor Last Sync Offset Actions
No connectors configured. Setup a connection above to establish a live pipeline.

Pre-built Distillation Pipelines (Syncoriq Gold Models)

Deploy out-of-the-box analytical models (dbt-style structures) to aggregate and clean synced database schemas into Gold-level XDS tables.

Ingestion Execution Logs & Schema Drift Telemetry
[14:40:00] INGESTION Syncoriq Schema Drift and Incremental CDC watcher listening...

Business Intelligence

Powered by Apache Superset and Snowflake Datawarehouse.

SYNCORIQ MLOps Hub

End-to-End Data Science & Machine Learning Lifecycle.

Master Data Store (MDS)

Connected

Source: Fabric Tab

Transaction Data Store (XDS)

Connected

Source: Fabric Tab

Transformation Engine (TX)

Online

Source: Fabric Tab

Snowflake Datawarehouse

Connected

CORTIQ_DB.PUBLIC

Automated EDA & Preprocessing

Missing Values Matrix

Select a dataset and run profiling to view missingness ratios.

Pearson Correlation Heatmap

Select a dataset and run profiling to view correlation scores.

Feature Engineering

Target Column: churn_flag

Map feature scaling (Standard, MinMax) and categorical encoders (One-Hot, Ordinal) per column.

Dedicated System Terminal (Log History)
Console pipeline initialized. View logs in real-time under the Ingestion tab or trigger Push/Pull processes.

Kafka Cluster Brokers

2 Active

broker-01, broker-02

Active Database Topics

0

Isolated db streams

Kafka Message Throughput

0

Total queued offset

Topic Ingestion Pipelines

Topic Name Partitions Offset Consumers
No active topics. Run ingestion to register pipelines.

Kafka Partition Queue Monitor

Waiting for ingestion records to stream into Kafka partitions...

Log-Based CDC Events (Write-Ahead Database Storage Triggers)

Active SQLite update_hook trigger capture
Timestamp Source Table Operation Record Row ID Transaction Payload
Waiting for storage events to trigger CDC updates...
Kafka Live Message Broker Stream (Topic Consumer Log)
Kafka message broker connected. Waiting for adapter payloads to write to source topics...

Schema Mapping Templates

Persisted in Local Storage

Target Database Schema Builder

No target schemas registered yet. Create a table above.

Many-to-One Mapping Rules

No ingested source fields found. Run ingestion to fetch columns.
Batch Rule Queue 0 rules staged
|
Queue is empty. Use "Add to Batch Queue" above to stage multiple rules before committing.
Active Mapping Rules Registry
Target Table Target Column Source Database Columns (Many-to-One) Delimiter
No mapping rules configured yet. Establish rules in the builder above.

Batch Execution — Select Target Tables

|
No target tables registered. Create tables in the Schema Builder above.

Interactive Schema Mapping Canvas (Visual Schema Linker)

Source Fields
Target Columns

Data Quality & Policy Rules Configurator

Active Validation Rules:

DQ Violations & DLQ Logs

DLQ (Blocked): 0
Warnings: 0
Real-Time Validation Violations Log
Col Rule Value Action
No quality violations logged.
Mapping Application pipeline execution
Transformation log active. Establish rules and click "Run Mapping Job" to execute processing.
Unified Consolidated Database Inspector 
/* Target database query return: Empty. Execute mapping job to view compiled datasets. */

Master Data Store (MDS) Catalog

No MDS schemas available.

Transformation Data Store (XDS) Catalog

No XDS tables generated. Run a Transformation Job first.

Transformation Job Builder (TX)

Staged Redshift TX Jobs (SQL)
No jobs staged.
Lakehouse Engine Console
Engine standing by...
XDS Output Inspector 
/* Target XDS query return: Empty. Execute TX job to view data. */

Simulated Distributed Scale-Out Cluster (Multiprocessing Pool)

Active Worker PIDs: Idle
Configure Spark workers above and click "Run All Staged Jobs" to spin up parallel execution nodes.

Delta ACID Commit History (Time Travel & Rollback)

Ver Op Records Timestamp
Select a table to view Delta commits history.

Unified Data Catalog & Enterprise Assets

Asset Name Type Steward / Data Owner

Snowflake Dataset Builder

Select Source Table

Select Columns for Dataset

Select a source table first to view its columns.

Dataset Construction Payload

/* Configure your dataset above */

Push Ingestion Auth Secrets

  • Simple Auth Key
    cq_pk_live_8f3...
    •••••••••••••••••••••••••••••••• cq_pk_live_8f3d62ea0b912c4c
  • OAuth 2.0 Client Secret
    sec_prod_8f3c...
    •••••••••••••••••••••••••••••••• sec_prod_8f3c2e11a2f64e10b411d9f8e
  • JWT HMAC Secret Key
    key_sig_jwt...
    •••••••••••••••••••••••••••••••• key_sig_jwt_hmac256_prod_env

Pull Ingestion Auth Secrets

  • Simple Auth Key
    cq_pk_live_0a3...
    •••••••••••••••••••••••••••••••• cq_pk_live_0a3f91bc8d62ef83
  • OAuth 2.0 Client Secret
    sec_prod_4a8...
    •••••••••••••••••••••••••••••••• sec_prod_4a8d3f11e9b24f60c511d7f2a
  • JWT Fixed Token
    eyJhbGciOiJ...
    •••••••••••••••••••••••••••••••• eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzb3VyY2UiLCJleHAiOjE4MDAwMDAwMDB9.sig_f89c2a

Airflow Workflow Orchestrator

SCHEDULER ACTIVE

The Workflow Orchestrator automatically monitors ingestion sequences and triggers downstream transformations as sequential DAGs.

1. Monitor Kafka
Awaiting Trigger
2. Push to Mapping App
Pending
3. Populate MDS
Pending
4. Transform MDS to XDS
Pending
5. Populate Snowflake
Pending
Active Run ID: None

Job Manager Console

Job ID Type / Trigger Status Actions
No jobs currently registered in the orchestrator.