This document defines the policies and procedures governing how DCKAP Integrator processes data after it has been ingested from source systems — including transformation, routing, error handling, and delivery to destination systems — with controls ensuring accuracy, timeliness, and consistency.
2. Processing Architecture
Ingestion
Data received via API, webhook, or scheduled pull from source systems
Queue
Records placed in a durable message queue to ensure no data loss during processing
Transformation
Field mapping, type conversion, value translation, enrichment applied per customer configuration
Routing
Records routed to correct destination based on business rules
Delivery
Records posted to destination system API with acknowledgement tracking
Logging
Every step recorded in the audit log; errors captured with full context
3. Processing Policies
3.1 Accuracy Policy
All transformation rules are version-controlled and validated before activation.
Transformation outputs are checksummed and compared against expected output in test mode.
Any transformation resulting in data loss (null output for non-nullable destination field) triggers an error and halts processing for that record.
3.2 Timeliness Policy
Sync Type
Target Processing Time
Alert Threshold
Real-time / Event-triggered
< 60 seconds end-to-end
> 120 seconds
Scheduled (standard)
< 5 minutes per batch
> 10 minutes
Bulk / Large batch
< 30 minutes per 50,000 records
> 60 minutes
3.3 Consistency Policy
Each record is processed exactly once — the queue is deduplicated before transformation begins.
Idempotency keys are used when posting to destination APIs to prevent duplicate writes.
Processing state is checkpointed so that interrupted jobs resume from the last successful record, not from the beginning.
3.4 Error Handling Policy
Processing errors are classified as: Transient (retry), Validation (reject), or Fatal (halt and alert).
