Batch pipelines
Schedule full and incremental loads.
Build manual, hourly, daily or weekly loads from reusable connections. Set each table’s behavior, review every run and check supported results after publication.
Core capabilities
Full loads
Create a controlled destination baseline and publish only after the staged write succeeds.
Incremental loads
Use a watermark and merge keys, with an optional late-arrival reread window.
Post-load checks
Compare fresh source and final-target counts for supported table-mode loads.
Data reconciliation
Catch discrepancies after each load.
For supported table loads, PipePilot compares fresh source and final-target row counts after publication. The dashboard flags a mismatch or measured rule risk for review; equal counts remain a limited signal, not proof that every value is correct.
Review the recorded counts, cutoff, watermark and merge keys before choosing the next action.
- 01Compare countsFresh source vs final target
- 02Review the ruleWatermark, merge key and cutoff evidence
- 03Approve a repairEligible table reload requires confirmation
- 04RecheckA new independent audit records the result
PipePilot does not run Assistant-generated SQL or automatic repairs. A published load is not failed solely because its audit is unavailable.
Workflow
01Configure, run and checkpoint.
Choose connections and tables, set the load mode, map the target and select a schedule. An incremental pipeline starts with a full baseline and advances its checkpoint only after a successful write.
Per-table load mode and target mapping
Watermark and merge-key validation
Manual, hourly, daily and weekly schedules
Queued, running, succeeded, failed and cancelled states
Configuration and runtime evidence stay attached to the workflow.
Reconciliation
02Review discrepancies before repair.
A post-load check records independent counts, scope, committed cutoff, watermark and merge keys. Ask PipePilot explains measured evidence and possible causes without changing data.
Equal counts do not prove row contents match
Refresh reads saved evidence; a new load or repair runs a new check
Confirmed mismatches can offer a reviewed single-table repair on eligible targets
Repair republishes that table and rechecks counts without changing sibling tables or saved watermarks
Configuration and runtime evidence stay attached to the workflow.
Next step
Plan your data workflow.
Share your source, destination and network needs. We will confirm the supported workflow and prerequisites.