Documentation
Set up. Run. Verify.
Practical guides for connections, data movement and recovery in PipePilot.
Start with your first pipelineYou need workspace access, reachable endpoints and permissions for your chosen load.
Browse all connectorsCreate your first pipeline
Begin with a small representative table, then verify its mapping and result before adding more data.
Choose execution
Use a Private Agent on your Linux host or assigned PipePilot Compute. Both connections must use the same execution identity.
Save and test connections
In Connections, choose the engine and role, enter the endpoint and credentials, and test access. Confirm source read/capture and destination write permissions.
Choose the movement mode
Use Batch for manual or scheduled loads, CDC for continuous changes, or Migration for a one-time copy. Confirm each endpoint’s available options during connection setup.
Map and verify
Select one source table and review the destination name, columns and keys. Run the pipeline, inspect the table outcome and verify the destination data.
Configure a batch load
Full loads copy selected data. Incremental loads use a saved watermark and merge rule.
Select tables or a supported query
Discover tables from the source connection and choose the fields you need. Query mode is engine-specific; MongoDB uses collection mode.
Choose the load rule
For incremental work, select a watermark that advances when relevant rows change and stable merge keys. Changes that do not advance the watermark can be missed.
Review the target
Keep the new pipeline’s own destination table, or explicitly choose Map to an existing target table. Review write permissions and column mappings.
Run and reconcile
Run manually before scheduling. Inspect table progress and the latest independent source/target count audit where supported. Equal counts do not prove equal values.
Set up continuous capture
CDC reads database change logs. Prepare logging and capture permissions before creating the stream.
Prepare the database
Enable the engine’s required change logging. PostgreSQL needs logical replication and publication access. Follow the engine-specific setup checks to confirm readiness.
Select scope and destination
Choose the source tables and destination schema, topic or stream. Review keys and update/delete behavior. Event topics and streams must already exist.
Review landing behavior
Warehouse and event destinations preserve changes. A healthy landing stream does not mean updates and deletes have been merged into a final business table.
Verify delivery
Make a controlled source change, confirm capture and delivery, and inspect the destination record. Use lag and task evidence to locate a stopped flow.
Plan a one-time migration
Migration copies data. Stored procedures, application logic and continuous change capture are separate work.
Select scope
Choose available source tables, schemas or database scope. Confirm the destination supports the selected runtime.
Map explicitly
Review each source schema and table against its target schema or dataset. Approve schema creation only where offered. Object storage uses dataset prefixes.
Check target ownership
Use new unoccupied targets. Setup checks ownership and publication safeguards before writing; an existing target is not blanket permission to overwrite it.
Validate the copy
Inspect per-table completion and verify the copied data. Review application cutover and any subsequent change capture separately.
Connect a Private Agent
The Agent runs on your Linux host and must reach both endpoints.
Prepare the host
Use a supported Linux host with the required network routes and capacity. Allow outbound access to the PipePilot control plane.
Install from Private Agents
Follow the installation instructions generated in your workspace. Keep enrollment material private and confirm the Agent is connected.
Assign both connections
Select the same Agent for source and destination, even for a public destination. Connections cannot bridge an Agent and managed Compute.
Verify access
Test both endpoints from the chosen execution environment. Review Agent status and the available update action for connectivity or compatibility issues.
Diagnose and recover a run
Review the affected run and table before retrying or changing a load rule.
Locate the failed layer
Check execution availability, connection validation, source reads, destination writes and table outcomes. A queued run may be waiting for its execution environment.
Fix the cause
Restore network access or permissions, correct a mapping, or make the selected execution environment available. Preserve the run details for support.
Use available recovery controls
Retry or resume through pipeline controls where offered. Review the checkpoint; a failed run may already have written data.
Review discrepancies
For a count mismatch, inspect the cutoff, watermark and merge keys. Eligible single-table repair requires confirmation; recheck the audit afterward.
Next step
Plan your data workflow.
Share your source, destination and network needs. We will confirm the supported workflow and prerequisites.