Ingesting Data from Your Pipeline
You don’t need a native warehouse connector to keep Sequenzy in sync with Snowflake, BigQuery, Redshift, or any other source of truth. The API is designed so an existing DAG - an Airflow task, a Dagster job, a dbt post-hook, or a plain cron script - can push contacts and events in bulk, re-run safely after failures, and never create duplicates. This guide covers the endpoints a pipeline needs, how deduplication works on each of them, and how to structure a task that moves large volumes of data within the API rate limits.Endpoints at a Glance
All endpoints use API key authentication and share the
rate limits of 100 requests per
minute and 20 requests per second per key.
Why Re-Runs Are Safe
Every ingestion path deduplicates, so your retry strategy can simply be “re-run the whole task”:- Contacts match on
email,externalId, orphone. You choose what happens on a match withduplicateStrategy:skip(default),merge, oroverwrite. Running the same import twice never creates a second contact. - Events accept your own
eventIdas an idempotency key. A retried event with the same ID for the same contact and event name is acknowledged withduplicate: trueand records nothing. - Historical contacts are those whose rows in a request are all more than an hour old. Their events appear in analytics and segments without firing automations or double-opt-in mail. If any row for a contact is recent, that contact’s whole group is live; split live and historical extracts when you need the silent path.
Syncing Contacts at Scale
Use the bulk import endpoint with 5,000-record batches. At the standard rate limit that sustains roughly 500,000 contact rows per minute, so contact volume is effectively unconstrained for a scheduled sync. A typical incremental task:- Query your warehouse for rows changed since the last successful run (a
simple
updated_at > :watermarkworks; you own the watermark). - Chunk the rows into batches of up to 5,000.
- POST each batch with
duplicateStrategy: "merge"so existing contacts are updated in place. - Poll each returned import ID until
statusiscompleted, then advance your watermark.
subscriber_import.completed. It has one stable logical event ID per import,
and transient enqueue failures are repaired automatically. HTTP delivery is
retried, so consumers should deduplicate by event ID as usual.
If you prefer polling, confirm every import finished cleanly before advancing
your watermark:
Counts on the status endpoint reconcile exactly:
addedCount + updatedCount + skippedCount + failedCount == totalRows. Inspect skippedReasons and
failedReasons before treating a run as successful - a duplicate_csv_row
skip means two rows in your own extract carried the same email, phone, or
external ID.Preserve Signup Dates
Pass each contact’s original signup date increatedAt. Date-relative
segments and analytics then reflect real history instead of the day your
pipeline first ran. An existing contact’s date only ever moves earlier, so
repeated syncs cannot corrupt it.
Do Not Enroll Backfills in Automations
Bulk imports do not fire signup automations by default. LeaveenrollInSequences off for warehouse syncs unless you explicitly want
imported contacts to enter matching sequences.
Recording Events at Scale
For behavioral data (purchases, product usage, lifecycle events), use the events import endpoint: bounded batches of up to 25 events per request, each carrying its own contact identity. Every event must have a deterministiceventId derived from your source data - an order ID, a
warehouse row key, or a hash of the row - so re-running the task records
nothing twice:
Handling Rate Limits
When you exceed the limit the API returns429 Too Many Requests with a
Retry-After header. An overlapping subscriber-import request using the same
idempotency key can similarly return a retryable 503 while the first request
still owns its short enqueue lease. For pipeline work:
- Honor
Retry-Afteron429and retryable503responses, then resend the same request. Idempotency makes the resend safe even if the original was actually processed. - Keep the task single-threaded or coarsely parallel. One worker sending 5,000-row import batches already saturates the useful throughput; extra concurrency only spends the budget faster.
- Fail the task on anything other than
429and re-run it from the top. You do not need checkpoint bookkeeping for correctness - only for speed.
Syncing Data Out
For the reverse direction - warehouse tables fed from Sequenzy - schedule a task that pulls aggregated metrics, or subscribe to outbound webhooks and land the payloads in your ingestion bucket.Checklist
- Store the API key in your pipeline’s secret manager, never in DAG code.
- Sync contacts with 5,000-record batches and
duplicateStrategy: "merge". - Pass an
idempotencyKeyper batch so timed-out requests can be resent. - Pass
externalIdon every record so identity survives email changes. - Pass
createdAt(contacts) andoccurredAt(events) from source data. - Derive
eventIddeterministically from your source rows. - Subscribe a webhook to
subscriber_import.completed, or poll import status, and reconcile counts before advancing your watermark. - On failure, re-run with the same IDs: receipts deduplicate and downstream recovery is re-attempted idempotently.