Skip to main content

Ingesting Data from Your Pipeline

You don’t need a native warehouse connector to keep Sequenzy in sync with Snowflake, BigQuery, Redshift, or any other source of truth. The API is designed so an existing DAG - an Airflow task, a Dagster job, a dbt post-hook, or a plain cron script - can push contacts and events in bulk, re-run safely after failures, and never create duplicates. This guide covers the endpoints a pipeline needs, how deduplication works on each of them, and how to structure a task that moves large volumes of data within the API rate limits.

Endpoints at a Glance

All endpoints use API key authentication and share the rate limits of 100 requests per minute and 20 requests per second per key.

Why Re-Runs Are Safe

Every ingestion path deduplicates, so your retry strategy can simply be “re-run the whole task”:
  • Contacts match on email, externalId, or phone. You choose what happens on a match with duplicateStrategy: skip (default), merge, or overwrite. Running the same import twice never creates a second contact.
  • Events accept your own eventId as an idempotency key. A retried event with the same ID for the same contact and event name is acknowledged with duplicate: true and records nothing.
  • Historical contacts are those whose rows in a request are all more than an hour old. Their events appear in analytics and segments without firing automations or double-opt-in mail. If any row for a contact is recent, that contact’s whole group is live; split live and historical extracts when you need the silent path.

Syncing Contacts at Scale

Use the bulk import endpoint with 5,000-record batches. At the standard rate limit that sustains roughly 500,000 contact rows per minute, so contact volume is effectively unconstrained for a scheduled sync. A typical incremental task:
  1. Query your warehouse for rows changed since the last successful run (a simple updated_at > :watermark works; you own the watermark).
  2. Chunk the rows into batches of up to 5,000.
  3. POST each batch with duplicateStrategy: "merge" so existing contacts are updated in place.
  4. Poll each returned import ID until status is completed, then advance your watermark.
Instead of polling, you can subscribe an outbound webhook to subscriber_import.completed. It has one stable logical event ID per import, and transient enqueue failures are repaired automatically. HTTP delivery is retried, so consumers should deduplicate by event ID as usual. If you prefer polling, confirm every import finished cleanly before advancing your watermark:
Counts on the status endpoint reconcile exactly: addedCount + updatedCount + skippedCount + failedCount == totalRows. Inspect skippedReasons and failedReasons before treating a run as successful - a duplicate_csv_row skip means two rows in your own extract carried the same email, phone, or external ID.

Preserve Signup Dates

Pass each contact’s original signup date in createdAt. Date-relative segments and analytics then reflect real history instead of the day your pipeline first ran. An existing contact’s date only ever moves earlier, so repeated syncs cannot corrupt it.

Do Not Enroll Backfills in Automations

Bulk imports do not fire signup automations by default. Leave enrollInSequences off for warehouse syncs unless you explicitly want imported contacts to enter matching sequences.

Recording Events at Scale

For behavioral data (purchases, product usage, lifecycle events), use the events import endpoint: bounded batches of up to 25 events per request, each carrying its own contact identity. Every event must have a deterministic eventId derived from your source data - an order ID, a warehouse row key, or a hash of the row - so re-running the task records nothing twice:
At 25 events per request within the standard request limit, that sustains up to 2,500 events per minute while keeping each synchronous request bounded. Only a contact whose rows in the request are all more than an hour old uses the silent historical path; any recent row makes that contact’s whole group live. Separate live and historical rows when a backfilled purchase must not trigger a “thanks for your order” sequence. For single fresh events (a checkout happening right now) where you do want automations, use the single-event endpoint.

Handling Rate Limits

When you exceed the limit the API returns 429 Too Many Requests with a Retry-After header. An overlapping subscriber-import request using the same idempotency key can similarly return a retryable 503 while the first request still owns its short enqueue lease. For pipeline work:
  • Honor Retry-After on 429 and retryable 503 responses, then resend the same request. Idempotency makes the resend safe even if the original was actually processed.
  • Keep the task single-threaded or coarsely parallel. One worker sending 5,000-row import batches already saturates the useful throughput; extra concurrency only spends the budget faster.
  • Fail the task on anything other than 429 and re-run it from the top. You do not need checkpoint bookkeeping for correctness - only for speed.

Syncing Data Out

For the reverse direction - warehouse tables fed from Sequenzy - schedule a task that pulls aggregated metrics, or subscribe to outbound webhooks and land the payloads in your ingestion bucket.

Checklist

  1. Store the API key in your pipeline’s secret manager, never in DAG code.
  2. Sync contacts with 5,000-record batches and duplicateStrategy: "merge".
  3. Pass an idempotencyKey per batch so timed-out requests can be resent.
  4. Pass externalId on every record so identity survives email changes.
  5. Pass createdAt (contacts) and occurredAt (events) from source data.
  6. Derive eventId deterministically from your source rows.
  7. Subscribe a webhook to subscriber_import.completed, or poll import status, and reconcile counts before advancing your watermark.
  8. On failure, re-run with the same IDs: receipts deduplicate and downstream recovery is re-attempted idempotently.