Data Sync Patterns

Sync Modes

Backfill Pattern

1. Start with a minimal field set

   Start with the smallest set of fields that proves the pipeline works.

   You can prove the pipeline end-to-end before chasing edge cases.
2. Adopt cursor pagination

   Use cursor pagination wherever the endpoint supports it.

   The walk is deterministic and can resume after a failure.
3. Persist the checkpoint alongside the write

   Persist both the downstream write result and the last successful cursor checkpoint.

   The next run restarts exactly where the last left off.
4. Expand the field set

   Only after the happy path is stable, widen the field set.

   You can load new fields without risking the existing load.

let cursor: string | null = null

while (true) {
const url = new URL("https://api.lev.com/api/external/v2/deals")
url.searchParams.set("limit", "100")
if (cursor) url.searchParams.set("cursor", cursor)

const response = await fetch(url, {
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "X-Origin-App": "warehouse-sync",
  },
})

const payload = await response.json()
await writeBatch(payload.data)

if (!payload.pagination?.has_more) break
cursor = payload.pagination.next_cursor
}

Incremental Pattern

Use an updated_at or similar time-based filter together with a persisted checkpoint:

• Store the timestamp of the last fully successful run.
• Re-read a small overlap window to tolerate clock skew and delayed writes.
• Deduplicate in your destination using resource IDs.

Syncing People

Mirroring deals into a CRM usually means mirroring the humans attached to them, not only the company IDs. Two includes on GET /deals cover that in the same page you already walk, so neither costs an extra request per deal:

• ?include=team embeds your own account's team members on the deal.
• ?include=sponsor_contacts embeds the people at the deal's sponsor company (see Deals).

curl -X GET "https://api.lev.com/api/external/v2/deals?limit=100&include=sponsor_contacts" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "X-Origin-App: warehouse-sync"

Both embeds are capped per deal, at 50 team members and 25 sponsor contacts. Each cap is applied against a deterministic order, so a truncated list is a stable slice and not a random sample. Do not assume the embedded list is exhaustive on an unusually large deal.

Two more properties matter once the rows land downstream:

• An empty list is an answer, but confirm which kind. The key is always present when the include is requested. Usually [] means the deal has no sponsor company, or that company has no connected sponsor contacts, and clearing the destination is correct. But a deal shared to you from another account also embeds [], because contacts are scoped to the deal's owning account and to yours at the same time. Clearing on that one deletes rows your key was never allowed to read. Compare the deal's owner_account_id against your own before you treat [] as authoritative.
• null and [] mean different things on phones. An empty array means the contact has no numbers on file. null means the numbers are withheld. The published contract allows both, so map the field to a nullable column. Collapsing them makes a withheld number indistinguishable from a missing one.

Within a single account, contact visibility is not narrowed any further. Every API key at an account reads the same sponsor contacts regardless of which broker owns the deal, so scope the destination's own permissions deliberately rather than assuming the API narrowed the set for you.

Syncing Fees

?include=fees embeds the brokerage's own fee on each deal — see Fees Object for the payload. It behaves unlike every other include on the deals endpoints, in ways that decide whether a nightly sync works at all.

The key has to belong to an admin. On GET /deals the include is admin-only, and a non-admin key gets a 403 for the whole request rather than a page of nulls. Since minting an API key already requires an admin, there is no self-serve path here: an integrator cannot create a fee-capable key for themselves, and cannot upgrade one they already hold. Have an admin at the account mint the key the sync will run under, and re-check it whenever that person's role changes. A demotion does not invalidate the key — it keeps reading deals — but it ends fee access loudly rather than quietly: ?include=fees starts returning 403 on the collection, and on the detail route it returns 403 for every deal that person does not personally own. Alert on a 403 from the fee sync. It is the only signal that the key lost fee access, and it will never arrive as thinner data.

Admin standing is per account. An admin key reads fees on its own account's deals and reports fees: null on deals another account owns, which land on the same page wherever accounts are commingled. fees: null means "not readable with this key", never "no fees on this deal". Store it as unknown and leave the destination row alone. Writing a zero there reports a deal as fee-free when it is not.

gross_fee_amount: null is not zero either. It is null whenever the total cannot be stated exactly: any row's amount is null, there are no rows at all, or the deal carries more than 50 fee rows. Map it to a nullable column and let it stay null. A genuine zero fee arrives as rows that each resolve to 0.0, so the two stay distinguishable as long as you keep the rows.

Rates are percents, and amount is already resolved. rate_percent: 1.5 means 1.5%, not 150% and not 0.015. Read amount wherever you can: Lev has already applied the rate, the basis, and any flat override, and re-deriving the number downstream is how a destination and Lev end up disagreeing about the same deal.

One admin key walking the collection is the whole story: fees need no per-deal fan-out. The detail route is the fallback for a non-admin who happens to own deals, since GET /deals/{deal_id}?include=fees also admits a deal's primary owner — one request per deal, and worth the cost only for a small owned set.

Operational Checklist

• Monitor request_id values for failed batches.
• Alert on repeated 401, 403, and 429 responses.
• Keep writes idempotent in your destination so replaying a batch is safe.
• Version your destination schema deliberately as Lev fields expand.