Webhook Ingestion & Web Tracking

Keystone CDP utilizes a highly resilient ingestion pipeline capable of absorbing continuous webhook traffic from external services and client-side web applications.

Primary Endpoints

1. Standard Synchronous Ingest (POST /api/ingest)

Used for single real-time tracking events (Identify, Track, and Page views). Note: The core endpoint is technically /ingest, but /api/ingest is provided as an alias to simplify proxy routing and bypass restrictive CORS boundaries in modern frontend networks.

Headers: - X-API-Key: Your Scoped Tracker API Key or Integration Key.

Basic Payload:

{
  "anonymous_id": "f5f1345d-...",
  "identity": "[email protected]", 
  "event_type": "track",
  "event_name": "Added to Cart",
  "payload": {
    "product_id": "P-1234",
    "price": 100.00
  }
}

2. High-Performance Batch Ingestion (POST /tenant/ingest/batch)

Used for historical backfills or server-side bulk updates. This endpoint is entirely asynchronous. It immediately returns an "enqueued" status and passes the payload to a Redis-backed Arq worker pool (process_ingest_batch_job).

Payload Structure: Needs to be wrapped as a list in an IngestRequest.

Identity Resolution & Merging

When your webhook hits the ingest service, Keystone routes the record through a deterministic identity graph engine.

  1. Shadow Profile Link: If your payload contains only an anonymous_id (cookie ID), the system locates or creates a "Shadow Profile" designed to accrue historical behavioral data prior to authentication.
  2. Anchor Profile Link: If an explicit identity (email, phone, CRM ID) is provided, it updates the definitive "Anchor Profile".
  3. The Merge Protocol: If a webhook provides both an anonymous_id and an identity simultaneously, Keystone initiates a deep Profile Collapse. The historical shadow profile's entire behavioral history and attribute state is mathematically merged directly onto the verified anchor profile.

Granular Attribute Merging Strategy

When webhook payloads contain new user identity characteristics, the metadata is recursively merged into the attributes blob using the following rules:

  • New Keys: Added incrementally.
  • List Replacement: If both the existing database value and your new webhook value are arrays ([]), the incoming list entirely overwrites the target array to permit absolute state overrides.
  • List Append: If an incoming array hits a scalar string, they union. If an incoming string hits an array, it appends.
  • Granular Lineage Logging: Keystone calculates absolute diffs (deltas) of the data change during the merge. Changing a city from Chicago to Miami logs a specific audit change of just that key, saving system resources and improving auditability.

Behavioral Order Inception (Lifecycle Tracking)

Webhooks can trigger transactional states before an actual payment is processed. If any track or page event contains an order_id in its payload, the system dynamically spins up or modifies an Order object.

  • Abandonment Status: Sending events like Cart Updated or Removed from Cart sets the synthetic order to incomplete.
  • Completion: A final Order Completed webhook resolves the state to completed.
  • Auto-Provisioning: If your webhook includes line items with foreign product_id strings the CDP hasn't discovered yet, it will provision skeleton products so your backend analytics never fragment.

The JavaScript Web Tracker

If you are ingesting straight from a client's browser, use the dynamic tracker.js.

Snippet Location: https://<cdp-domain>/v1/js/tracker.js

  • Dynamic Detection: The script inspects its own load path (document.currentScript.src) to dynamically determine where to bounce events back (automatically targeting the /api/ingest alias).
  • Buffer Queue: If the network drops, window.keystoneData maintains a volatile buffer of events that are processed upon reconnection.
  • Persistence: Relies on localStorage to persistently govern the user's canonical anonymous_id across distinct browser sessions.