Webhooks & Ingestion Gateway

Keystone CDP provides a high-throughput webhook ingestion gateway capable of absorbing thousands of real-time webhooks per second from external storefronts, mobile apps, CRM systems, and backend servers.

1. Ingestion Endpoints

1.1 Synchronous Ingest (`POST /api/ingest`)

Designed for real-time client-side tracking and instant event logging.

Headers:
Content-Type: application/json
X-API-Key: YOUR_SCOPED_API_KEY
Response: 202 Accepted with JSON { "status": "enqueued", "event_id": "evt_..." } within sub-20ms latency.

curl -X POST https://cdp.your-domain.com/api/ingest \
  -H "Content-Type: application/json" \
  -H "X-API-Key: kst_live_a8b9c0d1e2f3..." \
  -d '{
    "anonymous_id": "f5f1345d-79e2-4f3b-b2b9-112233445566",
    "identity": "[email protected]",
    "event_type": "track",
    "event_name": "Added to Cart",
    "payload": {
      "product_id": "SKU-TURB-01",
      "price": 349.99,
      "currency": "USD"
    }
  }'

1.2 High-Performance Batch Ingestion (`POST /tenant/ingest/batch`)

Used for historical CRM data imports, POS batch syncs, and night-run transaction reconciliations.

Payload Structure: A JSON array of up to 1,000 event objects per HTTP request.
Processing: Asynchronously enqueued directly into Redis for consumption by ARQ worker pools.

[
  {
    "identity": "[email protected]",
    "event_type": "track",
    "event_name": "Order Completed",
    "payload": { "order_id": "ORD-101", "total": 85.00 }
  },
  {
    "identity": "[email protected]",
    "event_type": "track",
    "event_name": "Order Completed",
    "payload": { "order_id": "ORD-102", "total": 142.50 }
  }
]

2. Ingestion Event Schema

Every event submitted to the ingestion gateway follows a standard contract:

Field	Type	Required	Description
`anonymous_id`	String	Conditional	Client cookie ID (UUID). Required if `identity` is absent.
`identity`	String	Conditional	Authoritative identifier (email, phone, CRM ID).
`event_type`	String	Required	One of: `track`, `page`, `identify`, `order`.
`event_name`	String	Required	Descriptive label (e.g. `Product Viewed`, `Checkout Step 1`).
`payload`	Object	Optional	Arbitrary key-value properties dictionary.
`timestamp`	ISO-8601	Optional	Timestamp when event occurred. Defaults to gateway receipt time.

3. Attribute Merging Strategy

When event payloads contain user demographic traits or profile updates, Keystone merges attributes using deterministic rules:

New Keys: Added incrementally to the profile's attributes JSONB document.
Array Fields: Incoming arrays overwrite existing arrays if full state replacement is intended; scalar-to-array collisions append automatically.
Granular Lineage Diffing: Keystone logs field-level diffs in the audit trail, ensuring full historical data lineage.

4. Outbound Webhooks (Event Dispatching)

In addition to receiving inbound webhooks, Keystone can dispatch outbound event webhooks to your external services (e.g. ERP, custom data lakes, or inventory systems).

Configuring Outbound Webhooks:

Navigate to Activations -> Webhooks (/activations/webhooks).
Provide your external destination URL: https://api.your-system.com/webhooks/keystone.
Select subscribed events: profile.created, profile.merged, order.completed, churn.risk_escalated.
Enter an HMAC Secret for payload signature verification.

HMAC Signature Verification

Every outbound webhook includes the X-Keystone-Signature header:

import hmac
import hashlib

def verify_keystone_webhook(payload_body: bytes, received_signature: str, secret: str) -> bool:
    expected_sig = hmac.new(
        secret.encode('utf-8'),
        payload_body,
        hashlib.sha256
    ).hexdigest()
    return hmac.compare_digest(expected_sig, received_signature)