System Architecture & Privacy Engineering

Keystone CDP is built on an enterprise-grade microservice architecture designed for strict tenant isolation, sub-second query execution, and cryptographic privacy guarantees.

High-Level Topology

graph TD
    subgraph Client Touchpoints
        Web[Web Applications / E-Commerce] -->|tracker.js SDK| Gateway[Ingestion Gateway :8002]
        Mobile[Mobile Applications] -->|REST API /api/ingest| Gateway
        Offline[CSV / Batch POS Imports] -->|Batch Ingest Engine| Gateway
    end

    subgraph Core Platform Services
        Gateway --> PG[(PostgreSQL + pgvector)]
        Gateway --> Redis[(Redis In-Memory Broker)]
        Redis --> Arq[ARQ Async Worker Pool]
        Airflow[Apache Airflow Scheduler] --> Arq
        Arq --> ML[Predictive AI Models\nRidge CLV / Gemma / RFM]
        ML --> PG
    end

    subgraph Presentation & Control
        PG --> UI[Keystone React Console :80]
        Redis -->|WebSockets| Pulse[Live System Pulse Telemetry]
        Pulse --> UI
        UI --> Privacy[Dynamic PII Obfuscation Layer]
        PG --> DocSite[Documentation Site :443]
    end

1. Multi-Tenant Schema Isolation

Keystone uses a PostgreSQL multi-tenant architecture with schema-level isolation:

Each client workspace (e.g., eq, demo, tenant_alpha) has a completely distinct PostgreSQL schema.
Database tables, foreign key constraints, indexes, and sequences are physically segregated.
No cross-tenant queries are possible at the SQL layer, preventing multi-tenant data leakage.
Database connection pools dynamically bind search paths per request based on the verified tenant token.

-- Schema binding per tenant request
SET search_path TO eq, public;
SELECT * FROM profiles WHERE id = '...';

2. Ingestion Pipeline & Queueing

The Keystone Ingestion Gateway is engineered to absorb tens of thousands of incoming events per second without dropping payloads or degrading query performance:

sequenceDiagram
    participant Client as Client Browser / App
    participant Gateway as Ingestion Gateway
    participant Redis as Redis Broker
    participant Arq as ARQ Worker Pool
    participant DB as PostgreSQL (Schema)

    Client->>Gateway: POST /api/ingest (Event Payload)
    Gateway->>Gateway: Validate API Key & Schema
    Gateway->>DB: Upsert Identity Node (Fast Anchor/Shadow Check)
    Gateway-->>Client: 202 Accepted { "status": "enqueued", "event_id": "..." }
    Gateway->>Redis: Push to Event Queue
    Redis->>Arq: Pull Batch
    Arq->>DB: Deep Attribute Merge & Trajectory Update
    Arq->>Redis: Broadcast Event via WebSocket to Pulse

Micro-Batch Processing

Immediate Ingestion Acknowledgement: Synchronous requests to /api/ingest or /ingest are validated and queued within sub-20ms response times.
Background Reconciliation: Worker pools consume batches from Redis, computing attribute deltas, checking order inception rules, and updating the identity graph.
Fault Tolerance: If downstream database replication lags or network hiccups occur, events remain persisted in Redis queues with exponential backoff retry policies.

3. The Anchor & Shadow Identity Engine

Identity resolution in Keystone operates across two primary states:

[ Anonymous Visitor ] ──> Generates Cookie ID (UUID) ──> Creates Shadow Profile
                                                               │
[ Later Signup/Login ] ──> Provides Email / Phone ────────────> Resolves to Anchor Profile
                                                               │
                                                   [ THE COLLAPSE PROTOCOL ]
                                                   Shadow Node collapses into Anchor
                                                   Historical behavioral events merged
                                                   Vector embeddings re-calculated

Shadow Profiles: Automatically instantiated when an unknown anonymous_id appears in event payloads. Shadow profiles accrue clickstreams, search queries, and cart updates.
Anchor Profiles: Created or located when an authoritative identifier (email, phone, user_id) is submitted.
Profile Collapse: When an event contains both a shadow anonymous_id and an anchor email, Keystone triggers a deterministic collapse. The shadow node is retired and linked, while all historical event streams and attributes are merged onto the canonical anchor profile.

4. Vector-Based Trajectory Attribution

Keystone incorporates pgvector inside PostgreSQL to store high-dimensional embeddings representing customer journeys:

Customer touchpoint sequences are vectorized into a 1536-dimensional latent space.
Every marketing channel, campaign touchpoint, and organic referral is projected along a behavioral trajectory.
When an order conversion happens, Keystone computes the cosine distance between each prior touchpoint vector and the terminal conversion vector.
This produces continuous, mathematically rigorous multi-touch attribution far superior to crude first-click or last-click rules.

5. Security & Privacy Architecture

Role-Based Access Control (RBAC)

Keystone enforces five distinct role groups with 35+ granular permission toggles: - admin: Full workspace control, API key provisioning, billing, and schema configuration. - admin_incognito: Global administrative capabilities with continuous real-time PII obfuscation for live presentations, client demos, and compliance training. - editor: Audience creation, journey workflow editing, and activation synchronization. - viewer: Read-only dashboard inspection and report generation. - support: Restricted customer lookup with audit-logged access.

Cryptographic Obfuscation Protocol

In admin_incognito mode: - Customer names are mapped deterministically through a SHA-256 salted pseudonymization dictionary. - Email addresses are transformed into format-preserving dummy strings: anon_<hash>@dummy-obfuscated.com. - Street addresses and telephone numbers are substituted with synthetic demographic data.

6. Real-Time Telemetry & System Pulse

The console integrates a WebSocket connection to the Redis Pub/Sub layer: - Ingested events, identity merges, and background worker completions are published instantly. - The UI's System Pulse terminal receives this telemetry without polling the primary relational database. - Anomaly detection monitors track spike and drop thresholds, notifying operations teams of data ingestion interruptions immediately.