khotan-dataalpha
View as Markdown
Get Started

Configuration

Most khotan configuration lives in environment variables and your khotan.ts factory registration file.

Today, khotan-data is opinionated toward Vercel-style Next.js deployments. That means the default durable workflow story assumes Vercel Workflow and the default scheduling story assumes a single dispatcher cron hitting your app over HTTP.

Environment variables

  • DATABASE_URL — used by Drizzle for migrations and runtime DB access
  • KHOTAN_SECRET — random 32+ byte secret used to encrypt stored vars
  • KHOTAN_WEBHOOK_URL — public origin used when building default wire callback URLs
  • KHOTAN_DEBUG — enables plug debugging routes and CLI access to them
  • CRON_SECRET — optional bearer secret for protecting the built-in /api/khotan/cron dispatcher route
env
# .env.local (dev)
DATABASE_URL=postgres://...
KHOTAN_SECRET=...
KHOTAN_WEBHOOK_URL=https://abc123.ngrok.app
KHOTAN_DEBUG=1
CRON_SECRET=...

In development, KHOTAN_WEBHOOK_URL should usually point at a tunnel rather than localhost.

`khotan.config.ts`

This is the CLI config file. The most important value is outputDir, which tells khotan where to place generated runtime source files.

`khotan.ts`

This is the factory registration file. It wires khotan to your database and declares what should exist in the runtime.

ts
import { khotan, drizzleAdapter } from "khotan-data/factory";
import { db } from "@/db";
import { pollinatePlug } from "./plugs/pollinate";
import { slackPlug } from "./plugs/slack";
import { pollinateProductsSnapshotCache } from "./caches/pollinate-products-snapshot";
import { pollinateProductsInflow } from "./flows/pollinate-products";
import { pollinateWire } from "./wires/pollinate-wire";
import { pollinateCatch } from "./webhooks/pollinate-catch";
import { pollinateToSlack } from "./webhooks/pollinate-to-slack";

const khotanData = khotan({
  adapter: drizzleAdapter(db),
  resources: [
    {
      name: "products",
      mapping: {
        connectField: "sku",
        plugs: {
          pollinate: { uniqueIdentifier: "id" },
        },
      },
      description: "Shared product identity",
    },
    {
      name: "customers",
      mapping: {
        connectField: ["tenantId", "email"],
        plugs: {
          pollinate: { uniqueIdentifier: "id" },
          cin7: { uniqueIdentifier: "id" },
        },
      },
      description: "Shared customer identity across systems",
    },
  ],
  caches: [
    pollinateProductsSnapshotCache,
  ],
  plugs: [
    {
      name: "pollinate",
      plug: pollinatePlug,
      flows: [pollinateProductsInflow],
      wires: [pollinateWire],
      catches: [pollinateCatch],
      passes: [pollinateToSlack],
    },
    {
      name: "slack",
      plug: slackPlug,
    },
  ],
});

export default khotanData;

What each key does

  • adapter — connects khotan to your Drizzle database
  • resources — shared logical entities for cross-system mapping
  • caches — durable named key/value stores for workflow state
  • plugs — external services your app talks to
  • flows — durable data movement for one plug
  • wires — webhook registration lifecycle for one plug
  • catches — verified inbound webhook processing
  • passes — verified inbound webhook forwarding to another plug

Resource mapping contracts

Resources define the shared identity contract for mappings through their mapping block.

  • mapping.connectField can be a single field name such as "email"
  • or an ordered field list such as ["tenantId", "email"]
  • mapping.plugs declares which plug names are allowed in mapping refs
  • each declared plug provides exactly one uniqueIdentifier definition
ts
resources: [
  {
    name: "customers",
    mapping: {
      connectField: ["tenantId", "email"],
      plugs: {
        pollinate: { uniqueIdentifier: "id" },
        cin7: { uniqueIdentifier: "id" },
      },
    },
  },
]

This matters because khotan stores one canonical connectValue per mapping row, keeps external per-plug IDs in refs, and reserves metadata for contextual non-identity fields.

Plug configuration

A plug usually lives in src/khotan/plugs/<name>.ts or khotan/plugs/<name>.ts.

  • baseUrl
  • auth strategy
  • retry and timeout behavior
  • typed endpoint metadata
  • encrypted vars
  • request hooks for token exchange or custom signing

Flow configuration

Flow builders like inflow, outflow, and relay create durable work definitions.

  • name
  • resource
  • schedule
  • workflow

Flow, relay, catch, and pass workflows also receive khotanCache(ctx, "name"), which resolves a registered durable cache for snapshots, cursors, and dedupe markers.

Cache configuration

A cache usually declares name, scope, and ttl.

ts
import { cache } from "./caches/cache";

export const pollinateProductsSnapshotCache = cache({
  name: "pollinate-products-snapshot",
  scope: {
    plug: "pollinate",
    resource: "products",
    flow: "pollinate-products",
  },
  ttl: "6h",
});

Registered caches are upserted into khotan_caches. Individual key/value rows live in khotan_cache_entries, where khotan stores the latest value per (cache, key) pair and respects optional TTL expiry.

Wire configuration

A wire usually declares events, onSubscribe, onUnsubscribe, and optional onVerify.

Catch and pass configuration

Catch and pass builders define durable handlers for verified inbound events:

  • catchEvent({ name, events, workflow })
  • pass({ name, to, events, workflow })

Pass handlers receive destVars from the destination plug automatically.

Runtime tables

  • khotan_plugs
  • khotan_resources
  • khotan_flows
  • khotan_wires
  • khotan_webhook_handlers
  • khotan_webhook_events
  • khotan_runs
  • khotan_mappings
  • khotan_caches
  • khotan_cache_entries

That schema is why the CLI, UI, and runtime all see the same system state.

Scheduling and run types

Flows can be listed and triggered through the flows CLI. Mappings can be browsed and mutated through the mappings CLI and browser UI.

  • full
  • delta
  • backfill
  • reconcile
  • dry-run

For scheduled production runs, prefer one dispatcher cron in vercel.json instead of one platform cron per flow:

json
{
  "crons": [
    { "path": "/api/khotan/cron", "schedule": "* * * * *" }
  ]
}

Then define schedules only on flows in khotan.ts:

ts
{
  name: "products-inflow",
  type: "inflow",
  schedule: "0 * * * *",
  resource: "products",
}

The dispatcher route checks which flows are due for the current minute and starts them through the normal run-tracking path. If CRON_SECRET is set, Vercel should call /api/khotan/cron with Authorization: Bearer <CRON_SECRET>.

Agent-facing configuration

npx khotan init --yes also installs built-in skills and writes AGENTS.md.

  • how khotan is installed in the project
  • which skills to read
  • where generated source files live
  • which commands to use for debugging and runtime operations