Roadmap
The path toward a complete, agent-first data pipeline foundation. Everything here is open to feedback — open an issue or start a discussion on GitHub.
Multi-tenancy & Auth
The foundation for running khotan-data in a product with real users and organisations.
Scoped plugs, flows, and credentials per-tenant. Row-level isolation in the standard khotan tables out of the box.
Each tenant gets isolated plug configs, run history, and secrets. Agent tooling respects tenant boundaries automatically.
First-class integration with better-auth. Tenant resolution from the session, per-org credential vaults, and auth-aware Hub UI.
Drop-in if you are already using better-auth. Connects multi-tenancy to the session layer with zero glue code.
Skills ecosystem
Making khotan skills a community-owned, independently installable layer.
Split skills out of the core repo into their own versioned registry. Install any skill with a single command.
npx khotan add skill bootstrap — installs the right SKILL.md straight into your .cursor, .claude, or .agents folder. Skills are versioned and community-contributed.
Data quality
Tools for making sure the data that moves through khotan pipelines is complete, correct, and trustworthy.
Define required fields on a plug endpoint. khotan flags any records where those fields are null or fail validation before they land in your DB.
Per-endpoint required field schemas. Flagged records are written to a quarantine table with the reason. Runs can be configured to fail fast or continue with flagging.
When required fields are missing, automatically go and find them. Integrations with Firecrawl and Exa let khotan web-search for missing supplier addresses, ABNs, and other public data.
Define an enrichment strategy per field — search query template, source (Firecrawl / Exa / custom plug), and confidence threshold. Enriched values are written back with a provenance tag.
Drop-in shadcn components for cleaning and enrichment. An operator can review flagged records, accept enriched suggestions, or manually enter values — all from the Hub.
DataCleaningTable and DataEnrichmentReview components. Fully typed, wired to the standard khotan quarantine tables. Approve, reject, or edit per-field.
Plug ecosystem
A community registry of typed, production-grade plugs that anyone can install and own.
Community-contributed plugs installable with npx khotan add plug stripe. Like shadcn's registry, but for API integrations — you install the source, you own it.
Each registry plug ships with auth strategies, typed endpoints, retry config, and a matching skill. Plugs are reviewed for correctness and maintained by the community.
Detect when an external API's real response drifts from your typed endpoints. Surface a diff and suggest a migration — no manual reconciliation.
Compare live API responses against your Zod schemas on each debug probe. If shapes drift, khotan generates a suggested endpoint patch and optional Drizzle migration for your review.
Pipeline primitives
Expanding the core building blocks so agents can construct pipelines of any shape.
Agent-writable Zod transform functions between a plug's endpoint shape and your destination table. Composable, testable, and visible in the run trace.
Define a transform per inflow step. khotan validates input and output types, shows the transform in the Hub, and catches type errors before they reach your DB.
Trigger flows from incoming webhooks, pg_notify, or custom app events — not just cron. The missing link for fully reactive pipelines.
Catch component already handles inbound webhooks. Event triggers extend this so a Catch can fan out into a Flow directly, or pg_notify from your own tables can kick off an outflow.
Re-run any flow over a historical time window without code changes. Essential for recovering from failures or bootstrapping a new table from existing API data.
Pass --from and --to to the CLI or Hub. khotan replays the flow with the correct pagination and idempotency keys, skipping records already written.
Trigger a flow in different modes from the CLI or the Hub UI, where each variant controls what gets fetched, written, or skipped. Shipped in v0.7.0: pass --variant <name> (or variant in the body) and branch on ctx.variant.
Shipped (v0.7.0): the trigger mechanism — npx khotan flows trigger <name> --variant delta, with ctx.variant on the flow run context. Still planned: framework-defined modes with standard semantics (full, delta, partial, test, bust-cache), an --ids subset selector, and a variant picker on the Hub Run Now button.
Record real API responses from a plug probe and replay them in CI. Snapshot your transforms. No live credentials needed in tests.
khotan record saves a fixture set per endpoint. khotan test replays it through your transforms and assertions. Fixtures are committed to the repo and diffed on change.