# Roadmap The path toward a complete, agent-first data pipeline foundation. Everything here is open to feedback — open an issue or start a discussion on GitHub. --- ## Multi-tenancy & Auth The foundation for running khotan-data in a product with real users and organisations. ### Multi-tenancy · Planned Scoped plugs, flows, and credentials per-tenant. Row-level isolation in the standard khotan tables out of the box. Each tenant gets isolated plug configs, run history, and secrets. Agent tooling respects tenant boundaries automatically. ### better-auth integration · Planned First-class integration with better-auth. Tenant resolution from the session, per-org credential vaults, and auth-aware Hub UI. Drop-in if you are already using better-auth. Connects multi-tenancy to the session layer with zero glue code. --- ## Skills ecosystem Making khotan skills a community-owned, independently installable layer. ### Skills as a standalone package · Planned Split skills out of the core repo into their own versioned registry. Install any skill with a single command: ``` npx khotan add skill bootstrap ``` Installs the right SKILL.md straight into your .cursor, .claude, or .agents folder. Skills are versioned and community-contributed. --- ## Data quality Tools for making sure the data that moves through khotan pipelines is complete, correct, and trustworthy. ### Data cleaning · Planned Define required fields on a plug endpoint. khotan flags any records where those fields are null or fail validation before they land in your DB. Per-endpoint required field schemas. Flagged records are written to a quarantine table with the reason. Runs can be configured to fail fast or continue with flagging. ### Data enrichment · Planned When required fields are missing, automatically go and find them. Integrations with Firecrawl and Exa let khotan web-search for missing supplier addresses, ABNs, and other public data. Define an enrichment strategy per field — search query template, source (Firecrawl / Exa / custom plug), and confidence threshold. Enriched values are written back with a provenance tag. ### Operator data UI · Planned Drop-in shadcn components for cleaning and enrichment. An operator can review flagged records, accept enriched suggestions, or manually enter values — all from the Hub. `DataCleaningTable` and `DataEnrichmentReview` components. Fully typed, wired to the standard khotan quarantine tables. Approve, reject, or edit per-field. --- ## Plug ecosystem A community registry of typed, production-grade plugs that anyone can install and own. ### Plug registry · Exploring Community-contributed plugs installable with `npx khotan add plug stripe`. Like shadcn's registry, but for API integrations — you install the source, you own it. Each registry plug ships with auth strategies, typed endpoints, retry config, and a matching skill. Plugs are reviewed for correctness and maintained by the community. ### Schema inference & evolution · Exploring Detect when an external API's real response drifts from your typed endpoints. Surface a diff and suggest a migration — no manual reconciliation. Compare live API responses against your Zod schemas on each debug probe. If shapes drift, khotan generates a suggested endpoint patch and optional Drizzle migration for your review. --- ## Pipeline primitives Expanding the core building blocks so agents can construct pipelines of any shape. ### Typed transformation layer · Exploring Agent-writable Zod transform functions between a plug's endpoint shape and your destination table. Composable, testable, and visible in the run trace. Define a transform per inflow step. khotan validates input and output types, shows the transform in the Hub, and catches type errors before they reach your DB. ### Event-driven triggers · Planned Trigger flows from incoming webhooks, pg_notify, or custom app events — not just cron. The missing link for fully reactive pipelines. Catch component already handles inbound webhooks. Event triggers extend this so a Catch can fan out into a Flow directly, or pg_notify from your own tables can kick off an outflow. ### Replay & backfill · Planned Re-run any flow over a historical time window without code changes. Essential for recovering from failures or bootstrapping a new table from existing API data. Pass `--from` and `--to` to the CLI or Hub. khotan replays the flow with the correct pagination and idempotency keys, skipping records already written. ### Flow run variants · In progress Trigger a flow in different modes from the CLI or the Hub UI, where each variant controls what gets fetched, written, or skipped. **Shipped (v0.7.0):** the trigger mechanism. Pass `--variant ` (or `variant` in the start/request body) and your flow code branches on `ctx.variant`: ``` npx khotan flows trigger --plug --variant delta ``` **Still planned:** framework-defined built-in modes with standard semantics — - **full** — re-fetch and upsert everything - **delta** — only records changed since the last successful run (uses cursor state or `updated_at`) - **partial** — a named subset of pages or IDs you specify at trigger time (`--ids`) - **test** — dry-run with no writes; prints what would change - **bust-cache** — forces a fresh fetch, ignoring any ETags or cursor state — plus a variant picker dropdown on the **Run Now** button in the Hub UI. ### Testing mode · Exploring Record real API responses from a plug probe and replay them in CI. Snapshot your transforms. No live credentials needed in tests. `khotan record` saves a fixture set per endpoint. `khotan test` replays it through your transforms and assertions. Fixtures are committed to the repo and diffed on change. --- The end goal is a data system foundation where an agent can build any pipeline — inbound, outbound, webhook-driven, scheduled, enriched, cleaned, and observable — without needing to wire infrastructure from scratch each time. Have an idea? [Open an issue on GitHub](https://github.com/coreynolastname/khotan-data/issues).