Data platforms that stay trustworthy as the sources multiply
Willowark builds and repairs the machinery under analytics products and internal data platforms: ingestion from sources that change without warning, transformation that is testable, storage that does not surprise you on the invoice, and query paths fast enough that people open the dashboard instead of exporting to a spreadsheet. The problem we remove is the one where the platform works and nobody trusts the numbers.
We work at the pipeline and warehouse layer, not just the BI surface. Schema evolution is handled explicitly rather than by breaking: contracts on the way in, additive changes by default, a quarantine path for rows that no longer fit. Ingestion is idempotent and replayable, so a bad day is reprocessed rather than patched by hand. Lineage is recorded column to column, and batch or streaming is decided per source rather than as policy.
A first engagement usually starts where the trust broke: one pipeline that keeps failing silently, a warehouse invoice nobody can attribute, or a metric two dashboards disagree on. We instrument it, fix the cause rather than the symptom, and leave tests and contracts behind so the same failure cannot come back quietly. Multi-tenant products and customer-facing SLAs shape the approach; a change that is safe for an internal dashboard may not be safe when a customer's report depends on it, so we stage changes with backfills and comparisons before cutting over.
Reviewed
Sound familiar?
If you've said any of these, we should talk.
“Nobody trusts the dashboard, so everyone keeps their own spreadsheet.”
Trust gets rebuilt with evidence. We add tests at the transformation layer, record lineage from source column to reported metric, and put freshness and row-count expectations on every critical asset, so staleness shows up in the tool rather than in a meeting and a questioned number gets a traceable path.
“Our warehouse bill doubled and nobody can explain why.”
We attribute spend first — by query, model, team, and tenant — because the cause is usually a few patterns rather than general growth. Then we fix them: incremental models instead of full rebuilds, partitioning and clustering matched to the real predicates, pre-aggregation on the paths dashboards hit on load.
“An upstream change broke a pipeline and we heard about it from a customer.”
We put a contract at every boundary you do not control, so an unexpected schema change fails the load with a specific error and a quarantined sample instead of silently writing nulls. Loads are keyed and idempotent, so once the mapping is fixed you replay the window.
“Reports are fine for small tenants and unusable for our biggest one.”
That gap is architectural rather than a tuning problem. We profile the real query patterns and reshape storage around them: per-tenant partitioning, pre-aggregated rollups for the expensive cuts, and query paths that read summaries instead of scanning raw events.
“Leadership wants AI on top of our data and the data is not ready for it.”
Usually true, and saying so is part of the job. A model over inconsistent definitions produces confident wrong answers. We start by fixing the semantic layer — one definition per metric, tested — and then put retrieval or natural-language querying on top of that, with evals that check answers against known-good queries before anyone outside the team sees it.
How this industry actually runs
The operation as we usually find it.
The sources are outside your control — a vendor renames a field, an upstream API changes pagination, a customer's export gains a column — and each one arrives as a wrong number in a meeting rather than as an alert. Warehouse spend grows faster than adoption, because a dashboard that refreshes on page load and a model that rebuilds everything nightly are both easy to write and expensive to run. The team is a few data engineers, some analytics engineers living in dbt, and one platform person who also owns the infrastructure. Multi-tenant products add harder constraints: isolation that must be provably correct, row-level security that cannot be approximately right, and a largest customer whose workload must not degrade everyone else's.
Product, integrations, and the AI feature behind them
Components:
- Your product (web, mobile, API)
- Backend services (queues, jobs, evals)
- AI feature (retrieval, review gates)
- Data & integrations (warehouse, third parties)
Connections:
- Your product to Backend services (API)
- Backend services to AI feature (structured calls)
- Backend services to Data & integrations
What we build
Starting projects that fit Data & Analytics Platforms.
- Ingestion frameworks with idempotent, replayable loads and explicit schema-evolution handling
- Transformation layers with tests, data contracts, and column-level lineage from source to reported metric
- Data quality monitoring — freshness, volume, distribution, and referential checks with routed alerts
- Warehouse cost control: incremental models, partitioning and clustering, materialization, spend attribution
- Streaming paths where latency genuinely matters, including change data capture and exactly-once delivery
- Multi-tenant architecture with isolation, row-level security, per-tenant limits, and pre-aggregated query paths
- Semantic layer and metric definitions so every dashboard, export, and API answer agrees on the same number
- Natural-language querying and AI features over the warehouse, with evals against known-good queries before release
Capabilities we bring
Related work
Where this is running today.
AI operating software for training operations
A global automotive manufacturer's training operation · Automotive
AI operating software for training operations
AI operating software for the manufacturer's training schedulers — removing manual scheduling labor and logistics tracking, saving hundreds of hours per year — plus a training video game built to help trainers perform better. Used across the manufacturer's training organization.
Read the case study →Global loyalty & rewards platform modernization
A global loyalty & rewards platform · Financial services
Global loyalty & rewards platform modernization
Engineering with the platform's global loyalty and rewards teams: migrating legacy systems to event-driven microservices in Go, across point tracking and the ledger — distributed systems work through to war-room production deployments.
Read the case study →Working in Data & Analytics Platforms?
Tell us the line.
What runs by hand, what is not connected, what you are trying to build. An engineer replies within one business day with whether and how we would approach it.
Common questions
What Data & Analytics Platforms teams ask first.
Should we move to streaming, or is batch fine?
Per source, and the honest default is batch. Streaming earns its operating cost when a decision depends on data that is minutes old. For reporting a human reads once a day, a well-built incremental batch load is cheaper and far easier to backfill.
Have you worked on data systems at real scale?
Yes. We worked with American Express on global loyalty and rewards, moving legacy systems to event-driven microservices in Go across point tracking and the ledger. We also built AI operating software for Jaguar Land Rover's training schedulers, spanning full-stack development, cloud architecture, and distributed systems across the JLR ecosystem.
Do we have to adopt your preferred stack?
No. We work in what you already run — dbt, Airflow or Dagster, Snowflake, BigQuery, Databricks, Postgres, Kafka. We hold no vendor relationships and resell nothing, so a migration recommendation only comes when the current tool is genuinely the constraint.
How long before we see numbers we can trust again?
It depends on how many sources feed the metrics in question, but the first visible change is usually quick: the critical assets typically get freshness and volume checks early on, so staleness is at least visible in the tool. Rebuilding trust in a specific metric takes longer, because it means tracing lineage end to end and reconciling against a source of truth. We prioritize the handful of numbers leadership actually argues about and work outward from there rather than trying to fix everything at once.
Can you work alongside our data engineers rather than replacing them?
That is the usual arrangement. Your team knows the sources, the business definitions, and the history of why things are shaped the way they are, and none of that transfers quickly. We typically take one workstream — the ingestion framework, cost attribution, the quality layer, a multi-tenant redesign — and run it in your repos with your review, while your engineers keep the day-to-day platform running. Pairing on the handoff is part of the scope, so the pattern survives after we leave.
Strategy. Software. Systems.
Engineering for Data & Analytics Platforms.
Describe the problem in your own words. An engineer reads it — not a sales script — and tells you plainly what it would take.