stalance/services/data-pipelines
Data pipelines that tell you when they are lying.
The dangerous pipeline is not the one that crashes. It is the one that exits 0, loads half a feed and reports green. We build ingestion that checks its own output against an oracle it cannot quietly weaken, and pages a human when the numbers stop adding up.
- in production
- Daily, for our own 20+ storefronts
- sources
- Product feeds from 50+ merchant programmes, APIs, web pages
- scale
- 4,000,000+ product listings
- failure mode
- Quarantine and an alert, never a silent skip
What we build
Feed ingestion that streamsXML, CSV and JSON feeds read as streams, so a multi-gigabyte file costs memory for one row, not the whole file.
A census before the configWe measure a new source first: its real category tree, empty branches, dead fields, truncated downloads. Then we write the mapping.
Classification you can explainAI proposes, rules decide. Every category, brand and colour decision traces back to a rule a person can read and override.
Identity and de-duplicationThe same product from three sources becomes one record. Ambiguous matches go to quarantine instead of merging wrongly.
Polite web data collectionRate limits, back-off per proxy, bandwidth budgets per run and response caching. We collect what we need and stop.
Change historyPrices and availability kept as a log, so “what changed this week” is a query, not a guess.
How we would approach yours
01 censusMeasure the sourceReal record counts, field fill rates, category shape. A short written report.02 contractWrite down “correct”Expected counts, required fields, what gets rejected, what gets quarantined.03 pipelineBuild it to the contractStreaming ingest, transforms, classification, idempotent writes.04 watchAlert on the contractOn missing runs, short loads and dead rules, not just exceptions.Questions engineers ask us
Do you work with our existing stack?
Usually, yes. We pick tools per problem. If your warehouse, queue or language is a reasonable choice, we build on it rather than replace it.
Can you take over a pipeline someone else wrote?
Yes. We start with a census of what it actually loads today, which is often different from what it is believed to load.
Do you use AI for classification?
For proposals, yes. Decisions are applied through rules, so every outcome is explainable and reversible.
Tell us what is broken
One paragraph is enough. An engineer reads it, not a ticket bot.