Data Processing

Definition

Data processing is everything that happens to data between the moment a system records it and the moment a model, a query or a dashboard can use it: pulling it out of the systems that produced it, checking it against an expected shape, quarantining or repairing what fails, joining it to other data, and writing the result somewhere durable enough to build on. It is where the time goes — Anaconda's 2020 State of Data Science survey put data loading and cleaning at 45% of a data scientist's working week, the largest single block in the study, larger than model selection, training or deployment.

The reason that share is so stubborn is not that cleaning is hard. It is that cleaning is never finished. Model training is a command you run again unchanged; a pipeline sits downstream of systems owned by other teams, and those systems change shape without telling you. A column is renamed, a field that was always an integer starts arriving as the string "N/A", a partner begins sending timestamps in local time instead of UTC. None of these raise an error. The job succeeds, the row count looks normal, and something quietly wrong flows into the next model.

So data processing is engineering, and it is judged by an engineering test: does the same input produce the same output every time the job runs? It has no opinion about what the numbers mean — only that they are the numbers the source actually reported. Asking what they mean is Data Analysis.

How It Works

A production pipeline is a sequence of stages, and the useful way to think about each one is: what does it guarantee, and what happens when it is run twice?

Ingestion moves data from a source you do not control into storage you do. The two shapes are pull and push. A pull extracts on a schedule, usually with a high-water query such as WHERE updated_at > :last_run, which is simple and has a well-known hole: it never sees rows that were hard-deleted, because a deleted row has no updated_at to exceed the mark. Change data capture closes that hole by reading the database's own replication log, so deletes arrive as events. A push inverts the relationship — producers write to a durable log such as Kafka and consumers read at their own pace, which decouples a slow downstream job from the system generating the traffic.

Landing writes the extract to immutable storage before anything interprets it, partitioned by ingestion date. This is the cheapest insurance in the whole system. If the transformation logic has a bug, you fix the logic and re-run it against data you still hold. If you transformed on the way in, the bug destroyed the evidence and the only remedy is re-extracting from a source that may have aged the history out.

Validation is where a pipeline stops being hopeful. The checks that earn their keep are boring: a column is not null, a key is unique, a categorical field takes one of a known set of values, a numeric field falls in a plausible range, a foreign key resolves. The design decision is what a failure does. Failing the whole batch keeps the warehouse consistent and wakes someone up; quarantining the bad rows into a side table keeps the pipeline running and defers the problem. Both are defensible; silently coercing the bad rows is not, and it is the default behaviour of almost every permissive parser.

Put a number on why this matters. At 50 million events a day, a 0.1% malformed rate is 50,000 broken records daily and roughly 1.5 million across a 30-day training window. If those errors are random, a model shrugs them off. If they all come from one mobile SDK version, one region or one partner feed, they are not noise — they are a correlated signal, and the model will happily learn to predict from the shape of your own pipeline.

Transformation does the deduplication, joins, aggregation, type coercion and encoding — including turning categories and text into the numeric form models require, which is vectorization. The order of this stage relative to loading is the ETL / ELT distinction, and the order flipped for an economic reason rather than a technical one. In a classic warehouse, storage and compute were the same appliance bought by capacity, so you transformed first and loaded only what you had decided to keep. Cloud warehouses — BigQuery in 2011, Snowflake generally available in 2015 — separated the two, and object storage now costs on the order of two cents per gigabyte-month (S3 Standard, at the time of publication). Keeping a terabyte of raw JSON for a year costs a couple of hundred dollars; discovering in month nine that you needed a field you discarded in month one costs a quarter.

Publishing and reprocessing is the stage most first drafts forget, and it dictates how the earlier ones must be written. Suppose your daily job handles 50 million rows in 20 minutes. Reprocessing three years of history is about 55 billion rows — roughly 1,100× the daily volume, which at the same throughput is 365 hours, or over 15 days of serial compute. You can fan it out across 100 workers and finish in under four hours, but only if each partition can be rebuilt independently and rebuilding it twice is harmless. That property is idempotency, and it is not a nicety. Delivery in real systems is at-least-once: a consumer that writes its output and crashes before committing its offset will replay the batch. At 500 batches a day and one retry per 10,000, you duplicate a batch roughly every 20 days. A duplicated 100,000-row batch is 0.2% of the daily total and invisible in any global count — but for every user in that batch, every per-user sum is exactly double.

Types

Batch, micro-batch and streaming are a genuine typology: they are three distinct execution models with different fault-tolerance stories, and practitioners use the words to mean specific things.

Batch processes a bounded input — yesterday, the last hour, one partition — on a schedule. It is the cheapest per record because the whole set is visible at once, so the engine can sort, shuffle and join without keeping long-lived state, and a failed run is simply run again. The cost is freshness, and the arithmetic is unforgiving: a job that starts at 02:00 and covers up to midnight means the number on a dashboard at 09:00 describes a world that is nine hours old.

Micro-batch runs the same programming model over windows of seconds to minutes. Spark Structured Streaming works this way by default, and the appeal is that it inherits batch's fault tolerance — each micro-batch either commits or is retried — while cutting latency by two or three orders of magnitude. It is the pragmatic middle, and most "real-time" pipelines in production are this.

Streaming processes each record as it arrives with continuously maintained state, as in Flink or Beam. Two costs are systematically underestimated. First, capacity: a batch job is sized for the day's average, while a streaming job must absorb the peak minute, so a workload whose peak is five times its mean pays for roughly five times the average capacity around the clock. Second, duplication of logic — the same business rule now exists in a batch implementation and a streaming one, and they drift. That is precisely the objection Jay Kreps raised to the Lambda architecture in 2014 when proposing what became known as Kappa: run one code path over a replayable log and reprocess by replaying it, rather than maintaining two.

Streaming also forces the hardest question in the whole discipline: what is "now"? Events carry an event time (when the thing happened) and a processing time (when your system saw it), and these diverge — a phone in airplane mode buffers events and delivers them three hours late. A windowed aggregation cannot wait forever, so it closes on a watermark: an assertion that nothing older than time T will still arrive. The trade is exact and has no free setting. A 10-minute watermark on hourly windows publishes quickly and drops that phone's events, or forces you to restate a figure you already published. A 6-hour watermark captures them and makes every published number six hours stale. This is the model formalised in the Dataflow paper (Akidau et al., VLDB 2015), which is why event time, watermarks and triggers are named the same way across Beam, Flink and Spark.

Real-World Applications

The modern warehouse stack is two named tools doing the two halves. Apache Airflow, built at Airbnb in 2014, schedules pipelines as directed graphs — and the detail worth noticing is that backfill is a first-class command, which tells you the designers assumed re-running history is routine rather than exceptional. dbt turned the T of ELT into version-controlled SQL with tests attached: not_null, unique, accepted_values and relationships assertions live in the repo next to the model that depends on them, so a data quality rule is reviewed, diffed and blamed like any other code.

On the ingestion side, Apache Kafka — built at LinkedIn and open-sourced in 2011 — made the durable, replayable log the standard substrate: a consumer that got it wrong rewinds its offset and reads the same bytes again instead of asking the source system to re-export. Google Cloud Dataflow and Apache Flink implement the event-time semantics described above, which is what makes a windowed aggregate over out-of-order data a configuration choice rather than a research project.

Storage picked up the same guarantees. Apache Iceberg, opened by Netflix in 2018, alongside Apache Hudi from Uber and Delta Lake from Databricks, put atomic commits and schema evolution on top of object storage: overwriting a partition becomes all-or-nothing, and a renamed column tracks a stable column ID rather than its name — the difference between a rename and an accidentally all-null column. Uber's Michelangelo platform (2017) popularised the feature store for the adjacent problem: when a feature is computed by a batch job for training and by a separate service at inference time, the two implementations disagree, and the model in production sees inputs it never trained on.

Key Concepts

Exactly-once is a property of your sink, not of your transport. No messaging system can promise a message is delivered exactly once — the acknowledgement itself can be the thing that is lost. What you actually get is at-least-once delivery plus a way to make repetition harmless: a merge on a natural key, an overwrite of a whole partition, or a commit that ties the write and the offset together in one transaction. You design your way to exactly-once semantics; you do not buy them.

Schema-on-write versus schema-on-read decides when you find out. A warehouse that enforces types at load time rejects the bad record at 02:00 with a stack trace naming the column. A lake that accepts anything and interprets at query time takes the same record silently, and you learn about it weeks later when an analyst notices a column is 40% null. Schema-on-read is genuinely more flexible for semi-structured data; the flexibility is paid for in detection latency, which is why table formats and data contracts have been pulling the lake back toward enforcement.

Challenges

A silent schema change is the characteristic failure of this discipline because nothing fails. An upstream team changes a field from a number to a string, or renames user_id to userId. A permissive JSON parse yields null; the loading job reports success and a normal row count; the feature pipeline imputes the median for the nulls, as it was told to; the model retrains on a column that is now a constant. Every dashboard is green. The only visible symptom is a slow drift in a quality metric days later, which is exactly why monitoring has to watch input distributions and null rates rather than only job exit codes.

Time zones destroy aggregates in ways that look like seasonality. Bucket a daily total by local wall-clock time and the spring daylight-saving day has 23 hours while the autumn one has 25 — so the feature dips about 4.2% (one hour in 24) on one day a year and rises about 4.2% on another. Worse, 01:30 local occurs twice in autumn, so any key of the form (user_id, local_hour) collides and one of the two rows wins silently. Store UTC plus the originating zone and convert at the edge where a human reads it; every conversion earlier than that is a chance to lose information you cannot reconstruct.

A null encoded as a valid value is worse than a missing one. A source that writes 0 for "age not provided" does not just shift the mean — with 8% of rows affected, a column whose true mean is 42 reports 38.6 — it creates a fake signal. A gradient-boosted tree will find a genuinely predictive split at age < 1, because "this field is absent" correlates with the partner feed that produced the record, and that feed correlates with the label. The model has learned to identify its own data sources. It scores well in cross-validation and collapses in production the first time the feed mix changes, which is leakage arriving through the ingestion layer rather than the modelling code.

Reprocessing is a correctness problem, not just a cost problem. Re-running last quarter with today's transformation logic produces a table that no longer matches the numbers already reported from it. Any pipeline whose output someone has quoted needs an answer to "which version of the logic produced this row", which is what lineage metadata and versioned table snapshots exist to record.

Future Trends

Iceberg is settling into the role of interchange layer between engines — Databricks acquired Tabular in 2024, and the major warehouse vendors now read and write the format — which decouples the choice of query engine from the physical layout of the data.

Data contracts move the schema check to the producer: rather than a consumer discovering a breaking change after it lands, the producing service declares a versioned schema and its own CI fails on an incompatible edit. It is an organisational fix wearing a technical hat, and the only approach that attacks the silent-schema-change failure at its cause rather than its symptom.

The heaviest new load is unstructured. Pipelines built for rows are being asked to ingest PDFs, HTML and scanned documents for retrieval-augmented generation, where the failure modes are unfamiliar: a table split across a page boundary, a repeated header absorbed into a chunk, a footnote glued to the wrong sentence. Language models are now often the parser as well as the consumer, which makes extraction probabilistic — and a probabilistic extractor needs the same validation discipline as any other unreliable source.

Code Example

The difference between a pipeline you can re-run and one you cannot fits in two SQL statements.

-- Not idempotent: a replayed batch appends the same rows a second time.
-- The result depends on how many times the job ran, which is not a property
-- of the data.
INSERT INTO orders_clean
SELECT order_id, user_id, amount_cents, event_time
FROM staging_orders
WHERE ingest_date = DATE '2026-07-21';

-- Idempotent: the natural key decides. Running this once, twice or ten times
-- leaves the table in exactly the same state.
MERGE INTO orders_clean AS t
USING (
    SELECT order_id, user_id, amount_cents, event_time
    FROM staging_orders
    WHERE ingest_date = DATE '2026-07-21'
    -- The source of a MERGE must be unique on the join key, or the statement
    -- errors (or, worse, picks a row arbitrarily). At-least-once delivery
    -- means duplicates are expected, so collapse them here, deterministically.
    QUALIFY ROW_NUMBER() OVER (
        PARTITION BY order_id ORDER BY event_time DESC, ingest_sequence DESC
    ) = 1
) AS s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET
    amount_cents = s.amount_cents,
    event_time   = s.event_time
WHEN NOT MATCHED THEN INSERT (order_id, user_id, amount_cents, event_time)
    VALUES (s.order_id, s.user_id, s.amount_cents, s.event_time);

QUALIFY is warehouse syntax (Snowflake, BigQuery, Databricks); elsewhere the same deduplication is a subquery filtering on ROW_NUMBER() = 1. The tie-breaker matters as much as the ranking: ordering by event_time alone leaves two records with identical timestamps in a coin-flip, so the job becomes non-deterministic in exactly the case — duplicates — it exists to handle.