---
source: 'https://howaiworks.ai/glossary/data-processing'
section: glossary
title: Data Processing
description: >-
  What happens to raw data before a model can use it: ingestion, validation,
  transformation, ETL vs ELT, batch vs streaming, and what silently breaks.
tags:
  - Data Science
  - data quality
  - MLOps
  - feature extraction
category: Data & Retrieval
datePublished: '2025-08-20'
lastUpdated: '2026-07-22'
---

# Data Processing

> What happens to raw data before a model can use it: ingestion, validation, transformation, ETL vs ELT, batch vs streaming, and what silently breaks.

## Definition

Data processing is everything that happens to data between the moment a system records it and the
moment a model, a query or a dashboard can use it: pulling it out of the systems that produced it,
checking it against an expected shape, quarantining or repairing what fails, joining it to other
data, and writing the result somewhere durable enough to build on. It is where the time goes —
Anaconda's 2020 *State of Data Science* survey put data loading and cleaning at **45% of a data
scientist's working week**, the largest single block in the study, larger than model selection,
training or deployment.

The reason that share is so stubborn is not that cleaning is hard. It is that cleaning is never
finished. Model [training](https://howaiworks.ai/glossary/training) is a command you run again unchanged; a pipeline sits
downstream of systems owned by other teams, and those systems change shape without telling you. A
column is renamed, a field that was always an integer starts arriving as the string `"N/A"`, a
partner begins sending timestamps in local time instead of UTC. None of these raise an error. The
job succeeds, the row count looks normal, and something quietly wrong flows into the next model.

So data processing is engineering, and it is judged by an engineering test: does the same input
produce the same output every time the job runs? It has no opinion about what the numbers mean —
only that they are the numbers the source actually reported. Asking what they mean is
[Data Analysis](https://howaiworks.ai/glossary/data-analysis).

## How It Works

A production pipeline is a sequence of stages, and the useful way to think about each one is: what
does it guarantee, and what happens when it is run twice?

**Ingestion** moves data from a source you do not control into storage you do. The two shapes are
pull and push. A pull extracts on a schedule, usually with a high-water query such as
`WHERE updated_at > :last_run`, which is simple and has a well-known hole: it never sees rows that
were hard-deleted, because a deleted row has no `updated_at` to exceed the mark. Change data capture
closes that hole by reading the database's own replication log, so deletes arrive as events. A push
inverts the relationship — producers write to a durable log such as Kafka and consumers read at
their own pace, which decouples a slow downstream job from the system generating the traffic.

**Landing** writes the extract to immutable storage before anything interprets it, partitioned by
ingestion date. This is the cheapest insurance in the whole system. If the transformation logic has
a bug, you fix the logic and re-run it against data you still hold. If you transformed on the way
in, the bug destroyed the evidence and the only remedy is re-extracting from a source that may have
aged the history out.

**Validation** is where a pipeline stops being hopeful. The checks that earn their keep are boring:
a column is not null, a key is unique, a categorical field takes one of a known set of values, a
numeric field falls in a plausible range, a foreign key resolves. The design decision is what a
failure does. Failing the whole batch keeps the warehouse consistent and wakes someone up;
quarantining the bad rows into a side table keeps the pipeline running and defers the problem. Both
are defensible; silently coercing the bad rows is not, and it is the default behaviour of almost
every permissive parser.

Put a number on why this matters. At 50 million events a day, a **0.1%** malformed rate is 50,000
broken records daily and roughly **1.5 million** across a 30-day training window. If those errors
are random, a model shrugs them off. If they all come from one mobile SDK version, one region or one
partner feed, they are not noise — they are a correlated signal, and the model will happily learn
to predict from the shape of your own pipeline.

**Transformation** does the deduplication, joins, aggregation, type coercion and encoding —
including turning categories and text into the numeric form models require, which is
[vectorization](https://howaiworks.ai/glossary/vectorization). The order of this stage relative to loading is the ETL /
ELT distinction, and the order flipped for an economic reason rather than a technical one. In a
classic warehouse, storage and compute were the same appliance bought by capacity, so you
transformed first and loaded only what you had decided to keep. Cloud warehouses — BigQuery in 2011,
Snowflake generally available in 2015 — separated the two, and object storage now costs on the order
of two cents per gigabyte-month (S3 Standard, at the time of publication). Keeping a terabyte of raw
JSON for a year costs a couple of hundred dollars; discovering in month nine that you needed a field
you discarded in month one costs a quarter.

**Publishing and reprocessing** is the stage most first drafts forget, and it dictates how the
earlier ones must be written. Suppose your daily job handles 50 million rows in 20 minutes.
Reprocessing three years of history is about 55 billion rows — roughly **1,100×** the daily volume,
which at the same throughput is **365 hours, or over 15 days** of serial compute. You can fan it out
across 100 workers and finish in under four hours, but only if each partition can be rebuilt
independently and rebuilding it twice is harmless. That property is idempotency, and it is not a
nicety. Delivery in real systems is at-least-once: a consumer that writes its output and crashes
before committing its offset will replay the batch. At 500 batches a day and one retry per 10,000,
you duplicate a batch roughly every 20 days. A duplicated 100,000-row batch is 0.2% of the daily
total and invisible in any global count — but for every user in that batch, every per-user sum is
exactly double.

## Types

Batch, micro-batch and streaming are a genuine typology: they are three distinct execution models
with different fault-tolerance stories, and practitioners use the words to mean specific things.

**Batch** processes a bounded input — yesterday, the last hour, one partition — on a schedule. It is
the cheapest per record because the whole set is visible at once, so the engine can sort, shuffle
and join without keeping long-lived state, and a failed run is simply run again. The cost is
freshness, and the arithmetic is unforgiving: a job that starts at 02:00 and covers up to midnight
means the number on a dashboard at 09:00 describes a world that is nine hours old.

**Micro-batch** runs the same programming model over windows of seconds to minutes. Spark Structured
Streaming works this way by default, and the appeal is that it inherits batch's fault tolerance —
each micro-batch either commits or is retried — while cutting latency by two or three orders of
magnitude. It is the pragmatic middle, and most "real-time" pipelines in production are this.

**Streaming** processes each record as it arrives with continuously maintained state, as in Flink or
Beam. Two costs are systematically underestimated. First, capacity: a batch job is sized for the
day's average, while a streaming job must absorb the peak minute, so a workload whose peak is five
times its mean pays for roughly five times the average capacity around the clock. Second,
duplication of logic — the same business rule now exists in a batch implementation and a streaming
one, and they drift. That is precisely the objection Jay Kreps raised to the Lambda architecture in
2014 when proposing what became known as Kappa: run one code path over a replayable log and
reprocess by replaying it, rather than maintaining two.

Streaming also forces the hardest question in the whole discipline: what is "now"? Events carry an
*event time* (when the thing happened) and a *processing time* (when your system saw it), and these
diverge — a phone in airplane mode buffers events and delivers them three hours late. A windowed
aggregation cannot wait forever, so it closes on a **watermark**: an assertion that nothing older
than time *T* will still arrive. The trade is exact and has no free setting. A 10-minute watermark
on hourly windows publishes quickly and drops that phone's events, or forces you to restate a figure
you already published. A 6-hour watermark captures them and makes every published number six hours
stale. This is the model formalised in the Dataflow paper (Akidau et al., VLDB 2015), which is why
event time, watermarks and triggers are named the same way across Beam, Flink and Spark.

## Real-World Applications

The modern warehouse stack is two named tools doing the two halves. **Apache Airflow**, built at
Airbnb in 2014, schedules pipelines as directed graphs — and the detail worth noticing is that
`backfill` is a first-class command, which tells you the designers assumed re-running history is
routine rather than exceptional. **dbt** turned the T of ELT into version-controlled SQL with tests
attached: `not_null`, `unique`, `accepted_values` and `relationships` assertions live in the repo
next to the model that depends on them, so a data quality rule is reviewed, diffed and blamed like
any other code.

On the ingestion side, **Apache Kafka** — built at LinkedIn and open-sourced in 2011 — made the
durable, replayable log the standard substrate: a consumer that got it wrong rewinds its offset and
reads the same bytes again instead of asking the source system to re-export. **Google Cloud
Dataflow** and **Apache Flink** implement the event-time semantics described above, which is what
makes a windowed aggregate over out-of-order data a configuration choice rather than a research
project.

Storage picked up the same guarantees. **Apache Iceberg**, opened by Netflix in 2018, alongside
Apache Hudi from Uber and Delta Lake from Databricks, put atomic commits and schema evolution on top
of object storage: overwriting a partition becomes all-or-nothing, and a renamed column tracks a
stable column ID rather than its name — the difference between a rename and an accidentally
all-null column. Uber's **Michelangelo** platform (2017) popularised the feature store for the
adjacent problem: when a feature is computed by a batch job for training and by a separate service
at [inference](https://howaiworks.ai/glossary/inference) time, the two implementations disagree, and the model in
[production](https://howaiworks.ai/glossary/production-systems) sees inputs it never trained on.

## Key Concepts

**Exactly-once is a property of your sink, not of your transport.** No messaging system can promise
a message is delivered exactly once — the acknowledgement itself can be the thing that is lost. What
you actually get is at-least-once delivery plus a way to make repetition harmless: a merge on a
natural key, an overwrite of a whole partition, or a commit that ties the write and the offset
together in one transaction. You design your way to exactly-once semantics; you do not buy them.

**Schema-on-write versus schema-on-read** decides *when* you find out. A warehouse that enforces
types at load time rejects the bad record at 02:00 with a stack trace naming the column. A lake that
accepts anything and interprets at query time takes the same record silently, and you learn about it
weeks later when an analyst notices a column is 40% null. Schema-on-read is genuinely more flexible
for semi-structured data; the flexibility is paid for in detection latency, which is why table
formats and data contracts have been pulling the lake back toward enforcement.

## Challenges

**A silent schema change is the characteristic failure of this discipline** because nothing fails.
An upstream team changes a field from a number to a string, or renames `user_id` to `userId`. A
permissive JSON parse yields null; the loading job reports success and a normal row count; the
feature pipeline imputes the median for the nulls, as it was told to; the model retrains on a column
that is now a constant. Every dashboard is green. The only visible symptom is a slow drift in a
quality metric days later, which is exactly why [monitoring](https://howaiworks.ai/glossary/monitoring) has to watch
input distributions and null rates rather than only job exit codes.

**Time zones destroy aggregates in ways that look like seasonality.** Bucket a daily total by local
wall-clock time and the spring daylight-saving day has 23 hours while the autumn one has 25 — so the
feature dips about **4.2%** (one hour in 24) on one day a year and rises about 4.2% on another. Worse,
01:30 local occurs twice in autumn, so any key of the form `(user_id, local_hour)` collides and one
of the two rows wins silently. Store UTC plus the originating zone and convert at the edge where a
human reads it; every conversion earlier than that is a chance to lose information you cannot
reconstruct.

**A null encoded as a valid value is worse than a missing one.** A source that writes `0` for "age
not provided" does not just shift the mean — with 8% of rows affected, a column whose true mean is
42 reports 38.6 — it creates a fake signal. A gradient-boosted tree will find a genuinely predictive
split at `age < 1`, because "this field is absent" correlates with the partner feed that produced
the record, and that feed correlates with the label. The model has learned to identify its own data
sources. It scores well in [cross-validation](https://howaiworks.ai/glossary/cross-validation) and collapses in
production the first time the feed mix changes, which is leakage arriving through the ingestion
layer rather than the modelling code.

**Reprocessing is a correctness problem, not just a cost problem.** Re-running last quarter with
today's transformation logic produces a table that no longer matches the numbers already reported
from it. Any pipeline whose output someone has quoted needs an answer to "which version of the logic
produced this row", which is what lineage metadata and versioned table snapshots exist to record.

## Future Trends

Iceberg is settling into the role of interchange layer between engines — Databricks acquired Tabular
in 2024, and the major warehouse vendors now read and write the format — which decouples the choice
of query engine from the physical layout of the data.

Data contracts move the schema check to the producer: rather than a consumer discovering a breaking
change after it lands, the producing service declares a versioned schema and its own CI fails on an
incompatible edit. It is an organisational fix wearing a technical hat, and the only approach that
attacks the silent-schema-change failure at its cause rather than its symptom.

The heaviest new load is unstructured. Pipelines built for rows are being asked to ingest PDFs, HTML
and scanned documents for [retrieval-augmented generation](https://howaiworks.ai/glossary/retrieval-augmented-generation),
where the failure modes are unfamiliar: a table split across a page boundary, a repeated header
absorbed into a chunk, a footnote glued to the wrong sentence. Language models are now often the
parser as well as the consumer, which makes extraction probabilistic — and a probabilistic extractor
needs the same validation discipline as any other unreliable source.

## Code Example

The difference between a pipeline you can re-run and one you cannot fits in two SQL statements.

```sql
-- Not idempotent: a replayed batch appends the same rows a second time.
-- The result depends on how many times the job ran, which is not a property
-- of the data.
INSERT INTO orders_clean
SELECT order_id, user_id, amount_cents, event_time
FROM staging_orders
WHERE ingest_date = DATE '2026-07-21';

-- Idempotent: the natural key decides. Running this once, twice or ten times
-- leaves the table in exactly the same state.
MERGE INTO orders_clean AS t
USING (
    SELECT order_id, user_id, amount_cents, event_time
    FROM staging_orders
    WHERE ingest_date = DATE '2026-07-21'
    -- The source of a MERGE must be unique on the join key, or the statement
    -- errors (or, worse, picks a row arbitrarily). At-least-once delivery
    -- means duplicates are expected, so collapse them here, deterministically.
    QUALIFY ROW_NUMBER() OVER (
        PARTITION BY order_id ORDER BY event_time DESC, ingest_sequence DESC
    ) = 1
) AS s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET
    amount_cents = s.amount_cents,
    event_time   = s.event_time
WHEN NOT MATCHED THEN INSERT (order_id, user_id, amount_cents, event_time)
    VALUES (s.order_id, s.user_id, s.amount_cents, s.event_time);
```

`QUALIFY` is warehouse syntax (Snowflake, BigQuery, Databricks); elsewhere the same deduplication is
a subquery filtering on `ROW_NUMBER() = 1`. The tie-breaker matters as much as the ranking: ordering
by `event_time` alone leaves two records with identical timestamps in a coin-flip, so the job
becomes non-deterministic in exactly the case — duplicates — it exists to handle.

## Frequently Asked Questions

### Why does data processing take up most of the time on a machine learning project?

Because it is never done once. Every source system changes shape without warning — a renamed column, a widened type, a new null convention — so the pipeline has to be re-run, re-validated and often re-written, while the model training step is a command you run again unchanged.

### What is the difference between data processing and data analysis?

Data processing is engineering: it is judged on whether the same input produces the same output every time it runs. [Data Analysis](https://howaiworks.ai/glossary/data-analysis) is the act of asking a question of the result. Processing has no opinion about what the data means; it only guarantees that the numbers are the ones the source actually reported.

### What is the difference between ETL and ELT?

ETL transforms data before loading it into the warehouse; ELT loads the raw data first and transforms it with the warehouse's own compute. The order flipped once cloud object storage made keeping the raw copy cheap — under ELT a transformation bug is fixed by re-running SQL, because the evidence still exists.

### What makes a data pipeline idempotent, and why does it matter?

A pipeline is idempotent when running the same batch twice leaves the same result as running it once — typically by merging on a natural key or overwriting a whole partition rather than appending rows. It matters because delivery is at-least-once in practice: a job that crashes after writing but before committing its offset will be replayed.

### What is a watermark in stream processing?

A watermark is the system's assertion that no more events older than a given event time will arrive, which is what lets a windowed aggregation close and emit. It is a bet, not a fact: set it short and late records are dropped, set it long and every published number is that much staler.

### How bad is a 0.1% malformed-record rate?

At 50 million events a day it is 50,000 broken records daily and about 1.5 million across a 30-day training window. Whether that matters depends entirely on whether the breakage is random noise or correlated with one source, one region or one app version — correlated errors teach the model a pattern that does not exist outside your pipeline.

## Related

### Related terms

- [Data Analysis](https://howaiworks.ai/glossary/data-analysis)
- [Machine Learning Operations (MLOps)](https://howaiworks.ai/glossary/mlops)
- [Monitoring](https://howaiworks.ai/glossary/monitoring)
- [Vectorization](https://howaiworks.ai/glossary/vectorization)
- [Data Augmentation](https://howaiworks.ai/glossary/data-augmentation)
- [Production Systems](https://howaiworks.ai/glossary/production-systems)

---

Source: https://howaiworks.ai/glossary/data-processing — HowAIWorks.ai