When should you store raw data instead of cleaning it first?
You have data pouring in from databases, logs, and documents, and one decision shapes everything downstream: clean it up front, or keep it raw? The honest answer depends on who needs it and how fast.
Two storage models, one real difference
A data lake is a repository of data stored in its natural, raw format, usually object blobs or files, per Wikipedia's data lake entry. A data warehouse, by contrast, is centralized storage built around cleaned, structured knowledge. A lake can hold structured rows and columns, semi-structured data like CSV, logs, XML, and JSON, plus unstructured emails, documents, PDFs, and binary data such as images, audio, and video. A warehouse expects none of that variety without upfront shaping. So the choice is really: transform first, or store first and transform later.
- Lake: raw format, all data types, transform later
- Warehouse: centralized, cleaned, structured storage
The pipeline tax decides your cost
Wikipedia's ETL entry defines extract, transform, load as a three-phase computing process where data are extracted from an input source, transformed, and then loaded. That middle phase is where projects stall. Every schema change, new source, or fix to a past transformation means redoing pipeline work. A warehouse forces that tax before anyone sees the data. A lake defers it, which is why teams exploring machine learning often start there: the raw copy survives every wrong assumption.
Pick by question type, not by fashion
Wikipedia notes a lake holds both raw copies of source system data and transformed data used for reporting, visualization, advanced analytics, and machine learning. James Dixon, then CTO of Pentaho, coined the term by 2011 to contrast it with a data mart, a smaller repository of interesting attributes derived from raw data. The lesson: derived, shaped views are always built on top of something. If your questions are stable and known, build the shaped view directly. If your questions are unknown or changing, keep the raw layer so you can derive again without losing the source.
What this does not solve
Definitions cannot tell you about cost at your scale, query speed, governance, or who on your team maintains the pipeline. Wikipedia's own data warehouse article flags that it needs more citations. Raw storage also creates its own risk: a pile of undocumented files is only as useful as the effort spent understanding them later. Neither choice removes the modeling work; it only changes when you pay for it.
Which of your current data questions are stable enough to lock into a schema today?
Where in your pipeline are you paying the transform tax before anyone has asked a question?
List your last five data requests and mark each one exploratory or recurring; that split is your warehouse-versus-lake answer.
Sources
Recommended for you
- Data
Why does GDPR see personal data where you see none?
Your AI product's logs and inputs probably count as personal data under GDPR, because the law's definition is far broader than the everyday idea of PII.
- Data
Why do your dashboards lie while your pipelines tell the truth?
When a metric becomes a target, people optimize the number, not the reality. Your raw pipeline has no incentive to flatter you.
- Data
Why does data quality belong to the business, not the data team?
Your data team can check schemas, but only the business knows what the data is for. Quality is fitness for purpose, and purpose lives outside engineering.
- Data
Why does hiring one AI engineer still leave your data broken?
You hired for AI and still got outputs that contradict your dashboards. The problem is the role boundary nobody drew between data work and AI work.
Enjoyed this article?
Subscribe for new articles. No spam. Unsubscribe anytime.