Back to blog

When should you store raw data instead of cleaning it first?

Thắng Đoàn
Thắng Đoàn

You have data pouring in from databases, logs, and documents, and one decision shapes everything downstream: clean it up front, or keep it raw? The honest answer depends on who needs it and how fast.

Two storage models, one real difference

A data lake is a repository of data stored in its natural, raw format, usually object blobs or files, per Wikipedia's data lake entry. A data warehouse, by contrast, is centralized storage built around cleaned, structured knowledge. A lake can hold structured rows and columns, semi-structured data like CSV, logs, XML, and JSON, plus unstructured emails, documents, PDFs, and binary data such as images, audio, and video. A warehouse expects none of that variety without upfront shaping. So the choice is really: transform first, or store first and transform later.

  • Lake: raw format, all data types, transform later
  • Warehouse: centralized, cleaned, structured storage

The pipeline tax decides your cost

Wikipedia's ETL entry defines extract, transform, load as a three-phase computing process where data are extracted from an input source, transformed, and then loaded. That middle phase is where projects stall. Every schema change, new source, or fix to a past transformation means redoing pipeline work. A warehouse forces that tax before anyone sees the data. A lake defers it, which is why teams exploring machine learning often start there: the raw copy survives every wrong assumption.

Pick by question type, not by fashion

Wikipedia notes a lake holds both raw copies of source system data and transformed data used for reporting, visualization, advanced analytics, and machine learning. James Dixon, then CTO of Pentaho, coined the term by 2011 to contrast it with a data mart, a smaller repository of interesting attributes derived from raw data. The lesson: derived, shaped views are always built on top of something. If your questions are stable and known, build the shaped view directly. If your questions are unknown or changing, keep the raw layer so you can derive again without losing the source.

What this does not solve

Definitions cannot tell you about cost at your scale, query speed, governance, or who on your team maintains the pipeline. Wikipedia's own data warehouse article flags that it needs more citations. Raw storage also creates its own risk: a pile of undocumented files is only as useful as the effort spent understanding them later. Neither choice removes the modeling work; it only changes when you pay for it.

Which of your current data questions are stable enough to lock into a schema today?
Where in your pipeline are you paying the transform tax before anyone has asked a question?

List your last five data requests and mark each one exploratory or recurring; that split is your warehouse-versus-lake answer.

Sources

Wikipedia: Data warehouse

Wikipedia: Data lake

Wikipedia: Extract, transform, load

Share:

Recommended for you

Enjoyed this article?

Subscribe for new articles. No spam. Unsubscribe anytime.

By subscribing you agree to receive the newsletter. No spam, and you can unsubscribe anytime.