The basic idea
A data lake is a storage system that holds large amounts of data in its original, raw form, whatever the format. Spreadsheets, database exports, log files, JSON, images, audio and documents can all sit side by side. Nothing has to be reshaped before it goes in. The decision about how to use the data is made later, when someone needs it.
The “lake” image contrasts with a bottled-water store: water flows in from many streams and is kept as it arrived, and you draw from it when you need it. A data warehouse, in this picture, is the bottled version that has been filtered and packaged for a specific purpose.
Schema-on-read versus schema-on-write
The main technical difference is when structure is applied.
- A warehouse uses schema-on-write. Data must be cleaned and fitted to a defined table layout before it is stored. Anyone reading it later can rely on the structure.
- A lake uses schema-on-read. Data is stored as is, and structure is applied when you read it for a particular analysis.
Schema-on-write takes work up front and gives reliability. Schema-on-read saves effort at loading time and keeps flexibility, since a question nobody had thought of yet can still be answered from the raw material. The cost is that every reader must interpret the data and may do it differently.
What a lake typically holds
- Raw exports from operational databases
- Application and server logs
- Streams of readings from IoT devices
- Clickstream records of what visitors did on a website
- Documents, emails, images, video and audio
- Third-party files in assorted formats
A lot of this is unstructured or semi-structured data, which is awkward to put into rigid tables. The warehouse approach struggles with these, while a lake accepts them. The distinction is explained in the article on structured and unstructured data.
How lakes are built
A lake is usually built on inexpensive, scalable storage, often object storage offered by a provider of cloud infrastructure. Files are commonly organized into folders by source and date, sometimes in efficient formats designed for analysis. On top sit tools that can query the files, process them in bulk and move them into other systems. Because storage is separated from computing, you can store a lot without paying for heavy processing all the time. Exact costs and performance depend on the provider and setup.
Lakes versus warehouses
| Data lake | Data warehouse | |
|---|---|---|
| Data shape | Any: raw, mixed formats | Structured, cleaned, modeled |
| When structure is applied | On read | On write |
| Typical users | Data engineers and scientists exploring | Analysts and business users reading reports |
| Flexibility | High | Lower, but more consistent |
| Reliability of figures | Depends on how carefully it is managed | Usually a curated, agreed version |
| Typical cost of storage | Often lower per unit of data | Often higher, for optimized storage |
These are tendencies. Products increasingly mix both, sometimes under the name “lakehouse,” which tries to add warehouse-like management features on top of lake storage. Whether that fully delivers on both goals depends on the system.
Many organizations use both: raw data lands in a lake for safekeeping and exploration, and a refined subset is loaded into a data warehouse where reporting needs consistent numbers.
Why people choose a lake
- Keep everything. Storing raw data now preserves options later, such as retraining a model on older material or re-extracting details from documents with improved methods.
- Handle variety. Text, images and sensor streams have somewhere to go.
- Support experimentation. Data scientists can explore raw data without waiting for a formal model to be designed.
- Feed machine learning. Large and varied sets of training data are often assembled here, though quality still matters a great deal.
The data swamp problem
A lake that nobody manages tends to become a data swamp: a pile of files with unclear origin, unknown meaning and doubtful quality. Typical symptoms are folders with cryptic names, several versions of the same dataset, and no one who remembers what a column means.
Ways to prevent this include:
- A catalog that lists what datasets exist, where they came from and what the fields mean. This depends on good file metadata and descriptions.
- Clear ownership, so a named team is responsible for each dataset.
- Naming and folder conventions that people follow.
- Quality checks and the usual data cleaning before data is used for decisions.
- Retention rules that delete data when it is no longer needed.
Security and privacy
A lake that stores everything may store things it should not, including personal information scattered across logs, forms and recordings. Keeping data “just in case” can conflict with privacy rules and with the principle of keeping only what is needed. Access controls, encryption and clear policies about sensitive material are important, and regular backups of the lake apply just as they do for any important data. If a lake is exposed, the volume of material could be large, as with any serious data breaches.
Choosing between them
A short set of questions helps:
- Is most of your data tidy records, or does it include text, images and logs?
- Do you know the questions in advance, or do you need room to explore?
- How important is it that everyone sees identical, agreed figures?
- Do you have people who can manage a catalog and quality controls?
If your needs are reports on well-understood data, a warehouse is usually the direct route. If you need to collect varied data for later exploration, a lake fits. Many teams end up with both, working together.