When data grows to the point where it no longer fits or can be processed on a single machine, we talk about Big Data. It is not just a matter of size: the challenge is classically summarised in the five Vs—volume, velocity, variety, veracity and value—and it forces us to rethink how information is stored, processed and governed. This article walks through distributed processing paradigms (from MapReduce to Spark), storage architectures (data lake, data warehouse and lakehouse), batch versus stream processing, and the governance and data-protection obligations that no serious project can ignore.
The five Vs of Big Data
Volume is the obvious dimension: terabytes or petabytes that exceed the capacity of a conventional server. Velocity refers to the pace of generation and ingestion—IoT sensors, clicks, transactions—which often requires processing in near real time. Variety covers structured data (tables), semi-structured data (JSON, logs) and unstructured data (text, images, video). Veracity points to quality and reliability: dirty data produces false conclusions with the appearance of rigour. And value is the acid test: no piece of data is worth anything in itself, only for the decision it enables. Designing with all five Vs in mind at once avoids the mistake of optimising for volume and neglecting quality.
From the MapReduce paradigm to Apache Spark
Distributed processing was born with MapReduce, the model popularised by Hadoop: a map phase that transforms and filters data in parallel across many nodes, and a reduce phase that aggregates the results. Its strength is fault tolerance on cheap hardware; its weakness is slowness, because it writes to disk between each stage. The underlying storage, HDFS (Hadoop Distributed File System), distributes and replicates data blocks to survive the failure of individual nodes.
Apache Spark displaced MapReduce by processing in memory, which makes it orders of magnitude faster on iterative workloads such as model training. Spark offers a unified API for SQL (DataFrames), stream processing (Structured Streaming), graphs and machine learning (MLlib), so that a single engine covers the entire analytics cycle. Its lazy evaluation execution model builds a graph of operations and optimises it before running, avoiding unnecessary work.
Where to store the data: lake, warehouse and lakehouse
The data warehouse stores structured, clean and modelled data for fast analytical queries; it follows schema-on-write and is ideal for business reporting. The data lake stores raw data of any format at low cost, applying schema-on-read; it offers flexibility but, without governance, degenerates into a data swamp where no one finds or trusts anything.
The lakehouse architecture, dominant in 2026, combines the best of both: the flexibility and low cost of the lake with the transactional guarantees and performance of the warehouse, thanks to open table formats such as Delta Lake, Apache Iceberg or Hudi. These formats bring ACID transactions, version control (time travel) and schema evolution over files in object storage, which eliminates data duplication between lake and warehouse.
Comparison of storage architectures
| Criterion | Data Warehouse | Data Lake | Lakehouse |
|---|---|---|---|
| Data type | Structured | Any | Any |
| Schema | On write | On read | Flexible + transactional |
| Storage cost | High | Low | Low |
| ACID transactions | Yes | No | Yes |
| Best suited for | Business reporting | Raw data, ML | Unified BI and ML |
Batch and real-time processing
There are two fundamental modes of processing. Batch processing operates on complete datasets at scheduled intervals; it is efficient and simple for daily reports or historical loads. Stream processing analyses events as they arrive, with latency of seconds or milliseconds, essential for fraud detection, alerting or live personalisation. Tools such as Apache Kafka act as the backbone for event ingestion, and engines such as Apache Flink or Spark's own Structured Streaming consume those streams.
The major architectural pattern worth knowing is the debate between the Lambda architecture (two paths, batch and streaming, that reconcile) and the Kappa architecture (a single streaming path that also reprocesses history). The trend leans towards Kappa for its lower operational complexity, but the choice depends on the latency and accuracy requirements of each case.
Data quality: veracity as the foundation
Of the five Vs, veracity is the easiest to neglect and the most expensive to ignore. A flawless analysis on dirty data produces erroneous conclusions with the appearance of scientific rigour, which is worse than having no analysis at all. Data quality is managed by measuring several concrete dimensions: accuracy (does the data reflect reality?), completeness (are fields or records missing?), consistency (do the values match across systems?), timeliness (does the data arrive in time for the decision?) and uniqueness (are there duplicates inflating the counts?).
In modern architectures, these checks are automated as quality tests run at each step of the pipeline, so that a batch failing the thresholds is blocked or flagged before contaminating the consumption zones. A common pattern organises the lakehouse in layers (raw, refined and consumption), applying increasing validations at each hop, so that only data that has passed the quality filter reaches the analysis. Treating quality as a feature of the data product, and not as a one-off clean-up, is what distinguishes a sustainable data programme from an effort that degrades over time.
Data governance and regulatory compliance
A Big Data project without governance is a legal risk waiting to materialise. The General Data Protection Regulation (GDPR) imposes principles that directly affect the architecture: minimisation (not hoarding personal data "just in case"), purpose limitation, storage limitation and the right to erasure, which in an immutable data lake forces you to design selective-deletion mechanisms. Pseudonymisation and encryption at rest and in transit are required technical measures. Data governance adds catalogues, lineage (knowing where each piece of data comes from and what has been done with it) and role-based access control, all of it alignable with frameworks such as the ISO/IEC 27001 information-security standard. When data feeds AI models, you must also consider the quality and traceability obligations of the European AI Regulation.
Phased implementation
An orderly data programme advances as follows: (1) define the business questions and concrete use cases before moving a single byte; (2) design ingestion and the catalogue with lineage from day one; (3) establish the lakehouse zones (raw, refined, consumption) with quality rules at each hop; (4) apply governance and access control before opening access, not after; (5) choose the processing mode (batch or streaming) according to the latency the decision really needs; and (6) measure the cost per query and the value delivered, because cheap storage encourages hoarding data that nobody uses.
Common mistakes to avoid
The first is the data swamp: dumping data without catalogue or quality until the lake becomes unusable. The second is opting for streaming when a nightly batch is enough, multiplying operational complexity with no real value. The third is ignoring the GDPR in the architecture, discovering too late that you cannot delete the data of a user who exercises the right to erasure. The fourth is measuring success by terabytes stored rather than by improved decisions. And the fifth is underestimating cloud cost: a poorly designed query over petabytes can generate a surprising bill.
Frequently asked questions
When does it stop being "normal data" and become Big Data?
There is no fixed threshold in gigabytes. The turning point comes when volume, velocity or variety exceed what a single machine or a traditional relational database can handle efficiently, forcing distributed processing.
Does Spark completely replace Hadoop?
Spark replaces the MapReduce processing engine for its in-memory speed, but it usually coexists with distributed storage (HDFS or, more commonly today, cloud object storage). Hadoop as an ecosystem has lost prominence to the lakehouse.
What is a lakehouse and why has it prevailed?
It is an architecture that brings ACID transactions, versioning and warehouse performance over the cheap, flexible data of a data lake, by means of open table formats. It has prevailed because it eliminates duplication between lake and warehouse and unifies business analytics with machine learning.
How does the GDPR affect a data lake?
It requires being able to locate and delete specific personal data (right to erasure), apply minimisation, pseudonymisation and encryption, and maintain traceability. This forces you to design the lake with a catalogue, lineage and selective-deletion mechanisms from the start.
Conclusion
Big Data is not about storing everything possible, but about turning large volumes into reliable decisions without losing quality, cost control or legal compliance along the way. The lakehouse architecture and engines such as Spark have solved much of the technical problem, but the differentiating factor remains governance: a catalogue with lineage, quality rules in each zone and a GDPR-compatible design from day one separate a data asset from a legal liability. Hoarding petabytes that nobody queries is not a victory; it is a recurring cost disguised as strategy. At Summum Marketing we always start from the business question and work towards the infrastructure, not the other way round, because the value of a piece of data is measured by the decision it enables, never by its size.