Skip to content
Deadline to Die

The Data Lifecycle: From Source to Sink

🌀 The Data Lifecycle: From Source to Sink

The data lifecycle describes the various stages that data passes through within a data system. Understanding this is fundamental to designing robust data architectures.

🏗️ 1. Ingestion (Collect)

The first stage involves moving data from sources (databases, APIs, logs) into your system.

  • Batch Ingestion: Data is moved in chunks at scheduled intervals.
  • Stream Ingestion: Data is moved in real-time as events occur.

🧹 2. Transformation (Clean & Enrich)

Raw data is rarely ready for analysis. It must be:

  • Cleaned: Handling nulls, duplicates, and incorrect formats.
  • Standardized: Ensuring consistent units and naming conventions.
  • Enriched: Merging data with other sources to add context.

💾 3. Storage (Persist)

Where the data lives.

  • Data Lake: Raw, unstructured data (e.g., S3, ADLS).
  • Data Warehouse: Structured, optimized for queries (e.g., Snowflake, BigQuery).
  • Data Lakehouse: Merges the two for high-performance analytics on raw data.

📊 4. Serving (Analyze & Visualize)

The final stage where data is used by:

  • BI Tools: Dashboards for business users.
  • Data Scientists: Building ML models.
  • APIs: Powering applications.

🛡️ 5. Data Governance & Security

Transversal to all stages, ensuring data is secure, compliant (GDPR/CCPA), and high-quality.

🏁 Summary: Best Practices

  1. Idempotency: Ensure that running the same ingestion task twice doesn’t create duplicate data.
  2. Schema Enforcement: Validate data structure early in the lifecycle.
  3. Auditability: Keep track of who changed what data and when.