A data pipeline that fails silently is more dangerous than one that fails visibly. When failure is visible, someone acts. When it is silent, the dashboard keeps showing numbers, the decision is made based on them and the error surfaces weeks later when operational impact forces an investigation. Inconsistent data that appears correct is the most expensive kind of data problem because the cost is embedded in the decisions it fed.
Subcapability 02 of 05 · Data & Analytics
Data Engineering & Pipelines
Apache stack with orchestration, observability and idempotency that eliminates the artisanal pipeline without monitoring and ensures no executive dashboard ever shows a wrong number with the appearance of a correct one.
What is at stake
The pipeline ran. The dashboard is green. The decision was made. Three days later someone discovers the data has been wrong since Monday. That is the cost of a pipeline without observability: the silent failure feeds a wrong decision for days without anyone knowing. The visible failure at least generates an alert.
What it is, in practice
How we work
Orchestration with Apache Airflow
We implement DAGs with explicit dependencies, retry with backoff and observability for every execution. Failure at any step generates an immediate alert, not a late discovery in the results meeting. Every pipeline has an owner, documentation and alert criteria.
Data contracts and embedded data quality
We instrument every pipeline with automatic validations via Great Expectations or dbt tests. Data that arrives outside the contract blocks the pipeline before contaminating the consumption layer. Quality becomes an automatic gate rather than a periodic manual review.
Streaming and batch with Apache Kafka and Spark
We structure the decision between real-time data via streaming with Apache Kafka and historical data in batch with Apache Spark. The choice follows the business decision time, not the team technical preference.
Apache Iceberg as open table format
We adopt Apache Iceberg for ACID support in the data lake, schema evolution without breaking consumers and time travel that allows retroactive auditing of any historical point. Data in the lake now has the same guarantees as a transactional database.
Change Data Capture for integration without operational impact
We capture changes in transactional systems via CDC without batch queries that impact operations. Fresh data reaches analytics minutes after the event, without the source system perceiving the additional load.
Measurable gains
What changes in the result when this subcapability matures.
Pipeline execution success rate per week
Orchestration with observability and automatic alerts raises pipeline reliability. A team that operated with unmonitored scripts begins measuring and maintaining an execution rate above 99% in a verifiable way.
Time between event and data availability for decision
Pipeline freshness measures the lag between the event in the operational system and the moment the data is available to the analyst or model. Reducing that lag from hours to minutes changes the operational response speed.
Cost of incident per decision based on incorrect data
Data contracts and automatic validation in the pipeline eliminate the most expensive category of error: data that looks correct and is wrong. The cost of a decision based on an incorrect number rarely shows up in the IT report, but appears in the financial result.
Recovery time after detected pipeline failure
A pipeline with configured retry, immediate alerts and a recovery runbook reduces the time between failure detection and data availability to the consumer. What previously took a day of investigation now has a process with a defined deadline.
Frequently asked questions
What is ETL and what is the difference from modern ELT?
ETL (Extract, Transform, Load) processes and transforms data before loading it to the destination. ELT (Extract, Load, Transform) loads first and transforms after, leveraging the processing capacity of modern data warehouses or data lakes. ELT has become the standard with platforms like BigQuery, Snowflake and Redshift because it eliminates the transformation bottleneck outside the destination and allows versioned declarative transformations via dbt.
When should streaming be used versus batch?
Batch processes accumulated volumes in defined windows: hourly, daily, weekly. Streaming processes each event at the moment it occurs. The choice follows the business decision time: fraud detection requires streaming because the block needs to happen before the transaction confirms. A monthly sales report works perfectly in a nightly batch. Many organizations operate both in parallel for distinct use cases.
What is a data contract and why does it matter?
A data contract is the formal agreement between data producer and consumer that specifies schema, expected minimum quality, latency and notification process when something changes. With the contract, schema changes do not break the consumer unexpectedly. The producer notifies, the consumer adapts within the agreed timeframe. Without a contract, changes arrive as bugs in the dashboard of whoever consumes the data.
How do you measure the health of a data infrastructure?
Four indicators cover operational health: pipeline reliability (percentage of successful executions), pipeline freshness (lag between event and data availability), Mean Time to Recover (recovery time after detected failure) and data quality score by domain. Each indicator has a defined operational threshold. When exceeded, it generates an alert before the consumer notices.
Apache Airflow, Prefect or Dagster: which to choose?
Apache Airflow is the market standard with the largest adoption and integration ecosystem. Prefect and Dagster are more modern alternatives with a more user-friendly interface and better native observability. The choice depends on team maturity, pipeline volume and the need for integrations with specific tools. For teams starting out, Airflow with a managed cloud service reduces operational burden without sacrificing robustness.
Other subcapabilities in this capability
Data Architecture & Governance
Data Management Body of Knowledge and Data Mesh with federated governance that structure data as a formal asset with owner, traceable lineage and domain accountability that scales without a central bottleneck and enables AI in production.
Analytics & Business Intelligence
Semantic layer, governed self-service and KPIs wired to the result that replace expensive intuition with evidence-based decisions and eliminate the debate about which number is right in every executive meeting.
Data Quality & Master Data
Six data quality dimensions and MDM Hub Architecture that eliminate the three versions of the same customer across systems and transform data from a source of debate into a verifiable base for every executive decision.
Data Products
DATSIS principles and Data Contracts that transform ownerless datasets into products with SLA, defined consumers and explicit accountability, eliminating the central bottleneck no backlog can absorb.
Want clarity on where to invest first?
A complete technology capability assessment with an evolution roadmap connected to financial result.