{"course":{"slug":"ai-careers","title":"Working in AI","url":"https://addaly.com/learn/ai-careers","level":"beginner","track":"work"},"lesson":{"slug":"data-engineer-in-depth","title":"The data engineer","url":"https://addaly.com/learn/ai-careers/data-engineer-in-depth","position":13,"of":85,"minutes":10,"answer":"Data engineering is judged on availability and silence, so idempotency, backfills and loud failure matter more than any tool on the CV.","keyIdea":"Data engineering is judged on availability and silence, so idempotency, backfills and loud failure matter more than any tool on the CV."},"body":{"format":"markdown","text":"## The job is availability, not cleverness\n\nA data engineer is responsible for the fact that the data is there, on time, complete, and the same as it was yesterday. Nobody thanks you when this works. Everybody notices within eleven minutes when it does not.\n\nThe mental model that helps: you are running a small logistics company. Things arrive from suppliers you do not control, at times they choose, in formats they change without telling you, and your customers downstream have already promised their customers that the goods will be on the shelf by 07:00.\n\n## The week\n\nMonday's pipeline failed at 03:40 because an upstream API returned a 500 for eleven minutes. Your job is not to be awake at 03:40; it is to have written the retry so that nobody had to be. Tuesday, marketing wants three years of history for a new source, so you write a backfill and discover that running it will cost more in compute than the analysis is worth. Wednesday, a source system starts sending timestamps in local time instead of UTC and nothing errors: the numbers are just wrong for a day. Thursday, you add a schema check so that particular silence cannot happen again. Friday, capacity planning.\n\n## The ideas that carry the role\n\n**Idempotency.** Re-running yesterday's job must produce yesterday's result, not double it. This is why data engineers think in partitions: you replace the 3 March partition rather than appending to a table.\n\n**Backfills are a first-class feature, not an emergency.** Any pipeline you build should be runnable for an arbitrary past date range on day one, because you will need it in month two.\n\n**Failures are loud or they are worse.** A pipeline that crashes is a good pipeline. A pipeline that silently writes half the rows is the one that costs a quarter of trust. Every stage gets a row-count check, a freshness check and a null-rate check.\n\n**Data contracts.** An agreement with the source team about schema, meaning and notice before changes. Mostly social, occasionally enforced in code, and the difference between a calm team and a firefighting one.\n\n## The stack\n\n- **Python and SQL.** Both, properly.\n- **Orchestration**: Airflow, Dagster or Prefect. All free and open source; managed versions cost money and change nothing about the concepts.\n- **Storage and compute**: Snowflake, BigQuery, Databricks in industry. Free at home: PostgreSQL, DuckDB, and Parquet files on disk.\n- **Streaming**, in some jobs: Kafka, or the managed equivalents. Redpanda and Kafka itself run free locally.\n- **Containers and some infrastructure**: Docker, a little Terraform, enough Linux to read a log.\n- **Spark**, if you are near large volumes; PySpark runs free on a laptop, and increasingly does not need to, because DuckDB and Polars handle on one machine what needed a cluster in 2018.\n\n## Two routes in\n\n**From software engineering or IT.** The shortest path. You already have code, Git and operational instincts; you learn warehouses, orchestration and the failure modes of moving data.\n\n**From analyst or analytics engineer.** Also common. You know the data and what downstream users need; you learn Python properly, testing, containers and how to be on call.\n\nThe portfolio piece that works: a pipeline that runs on a schedule, pulls from a real public API, is idempotent, has data-quality checks, fails loudly, and has a README explaining what happens when the source is down. Hosted free on a small virtual machine or a scheduled GitHub Actions workflow. This is the closest thing in the field to a project that reliably converts to interviews, because it is exactly the job.\n\n## The failure that ends the first year\n\nBuilding for a scale you do not have. It is very easy to reach for Kafka, Spark and a Kubernetes cluster for 40 million rows that DuckDB would handle in eight seconds on a laptop. You then spend your year operating infrastructure instead of delivering data, the costs are visible to finance, and the value is not.\n\nThe seniority signal is the opposite instinct: choose the smallest thing that works, and be able to say precisely at what volume it stops working.\n\n## The limitation\n\nOn call is real. Many data engineering jobs carry a rota, and being woken at 04:00 because a vendor's API changed is part of the compensation you are being paid. Ask about the rota, the frequency of pages in the last quarter, and whether there is time-off compensation. A team that cannot answer with numbers has not measured it, and unmeasured on call is usually bad on call."},"check":{"question":"A nightly pipeline writes only 40% of the expected rows because a source API rate-limited it, but the job exits successfully and the dashboard updates. Why is this worse than the job crashing outright?","options":[{"text":"Because a crash would have triggered the retry logic, and retries always recover rate-limit failures.","correct":false,"tempting":"Assumes retries fix the underlying cause; rate limits often need backoff or a schedule change, not another attempt."},{"text":"Because incomplete data that looks complete is acted on, and the error is only discovered later when trust is already spent, whereas a crash is noticed immediately and blocks the bad numbers from being used.","correct":true},{"text":"Because partial writes corrupt the table's schema and require a full rebuild.","correct":false,"tempting":"Describes a structural failure that did not happen; the schema is fine, and that is precisely why nothing alerted."},{"text":"Because the pipeline used more compute than budgeted for a partial result.","correct":false,"tempting":"Names a real but minor cost while missing that decisions are now being made on numbers that are 60% short."}]},"prev":{"slug":"analytics-engineer-in-depth","title":"The analytics engineer","url":"https://addaly.com/learn/ai-careers/analytics-engineer-in-depth"},"next":{"slug":"ml-engineer-in-depth","title":"The machine learning engineer","url":"https://addaly.com/learn/ai-careers/ml-engineer-in-depth"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The data engineer', in Working in AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/ai-careers/data-engineer-in-depth (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The data engineer","courseTitle":"Working in AI","module":{"index":2,"slug":"the-roles-in-depth","title":"The roles, one at a time"},"answer":null,"keyIdea":"Data engineering is judged on availability and silence, so idempotency, backfills and loud failure matter more than any tool on the CV.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-job-is-availability-not-cleverness","runs":[{"text":"The job is availability, not cleverness"}]},{"t":"p","runs":[{"text":"A data engineer is responsible for the fact that the data is there, on time, complete, and the same as it was yesterday. Nobody thanks you when this works. Everybody notices within eleven minutes when it does not."}]},{"t":"p","runs":[{"text":"The mental model that helps: you are running a small logistics company. Things arrive from suppliers you do not control, at times they choose, in formats they change without telling you, and your customers downstream have already promised their customers that the goods will be on the shelf by 07:00."}]},{"t":"h2","id":"the-week","runs":[{"text":"The week"}]},{"t":"p","runs":[{"text":"Monday’s pipeline failed at 03:40 because an upstream API returned a 500 for eleven minutes. Your job is not to be awake at 03:40; it is to have written the retry so that nobody had to be. Tuesday, marketing wants three years of history for a new source, so you write a backfill and discover that running it will cost more in compute than the analysis is worth. Wednesday, a source system starts sending timestamps in local time instead of UTC and nothing errors: the numbers are just wrong for a day. Thursday, you add a schema check so that particular silence cannot happen again. Friday, capacity planning."}]},{"t":"h2","id":"the-ideas-that-carry-the-role","runs":[{"text":"The ideas that carry the role"}]},{"t":"p","runs":[{"text":"Idempotency.","b":true},{"text":" Re-running yesterday’s job must produce yesterday’s result, not double it. This is why data engineers think in partitions: you replace the 3 March partition rather than appending to a table."}]},{"t":"p","runs":[{"text":"Backfills are a first-class feature, not an emergency.","b":true},{"text":" Any pipeline you build should be runnable for an arbitrary past date range on day one, because you will need it in month two."}]},{"t":"p","runs":[{"text":"Failures are loud or they are worse.","b":true},{"text":" A pipeline that crashes is a good pipeline. A pipeline that silently writes half the rows is the one that costs a quarter of trust. Every stage gets a row-count check, a freshness check and a null-rate check."}]},{"t":"p","runs":[{"text":"Data contracts.","b":true},{"text":" An agreement with the source team about schema, meaning and notice before changes. Mostly social, occasionally enforced in code, and the difference between a calm team and a firefighting one."}]},{"t":"h2","id":"the-stack","runs":[{"text":"The stack"}]},{"t":"ul","items":[[{"text":"Python and SQL.","b":true},{"text":" Both, properly."}],[{"text":"Orchestration","b":true},{"text":": Airflow, Dagster or Prefect. All free and open source; managed versions cost money and change nothing about the concepts."}],[{"text":"Storage and compute","b":true},{"text":": Snowflake, BigQuery, Databricks in industry. Free at home: PostgreSQL, DuckDB, and Parquet files on disk."}],[{"text":"Streaming","b":true},{"text":", in some jobs: Kafka, or the managed equivalents. Redpanda and Kafka itself run free locally."}],[{"text":"Containers and some infrastructure","b":true},{"text":": Docker, a little Terraform, enough Linux to read a log."}],[{"text":"Spark","b":true},{"text":", if you are near large volumes; PySpark runs free on a laptop, and increasingly does not need to, because DuckDB and Polars handle on one machine what needed a cluster in 2018."}]]},{"t":"h2","id":"two-routes-in","runs":[{"text":"Two routes in"}]},{"t":"p","runs":[{"text":"From software engineering or IT.","b":true},{"text":" The shortest path. You already have code, Git and operational instincts; you learn warehouses, orchestration and the failure modes of moving data."}]},{"t":"p","runs":[{"text":"From analyst or analytics engineer.","b":true},{"text":" Also common. You know the data and what downstream users need; you learn Python properly, testing, containers and how to be on call."}]},{"t":"p","runs":[{"text":"The portfolio piece that works: a pipeline that runs on a schedule, pulls from a real public API, is idempotent, has data-quality checks, fails loudly, and has a README explaining what happens when the source is down. Hosted free on a small virtual machine or a scheduled GitHub Actions workflow. This is the closest thing in the field to a project that reliably converts to interviews, because it is exactly the job."}]},{"t":"h2","id":"the-failure-that-ends-the-first-year","runs":[{"text":"The failure that ends the first year"}]},{"t":"p","runs":[{"text":"Building for a scale you do not have. It is very easy to reach for Kafka, Spark and a Kubernetes cluster for 40 million rows that DuckDB would handle in eight seconds on a laptop. You then spend your year operating infrastructure instead of delivering data, the costs are visible to finance, and the value is not."}]},{"t":"p","runs":[{"text":"The seniority signal is the opposite instinct: choose the smallest thing that works, and be able to say precisely at what volume it stops working."}]},{"t":"h2","id":"the-limitation","runs":[{"text":"The limitation"}],"role":"limit"},{"t":"p","runs":[{"text":"On call is real. Many data engineering jobs carry a rota, and being woken at 04:00 because a vendor’s API changed is part of the compensation you are being paid. Ask about the rota, the frequency of pages in the last quarter, and whether there is time-off compensation. A team that cannot answer with numbers has not measured it, and unmeasured on call is usually bad on call."}]}],"check":{"question":[{"text":"A nightly pipeline writes only 40% of the expected rows because a source API rate-limited it, but the job exits successfully and the dashboard updates. Why is this worse than the job crashing outright?"}],"options":[{"text":[{"text":"Because a crash would have triggered the retry logic, and retries always recover rate-limit failures."}],"correct":false,"tempting":[{"text":"Assumes retries fix the underlying cause; rate limits often need backoff or a schedule change, not another attempt."}]},{"text":[{"text":"Because incomplete data that looks complete is acted on, and the error is only discovered later when trust is already spent, whereas a crash is noticed immediately and blocks the bad numbers from being used."}],"correct":true,"tempting":null},{"text":[{"text":"Because partial writes corrupt the table’s schema and require a full rebuild."}],"correct":false,"tempting":[{"text":"Describes a structural failure that did not happen; the schema is fine, and that is precisely why nothing alerted."}]},{"text":[{"text":"Because the pipeline used more compute than budgeted for a partial result."}],"correct":false,"tempting":[{"text":"Names a real but minor cost while missing that decisions are now being made on numbers that are 60% short."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"analytics-engineer-in-depth","title":"The analytics engineer","minutes":9},"next":{"slug":"ml-engineer-in-depth","title":"The machine learning engineer","minutes":10}}}