The data engineer

The job is availability, not cleverness

A data engineer is responsible for the fact that the data is there, on time, complete, and the same as it was yesterday. Nobody thanks you when this works. Everybody notices within eleven minutes when it does not.

The mental model that helps: you are running a small logistics company. Things arrive from suppliers you do not control, at times they choose, in formats they change without telling you, and your customers downstream have already promised their customers that the goods will be on the shelf by 07:00.

The week

Monday’s pipeline failed at 03:40 because an upstream API returned a 500 for eleven minutes. Your job is not to be awake at 03:40; it is to have written the retry so that nobody had to be. Tuesday, marketing wants three years of history for a new source, so you write a backfill and discover that running it will cost more in compute than the analysis is worth. Wednesday, a source system starts sending timestamps in local time instead of UTC and nothing errors: the numbers are just wrong for a day. Thursday, you add a schema check so that particular silence cannot happen again. Friday, capacity planning.

The ideas that carry the role

Idempotency. Re-running yesterday’s job must produce yesterday’s result, not double it. This is why data engineers think in partitions: you replace the 3 March partition rather than appending to a table.

Backfills are a first-class feature, not an emergency. Any pipeline you build should be runnable for an arbitrary past date range on day one, because you will need it in month two.

Failures are loud or they are worse. A pipeline that crashes is a good pipeline. A pipeline that silently writes half the rows is the one that costs a quarter of trust. Every stage gets a row-count check, a freshness check and a null-rate check.

Data contracts. An agreement with the source team about schema, meaning and notice before changes. Mostly social, occasionally enforced in code, and the difference between a calm team and a firefighting one.

The stack

• Python and SQL. Both, properly.
• Orchestration: Airflow, Dagster or Prefect. All free and open source; managed versions cost money and change nothing about the concepts.
• Storage and compute: Snowflake, BigQuery, Databricks in industry. Free at home: PostgreSQL, DuckDB, and Parquet files on disk.
• Streaming, in some jobs: Kafka, or the managed equivalents. Redpanda and Kafka itself run free locally.
• Containers and some infrastructure: Docker, a little Terraform, enough Linux to read a log.
• Spark, if you are near large volumes; PySpark runs free on a laptop, and increasingly does not need to, because DuckDB and Polars handle on one machine what needed a cluster in 2018.

Two routes in

From software engineering or IT. The shortest path. You already have code, Git and operational instincts; you learn warehouses, orchestration and the failure modes of moving data.

From analyst or analytics engineer. Also common. You know the data and what downstream users need; you learn Python properly, testing, containers and how to be on call.

The portfolio piece that works: a pipeline that runs on a schedule, pulls from a real public API, is idempotent, has data-quality checks, fails loudly, and has a README explaining what happens when the source is down. Hosted free on a small virtual machine or a scheduled GitHub Actions workflow. This is the closest thing in the field to a project that reliably converts to interviews, because it is exactly the job.

The failure that ends the first year

Building for a scale you do not have. It is very easy to reach for Kafka, Spark and a Kubernetes cluster for 40 million rows that DuckDB would handle in eight seconds on a laptop. You then spend your year operating infrastructure instead of delivering data, the costs are visible to finance, and the value is not.

The seniority signal is the opposite instinct: choose the smallest thing that works, and be able to say precisely at what volume it stops working.

The limitation

On call is real. Many data engineering jobs carry a rota, and being woken at 04:00 because a vendor’s API changed is part of the compensation you are being paid. Ask about the rota, the frequency of pages in the last quarter, and whether there is time-off compensation. A team that cannot answer with numbers has not measured it, and unmeasured on call is usually bad on call.