The job of building the systems that make an organisation's data answerable for everyone downstream: a different job from analysis, not a rung above it, and how UK teams build it.
What is Data Engineering?
By James Cotton · Last updated · 8 min read
Part of our topic guide on Data Engineering.
By James Cotton, Founder, iO-Sphere
Data engineering is the work of building and running the systems that make an organisation's data answerable: accurate, fresh, available, and shaped so that the people and processes relying on it can trust what they get. If a data analyst's job is to answer a question with data, a data engineer's job is to make sure the question can be answered at all.
The recognisable unit of the work is the data pipeline: the route that carries data from where it is produced, cleans and reshapes it, and lands it somewhere dependable on a schedule. Without that work, an analytics team spends its days pulling extracts out of dashboards and stitching spreadsheets together by hand.
What a data engineer actually builds
The work varies by organisation, but the core responsibilities are consistent:
- Build and run the pipelines that move data from source systems (a CRM, a billing system, the product database, third-party APIs) into a warehouse in a clean, consistent shape.
- Design the warehouse itself: how tables are structured for speed and clarity, and who is allowed to see what.
- Own data quality, catching broken sources, missing values, duplicates and schema changes before they ever reach a dashboard or a model.
- Model the data other teams depend on, so analysts and scientists work from reliable, well-named tables instead of rebuilding the same logic each time.
- Run the infrastructure: cloud warehouses, the schedulers that trigger pipelines, version control, and the same testing and release discipline software teams use.
The work has far more in common with software engineering than with spreadsheet analysis. A modern data team writes code, reviews each other's changes, and runs its pipelines like production services that other people depend on.
Analyst, engineer, scientist: three jobs, not three rungs
These three roles overlap, and on a small team one person often covers all three. The responsibilities behind them are distinct:
- A data analyst answers business questions with data: building dashboards, running one-off analyses, and turning what they find into decisions.
- A data engineer builds and runs the systems that let the analyst and the scientist do that work reliably and at scale.
- A data scientist builds statistical and machine-learning models on top of that data: forecasting, segmentation, recommendation, classification.
The common picture is a ladder: analyst, then senior analyst, then engineer, a tier you reach once you are good enough at analysis. It misleads, because engineering is a different kind of work, not simply harder analysis. The UK qualifications reflect that. The Level 5 Data Engineer standard (ST1386) is its own occupational standard with its own assessment, not an upgraded version of the Level 4 Data Analyst standard. You do not climb a tier into engineering; you cross into a different occupation.
Data science, for its part, sits on top of both. It combines analysis and engineering and runs them at scale, which is why the strongest data scientists have usually done both jobs first.
Why data reliability became a production problem
For a long time the last line of defence on data quality was a person. A dashboard was read on a Monday morning by someone who might frown at a figure that looked wrong and go and check it before acting. That human pause is disappearing. The systems that now consume data (scheduled jobs, automated decisions, increasingly AI models) do not stop to doubt a number. They take what they are given and act on it.
So freshness, accuracy and availability stopped being housekeeping the team could catch up on later. They became live production concerns, watched and maintained the way any service that other things depend on has to be. This is what raises the stakes on the engineer's work: when the data is late, or wrong in a way no one catches in time, there is often no longer anyone downstream who notices before it does damage.
Healthy data teams versus struggling ones
When a data team struggles, the instinct is to blame the tools and go shopping for a new warehouse or a new orchestrator. The difference between a healthy data team and a struggling one is rarely the vendor it chose. It is whether the team treats data as a product: with clear contracts between the layers that produce data and the ones that consume it, tests that catch a regression before a downstream user ever sees it, and a habit of reversibility, so any change can be rolled back cleanly.
Tooling helps, but it does not supply that discipline. Two teams can buy the identical stack and get opposite results, because what separates them is how they work: who owns a broken pipeline, how a change gets reviewed, and whether the team finds a problem before its users do.
Building the capability: grow engineers, or hire them
Employers who need data engineering capability have two broad options: hire it on the open market, or grow it from people already inside the organisation.
Hiring experienced engineers works, and for an urgent, well-funded need it can be the fastest route. The catch is that an experienced engineer commands a competitive salary, and a senior hire who does not already know your data or your business still has to learn both before the work pays off.
Growing engineers in-house takes longer to start but tends to produce people who fit the work better. In our experience training apprentice data engineers, the strongest rarely arrive from a computer science background. They come from adjacent roles, analytics, operations, software support, and they arrive already understanding what the data is for and what it costs when it is wrong, because they have been downstream of a broken pipeline themselves. That downstream understanding is hard to hire and quicker to grow.
In England, the funded route for this is the Level 5 Data Engineer apprenticeship, built on the ST1386 standard and approved for delivery. iO-Sphere runs it as about 15 months of training plus a 3-month end-point assessment window, set against the standard's typical 24 months, and delivers up to Level 5. The learner trains on real production work, so the capability lands inside your team rather than in a classroom.
A single workplace cannot reliably produce every situation a data engineer needs to face. iO-Sphere fills that gap with realistic business case studies and simulations, so learners meet problems they would otherwise only hit years later.
For apprenticeship starts from 1 August 2026, the 2026-27 funding rules apply:
- A learner aged 16 to 24 is fully funded at any employer: the employer pays nothing.
- A learner aged 25 or over at a non-levy employer is 95% government funded, so the employer pays 5%.
- A levy-paying employer funds it from its Growth and Skills Levy account; where that account cannot cover a learner aged 25 or over, the employer co-invests 25%.
- Employers can also receive a levy transfer of up to 50% of another employer's unused funds.
A funded learner is never charged for their training. Off-the-job training happens inside normal paid working hours. Growing an engineer this way takes months, and you need a role for them to do the work in. In return you get capability grown to fit your business, with the training paid for through apprenticeship funding.
Where to go next
Two readers usually arrive at this page.
If you are weighing up data engineering as a career, the next question is which route in fits you: apprenticeship, degree, bootcamp or self-teaching. Our guide to how to become a data engineer in the UK walks through the routes, the entry requirements and how to choose. If you are deciding between engineering and analysis, data engineer versus data analyst draws that line.
If you are an employer building this capability, the Level 5 Data Engineering apprenticeship is a funded, practice-based way to grow engineers who already understand your data. To talk through whether it fits your team and how the funding works for you, start with apprenticeships for employers.
FAQ
What is data engineering, in simple terms?
Data engineering is the work of building and running the systems that make an organisation's data answerable: accurate, fresh, available, and in a shape people and downstream systems can trust. Where an analyst answers a question with data, an engineer builds and maintains the pipelines, warehouses and quality checks that make the question answerable at all, so that everyone relying on the data, from dashboards to automated processes, can depend on what they get.
What is the difference between a data engineer, a data analyst and a data scientist?
A data analyst answers business questions with data, building dashboards and analyses that inform decisions. A data engineer builds and runs the systems that make that data reliable and available in the first place. A data scientist builds statistical and machine-learning models on top of that data, such as forecasts and recommendations. The roles overlap on small teams where one person covers all three, but the work is distinct rather than three levels of the same job.
Is a data engineer just a senior data analyst?
No. Engineering is a different kind of work from analysis, and the UK qualifications reflect that: the Level 5 Data Engineer standard (ST1386) is its own occupational standard with its own assessment, not an upgraded version of the Level 4 Data Analyst standard. Analysts and engineers often share tools and a technical floor, but the analyst produces answers while the engineer produces the reliable conditions under which answers are possible, and owns the failure when the data is late or wrong.
What makes a data team healthy rather than just well-equipped?
The difference is rarely which warehouse or orchestrator the team bought. A healthy data team treats data as a product: it sets clear contracts between the layers that produce data and the ones that consume it, tests changes so a regression is caught before a downstream user sees it, and keeps every change reversible so it can be rolled back cleanly. Two teams can buy the identical stack and get opposite results, because what separates them is how they work.
How can a UK employer build data engineering capability in-house?
The funded route in England is the Level 5 Data Engineer apprenticeship, built on the ST1386 standard and approved for delivery. iO-Sphere delivers it as about 15 months of training plus a 3-month end-point assessment window, against the standard's typical 24 months, with the learner trained on real production work. For starts from 1 August 2026, a learner aged 16 to 24 is fully funded at any employer, a learner aged 25 or over at a non-levy employer costs the employer 5%, and levy-paying employers fund it from their Growth and Skills Levy account, so a funded learner is never charged.
Want to become a data engineer?
Our Level 5 Data Engineering apprenticeship is levy-funded, at no cost to you as the learner. 18 months from candidate to confident contributor.