Ask any data science student what they want to work on, and the answer is almost always some version of "build models," "do machine learning," "work with AI." Ask them how they feel about spending 70-80% of a real project cleaning and prepa...
Ask any data science student what they want to work on, and the answer is almost always some version of "build models," "do machine learning," "work with AI." Ask them how they feel about spending 70-80% of a real project cleaning and preparing data before any of that happens, and you get a very different reaction.
This gap between the fantasy and the actual job is one of the most consistent things I see training students moving into data roles.
The part nobody puts in the course trailer
Real datasets are messy in ways that are hard to appreciate until you've dealt with them. Missing values that aren't marked as missing — they're just blank, or zero, or "N/A" typed as text instead of an actual null. Dates stored in four different formats across the same column. Duplicate records that aren't exact duplicates, just close enough to cause problems. Categorical values with inconsistent spelling ("Chennai," "chennai," "CHENNAI," "Chenai") that all mean the same thing but won't group together without cleaning.
None of this is intellectually thrilling. All of it determines whether anything built afterward is trustworthy.
Why this is the actual skill, not a chore before the skill
A sophisticated model trained on messy, unvalidated data produces confident, wrong answers — which is worse than an obviously broken system, because people trust the output. The judgment involved in data cleaning — deciding how to handle a missing value, whether an outlier is a data entry error or a genuine signal, whether two records really represent the same entity — requires as much domain understanding and critical thinking as choosing between algorithms. It's just less visible, and it doesn't photograph well for a portfolio.
What this looks like in a real project
A typical week on a real data project might involve: pulling data from three different sources that don't use the same customer ID format, figuring out why 12% of a column is unexpectedly null, deciding whether to drop or impute those values and being able to justify the choice, and then — only after all of that — actually building and evaluating a model. The modeling step, ironically, is often the fastest part.
Students who've only practiced on pre-cleaned datasets (the kind that come bundled with a tutorial, already tidy and ready for .fit()) are often caught off guard by how much of a real job doesn't look like that at all.
What's worth practicing early
Deliberately work with a messy, real-world dataset instead of a pre-cleaned tutorial one — government open data portals are a good source of genuinely messy data
Practice explaining why you handled a missing value a certain way, not just that you handled it
Get comfortable with the tedious parts of pandas (or your tool of choice) — merging, reshaping, deduplicating — since this is where a huge amount of real time goes
Resist the urge to jump to modeling before you've actually understood what's wrong with your data first
Cleaning data isn't the boring prerequisite to the real work. For most practitioners, most of the time, it is the real work — and treating it that way early is what makes the modeling part, when it finally happens, actually trustworthy.
I teach Data Science and Python training in Chennai at RedYellow Technologies, and this expectation gap is one of the first things I address with new students. Curious how much of this matches what others have experienced moving from coursework into real data work.