It’s 2 AM. What is Your Data Doing?
You probably didn’t notice it. No alarms went off. No one paged you at 2 a.m. But somewhere in your organization, a column changed names, a data type shifted, or a DBA changed all the ZIPCode columns to integer (true story). That’s schema drift—and it’s quietly changing everything your users think they know. Data quality beware. Business users are going to lose confidence and trust in data.
Schema drift is what happens when the structure of your data changes without anyone telling the people (or systems) downstream. It might not crash your dashboards. It just makes them wrong.
“Where there’s data smoke, there’s business fire” – Tom Redmon
When your schema drifts on a whim, your data stops being as useful. It becomes misleading at best—and dangerous at worst.
The fix to this data quality issue isn’t about tools (yes, you should be using schema validation, data model-driven design, versioning, and alerts). The real solution is cultural. How your data is constructed is a contract. If your producers and users aren’t aligned on what’s being sent and what’s expected, you’re not doing data engineering—you’re playing telephone.
Real World Story (Names Have Been Changed)
On one of my projects, an experienced DBA named Clippy decided to refactor the customer address table. Clippy hated all character datatypes because he had read somewhere on the bloggerverse that numeric ones were so much faster. When he saw the ZIPCode column defined as CHAR(5), he decided to make ONE SMALL CHANGE, “That’s five bytes per row. Multiply that by 10 million customers…” It turns out, Clippy wasn’t as smart as he should have been. Have you worked with Clippy? I think we all have. Tell me your stories.
And, yes, he did that change overnight.
So what can you do?
Start small. Pick one critical pipeline. Compare the schema your consumers expect to what’s actually arriving. If they match, great. If not, congratulations—you’ve just found your first schema drift. Now fix it, document it, and make sure it doesn’t happen again without someone noticing.
Figure out why your data governance processes failed on these changes and fix that. If it’s a staff issue, recommend direction from superiors.
Your best bet, is to implement automated tools that will compare your data models to your development, test, and production environments. If you don’t have data models (well, that’s part of your problem), compare to a release version of you database. Configure them to identify drift, keep metrics, and notify you when they detect changes.
Calls to Action
First, tell me your favorite drift story.
Audit one of your key data flows or tables today. If you find a mismatch, don’t just patch it—start the conversation about data governance. Your future self (and your analysts) will thank you.
Comments are closed