Which came first, the Data Quality Tool or the Business Need?

This recent tweet by Andy Bitterer of Gartner Research (and ANALYSTerical) sparked an interesting online discussion, which was vaguely reminiscent of the classic causality dilemma that is commonly stated as “which came first, the chicken or the egg?”

An E-mail from the Edge

On the same day I saw Andy’s tweet, I received an e-mail from a friend and fellow data quality consultant, who had just finished a master data management (MDM) and enterprise data warehouse (EDW) project, which had over 20 customer data sources.

Although he was brought onto the project specifically for data cleansing, he was told from the day of his arrival that because of time constraints, they decided against performing any data cleansing with their recently purchased data quality tool.  Instead, they decided to use their data integration tool to simply perform the massive initial load into their new MDM hub and EDW.

But wait—the story gets even better.  The very first decision this client made was to purchase a consolidated enterprise application development platform with seamlessly integrated components for data quality, data integration, and master data management.

So long before this client had determined their business need, they decided that they needed to build a new MDM hub and EDW, made a huge investment in an entire platform of technology, then decided to use only the basic data integration functionality. 

However, this client was planning to use the real-time data quality and MDM services provided by their very powerful enterprise application development platform to prevent duplicates and any other bad data from entering the system after the initial load. 

But, of course, no one on the project team was actually working on configuring any of those services, or even, for that matter, determining the business rules those services would enforce.  Maybe the salesperson told them it was as easy as flipping a switch?

My friend (especially after looking at the data), preached data quality was a critical business need, but he couldn’t convince them, even despite taking the initiative to present the results of some quick data profiling, standardization, and data matching used to identify duplicate records within and across their primary data sources, which clearly demonstrated the level of poor data quality.

Although this client agreed that they definitely had some serious data issues, they still decided against doing any data cleansing and wanted to just get the data loaded.  Maybe they thought they were loading the data into one of those self-healing databases?

The punchline—this client is a financial services institution with a business need to better identify their most valuable customers.

As my friend lamented at the end of his e-mail, why do clients often later ask why these types of projects fail?

Blind Vendor Allegiance

In his recent blog post Blind Vendor Allegiance Trumps Utility, Evan Levy examined this bizarrely common phenomenon of selecting a technology vendor without gathering requirements, reviewing product features, and then determining what tool(s) could best help build solutions for specific business problems—another example of the tool coming before the business need.

Evan was recounting his experiences at a major industry conference on MDM, where people were asking his advice on what MDM vendor to choose, despite admitting “we know we need MDM, but our company hasn’t really decided what MDM is.”

Furthermore, these prospective clients had decided to default their purchasing decision to the technology vendor they already do business with, in other words, “since we’re already a [you can just randomly insert the name of a large technology vendor here] shop, we just thought we’d buy their product—so what do you think of their product?”

“I find this type of question interesting and puzzling,” wrote Evan.  “Why would anyone blindly purchase a product because of the vendor, rather than focusing on needs, priorities, and cost metrics?  Unless a decision has absolutely no risk or cost, I’m not clear how identifying a vendor before identifying the requirements could possibly have a successful outcome.”

SaaS-y Data Quality on a Cloudy Business Day?

Emerging industry trends like open source, cloud computing, and software as a service (SaaS) are often touted as less expensive than traditional technology, and I have heard some use this angle to justify buying the tool before identifying the business need.

In his recent blog post Cloud Application versus On Premise, Myths and Realities, Michael Fauscette examined the return on investment (ROI) versus total cost of ownership (TCO) argument quite prevalent in the SaaS versus on premise software debate.

“Buying and implementing software to generate some necessary business value is a business decision, not a technology decision,” Michael concluded.  “The type of technology needed to meet the business requirements comes after defining the business needs.  Each delivery model has advantages and disadvantages financially, technically, and in the context of your business.”

So which came first, the Data Quality Tool or the Business Need?

This question is, of course, absurd because, in every rational theory, the business need should always come first.  However, in predictably irrational real-world practice, it remains a classic causality dilemma for data quality related enterprise information initiatives such as data integration, master data management, data warehousing, business intelligence, and data governance.

But sometimes the data quality tool was purchased for an earlier project, and despite what some vendor salespeople may tell you, you don’t always need to buy new technology at the beginning of every new enterprise information initiative. 

Whenever, and before defining your business need, you already have the technology in-house (or you have previously decided, often due to financial constraints, that you will need to build a bespoke solution), you still need to avoid technology bias.

Knowing how the technology works can sometimes cause a framing effect where your business need is defined in terms of the technology’s specific functionality, thereby framing the objective as a technical problem instead of a business problem.

Bottom line—your business problem should always be well-defined before any potential technology solution is evaluated.

Related Posts

There are no Magic Beans for Data Quality

Do you believe in Magic (Quadrants)?

Is your data complete and accurate, but useless to your business?

Can Enterprise-Class Solutions Ever Deliver ROI?

Selling the Business Benefits of Data Quality

The Circle of Quality

Hell is other people’s data

I just read the excellent blog post Data Migration – and existentialist angst by John Morris, which asks the provocative question what can the philosophy of Jean-Paul Sartre tell us about data migration?

As a blogger almost as obsessive-compulsive about literature and philosophy as I am about data, this post resonated with me.  But perhaps Neil Raden is right when he remarked on Twitter that “anyone who works in Jean-Paul Sartre with data migration should get to spend 90 days with Lindsay Lohan.  Curse of liberal arts education.” (Please Note: Lindsay’s in jail for 90 days).

Part of my liberal arts education (and for awhile I was a literature major with a minor in philosophy) included reading Sartre, not only his existentialist philosophy, but also his literature, including the play No Exit, which is the source of perhaps his most famous quote: “l’enfer, c’est les autres” (“Hell is other people”) that I have paraphrased into the title of this blog post.

Being and Nothingness

John Morris used Jean-Paul Sartre’s classic existentialist essay Being and Nothingness, and more specifically, two of its concepts, namely that objects are “en-soi” (“things in themselves”) and people are “pour-soi” (“things for themselves”), to examine the complex relationship that is formed during data analysis between the data (an object) and its analyst (a person).

During data analysis, the analyst is attempting to discover the meaning of data, which is determined by discovering its essential business use.  However, in the vast majority of cases, data has multiple business uses.

This is why, as Morris explains, first of all, we should beware “the naive simplicity of assuming that understanding meaning is easy, that there is one right definition.  The relationship between objects and their essential meanings is far more problematic.”

Therefore, you need not worry, for as Morris points out, “it’s not because you are no good at your job and should seek another trade that you can’t resolve the contradictions.  It’s a problem that has confused some of the greatest minds in history.”

“Secondly,” as Morris continues, we have to acknowledge that “we have the technology we have.  By and large, it limits itself to a single meaning, a single Canonical Model.  What we have to do is get from the messy first problem to the simpler compromise of the second view.  There’s no point hiding away from this as an essential part of our activity.”

The complexity of the external world

“Machines are en-soi objects that create en-soi objects,” Morris explains, whereas “people are pour-soi consciousnesses that create meanings and instantiate them in the records they leave behind in the legacy data stores we then have to re-interpret.”

“We then waste time using the wrong tools (e.g., trying to impose an enterprise view onto our business domain experts which is inconsistent with their divergent understandings) only to be surprised and frustrated when our definitions are rejected.”

As I have written about in previous posts, whether it’s an abstract description of real-world entities (i.e., “master data”) or an abstract description of real-world interactions (i.e., “transaction data”) among entities, data is an abstract description of reality.

These abstract descriptions can never be perfected since there is always what I call a digital distance between data and reality.

The inconvenient truth is that reality is not the same thing as the beautifully maintained digital data worlds that exist within our enterprise systems (and, of course, creating and maintaining these abstract descriptions of reality is no easy task).

As Morris thoughtfully concludes, we must acknowledge that “this central problem of the complexity of the external world is against the necessary simplicity of our computer world.”

Hell is other people’s data

The inconvenient truth of the complexity of the external world plays a significant role within the existentialist philosophy of an organization’s data silos, which are also the bane of successful enterprise information management. 

Each and every business unit acts as a pour-soi (a thing for themselves), persisting on their reliance on their own data silos, thereby maintaining their own version of the truth—because they truly believe that hell is other people’s data.

El Festival del IDQ Bloggers (June and July 2010)

Welcome to the June and July 2010 issue of El Festival del IDQ Bloggers, which is a blog carnival by the IAIDQ that offers a great opportunity for both information quality and data quality bloggers to get their writing noticed and to connect with other bloggers around the world.

Definition Drift

Graham Rhind submitted his July blog post Definition drift, which examines the persistent problems facing attempts to define a consistent terminology within the data quality industry. 

It is essential to the success of a data quality initiative that its key concepts are clearly defined and in a language that everyone can understand.  Therefore, I also recommend that you check out the free online data quality glossary built and maintained by Graham Rhind by following this link: Data Quality Glossary.

Lemonade Stand Data Quality

Steve Sarsfield submitted his July blog post Lemonade Stand Data Quality, which explains that data quality projects are a form of capitalism, meaning that you need to sell your customers a refreshing glass and keep them coming back for more.

What’s In a Given Name?

Henrik Liliendahl Sørensen submitted his June blog post What’s In a Given Name?, which examines a common challenge facing data quality, master data management, and data matching—namely (pun intended), how to automate the interpretation of the “given name” (aka “first name”) component of a person’s name separately from their “family name” (aka “last name”).

Solvency II Standards for Data Quality

Ken O’Connor submitted his July blog post Solvency II Standards for Data Quality, which explains the Solvency II standards are common sense data quality standards, which can enable all organizations, regardless of their industry or region, to achieve complete, appropriate, and accurate data.

How Accuracy Has Changed

Scott Schumacher submitted his July blog post How Accuracy Has Changed, which explains that accuracy means being able to make the best use of all the information you have, putting data together where necessary, and keeping it apart where necessary.

Uniqueness is in the Eye of the Beholder

Marty Moseley submitted his June blog post Uniqueness is in the Eye of the Beholder, which beholds the challenge of uniqueness and identity matching, where determining if data records should be matched is often a matter of differing perspectives among groups within an organization, where what one group considers unique, another group considers non-unique or a duplicate.

Uniqueness in the Eye of the NSTIC

Jeffrey Huth submitted his July blog post Uniqueness in the Eye of the NSTIC, which examines a recently drafted document in the United States regarding a National Strategy for Trusted Identities in Cyberspace (NSTIC).

Profound Profiling

Daragh O Brien submitted his July blog post Profound Profiling, which recounts how he has found data profiling cropping up in conversations and presentations he’s been making recently, even where the topic of the day wasn’t “Information Quality” and shares his thoughts on the profound benefits of data profiling for organizations seeking to manage risk and ensure compliance.

Wanted: a Data Quality Standard for Open Government Data

Sarah Burnett submitted her July blog post Wanted: a Data Quality Standard for Open Government Data, which calls for the establishment of data quality standards for open government data (i.e., public data sets) since more of it is becoming available.

Data Quality Disasters in the Social Media Age

Dylan Jones submitted his July blog post The reality of data quality disasters in a social media age, which examines how bad news sparked by poor data quality travels faster and further than ever before, by using the recent story about the Enbridge Gas billing blunders as a practical lesson for all companies sitting on the data quality fence.

Finding Data Quality

Jim Harris (that’s me referring to myself in the third person) submitted my July blog post Finding Data Quality, which explains (with the help of the movie Finding Nemo) that although data quality is often discussed only in its relation to initiatives such as master data management, business intelligence, and data governance, eventually you’ll be finding data quality everywhere.

Editor’s Selections

In addition to the official submissions above, I selected the following great data quality blog posts published in June or July 2010:

• New species in the Data Zoo – Not Bovvered
• Good King Censusless
• How to Measure Data Accuracy?
• The Holy Grail Of Data Quality – Linking Data Quality To Business Impact
• The Triple Threat to Good Decisions: Data, Time and Emotion
• Continuous Partial Attention
• Warning! Information consumers are also information producers by default
• Are You Creating Duplicate Records?
• When it comes to data quality, it’s the last mile that counts
• Master Data and Reference Data
• “WOW! I didn't realize our data was that bad”
• Data Governance Remains Immature
• The Importance of Scope in Data Quality Efforts
• Data Governance: The People Make It Real
• Predictably Poor MetaData Quality
• What To Do About Your Stale Customer Data
• Data Quality Control vs Data Quality Assurance
• Falsehoods Programmers Believe About Names
• The role of a Business Analyst in the Management of Information
• Does Enterprise Data Quality need to push the boundary more? 

Check out the past issues of El Festival del IDQ Bloggers

El Festival del IDQ Bloggers (May 2010) – edited by Castlebridge Associates

El Festival del IDQ Bloggers (April 2010) – edited by Graham Rhind

El Festival del IDQ Bloggers (March 2010) – edited by Phil Wright

El Festival del IDQ Bloggers (February 2010) – edited by William Sharp

El Festival del IDQ Bloggers (January 2010) – edited by Henrik Liliendahl Sørensen

El Festival del IDQ Bloggers (November 2009) – edited by Daragh O Brien

El Festival del IDQ Bloggers (October 2009) – edited by Vincent McBurney

El Festival del IDQ Bloggers (September 2009) – edited by Daniel Gent

El Festival del IDQ Bloggers (August 2009) – edited by William Sharp

El Festival del IDQ Bloggers (July 2009) – edited by Andrew Brooks

El Festival del IDQ Bloggers (June 2009) – edited by Steve Sarsfield

El Festival del IDQ Bloggers (May 2009) – edited by Daragh O Brien

El Festival del IDQ Bloggers (April 2009) – edited by Jim Harris

Is your data complete and accurate, but useless to your business?

Ensuring that complete and accurate data is being used to make critical daily business decisions is perhaps the primary reason why data quality is so vitally important to the success of your organization. 

However, this effort can sometimes take on a life of its own, where achieving complete and accurate data is allowed to become the raison d'être of your data management strategy—in other words, you start managing data for the sake of managing data.

When this phantom menace clouds your judgment, your data might be complete and accurate—but useless to your business.

Completeness and Accuracy

How much data is necessary to make an effective business decision?  Having complete (i.e., all available) data seems obviously preferable to incomplete data.  However, with data volumes always burgeoning, the unavoidable fact is that sometimes having more data only adds confusion instead of clarity, thereby becoming a distraction instead of helping you make a better decision.

Returning to my original question, how much data is really necessary to make an effective business decision? 

Accuracy, which, thanks to substantial assistance from my readers, was defined in a previous post as both the correctness of a data value within a limited context such as verification by an authoritative reference (i.e., validity) combined with the correctness of a valid data value within an extensive context including other data as well as business processes (i.e., accuracy). 

Although accurate data is obviously preferable to inaccurate data, less than perfect data quality can not be used as an excuse to delay making a critical business decision.  When it comes to the quality of the data being used to make these business decisions, you can’t always get the data you want, but if you try sometimes, you just might find, you get the business insight you need.

Data-driven Solutions for Business Problems

Obviously, there are even more dimensions of data quality beyond completeness and accuracy. 

However, although it’s about more than just improving your data, data quality can be misperceived to be an activity performed just for the sake of the data.  When, in fact, data quality is an enterprise-wide initiative performed for the sake of implementing data-driven solutions for business problems, enabling better business decisions, and delivering optimal business performance.

In order to accomplish these objectives, data has to be not only complete and accurate, as well as whatever other dimensions you wish to add to your complete and accurate definition of data quality, but most important, data has to be useful to the business.

Perhaps the most common definition for data quality is “fitness for the purpose of use.” 

The missing word, which makes this definition both incomplete and inaccurate, puns intended, is “business.”  In other words, data quality is “fitness for the purpose of business use.”  How complete and how accurate (and however else) the data needs to be is determined by its business use—or uses since, in the vast majority of cases, data has multiple business uses.

Data, data everywhere

With silos replicating data as well as new data being created daily, managing all of the data is not only becoming impractical, but because we are too busy with the activity of trying to manage all of it, no one is stopping to evaluate usage or business relevance.

The fifth of the Five New Ideas From 2010 MIT Information Quality Industry Symposium, which is a recent blog post written by Mark Goloboy, was that “60-90% of operational data is valueless.”

“I won’t say worthless,” Goloboy clarified, “since there is some operational necessity to the transactional systems that created it, but valueless from an analytic perspective.  Data only has value, and is only worth passing through to the Data Warehouse if it can be directly used for analysis and reporting.  No news on that front, but it’s been more of the focus since the proliferation of data has started an increasing trend in storage spend.”

In his recent blog post Are You Afraid to Say Goodbye to Your Data?, Dylan Jones discussed the critical importance of designing an archive strategy for data, as opposed to the default position many organizations take, where burgeoning data volumes are allowed to proliferate because, in large part, no one wants to delete (or, at the very least, archive) any of the existing data. 

This often results in the data that the organization truly needs for continued success getting stuck in the long line of data waiting to be managed, and in many cases, behind data for which the organization no longer has any business use (and perhaps never even had the chance to use when the data was actually needed to make critical business decisions).

“When identifying data in scope for a migration,” Dylan advised, “I typically start from the premise that ALL data is out of scope unless someone can justify its existence.  This forces the emphasis back on the business to justify their use of the data.”

Data Memorioso

Funes el memorioso is a short story by Jorge Luis Borges, which describes a young man named Ireneo Funes who, as a result of a horseback riding accident, has lost his ability to forget.  Although Funes has a tremendous memory, he is so lost in the details of everything he knows that he is unable to convert the information into knowledge and unable, as a result, to grow in wisdom.

In Spanish, the word memorioso means “having a vast memory.”  When Data Memorioso is your data management strategy, your organization becomes so lost in all of the data it manages that it is unable to convert data into business insight and unable, as a result, to survive and thrive in today’s highly competitive and rapidly evolving marketplace.

In their great book Made to Stick: Why Some Ideas Survive and Others Die, Chip Heath and Dan Heath explained that “an accurate but useless idea is still useless.  If a message can’t be used to make predictions or decisions, it is without value, no matter how accurate or comprehensive it is.”  I believe that this is also true for your data and your organization’s business uses for it.

Is your data complete and accurate, but useless to your business?

The 2010 Data Quality Blogging All-Stars

The 2010 Major League Baseball (MLB) All-Star Game is being held tonight (July 13) at Angel Stadium in Anaheim, California.

For those readers who are not baseball fans, the All-Star Game is an annual exhibition held in mid-July that showcases the players with (for the most part) the best statistical performances during the first half of the MLB season.

Last summer, I began my own annual exhibition of showcasing the bloggers whose posts I have personally most enjoyed reading during the first half of the data quality blogging season. 

Therefore, this post provides links to stellar data quality blog posts that were published between January 1 and June 30 of 2010.  My definition of a “data quality blog post” also includes Data Governance, Master Data Management, and Business Intelligence. 

Please Note: There is no implied ranking in the order that bloggers or blogs are listed, other than that Individual Blog All-Stars are listed first, followed by Vendor Blog All-Stars, and the blog posts are listed in reverse chronological order by publication date.

Henrik Liliendahl Sørensen

From Liliendahl on Data Quality:

• What’s In a Given Name?
• Returns from Investing in a Data Quality Tool
• The Next Level
• Big Time ROI in Identity Resolution
• Data Quality from the Cloud
• What is a best-in-class match engine?
• What is Data Quality anyway?
• Bad word?: Data Owner
• The Myth about a Myth
• Create Table Homo_Sapiens

Dylan Jones

From Data Quality Pro:

• Rise of the Data Quality Practitioner 2.0
• Data Quality Project Selection - A Missing Skill?
• How Are You Creating a Pull for Data Quality in Your Organisation?
• Does Your Project Suffer From Data Quality Product Myopia?
• Are You "Cookie-Cutting" Your Data Quality Strategy?
• WANTED: Data Quality Change Agents 

Julian Schwarzenbach

From Data and Process Advantage Blog:

• Radioactive gold data
• Does data make you lonely ?!?
• The Data Accident Investigation Board 
• The Data Zoo (Five Part Series and White Paper): Part 1, Part 2, Part 3, Part 4, Part 5
• How tasty is your data quality cheese? 

Rich Murnane

From Rich Murnane's Blog:

• The Data Management "A-Team"
• Data Management: Marketing Your Data Governance Efforts
• How to identify your "Data Quality Savages"
• Something happened on the way to better data quality...
• Data Quality Scorecard by client - and beyond...
• Data Governance - We do in fact sweat the small stuff...
• Data Quality issue in my new database - or so we thought...

Phil Wright

From Data Factotum:

• How are you Executing your Data Quality Strategy?
• How do you identify your strategic data?
• The First Step on your Data Quality Roadmap
• Can motivations impact the state of data quality?
• A balanced approach to scoring data quality (Six Part Series): Part 1, Part 2, Part 3, Part 4, Part 5, Part 6
• The Great Expectations of BI
• Questions to measure BI DQ/DM success 

Initiate – an IBM Company

From Mastering Data Management:

• Uniqueness is in the Eye of the Beholder
• Surfing the Waves of MDM 
• A New Dimension in Data Governance Directives: Politics
• The Relationship Between MDM and Data Governance
• Entity Resolution Taken One Step Further: “Spot The Difference”
• My Data Governance Hero: A True Story
• Nerd Appeal or Boardroom Fare?
• Series: Building an MDM Roadmap
• The Peanut Butter and Chocolate of Data: MDM and SOA
• Hear Ye! Hear Ye! Declarations for Data Governance

Baseline Consulting

From their three blogs: Inside the Biz with Jill Dyché, Inside IT with Evan Levy, and In the Field with our Experts:

• The Importance of Scope in Data Quality Efforts
• A Data Governance Primer (Three Part Series): Part 1, Part 2, Part 3
• Blind Vendor Allegiance Trumps Utility
• Data Profiling with SQL is Hazardous to Your Company’s Health
• The Tyranny of Consensus
• MDM Can Challenge Traditional Development Paradigms
• Charm School: It’s Not Just for IT Anymore
• The Good News about MDM Market Consolidation
• The “And” Syndrome: How to Keep BI Development Proactive
• The Quality Gap: Why Being On-Time Isn’t Enough
• MDM and All That Jazz
• Could Data Governance Help the War on Terror? 

DataFlux – a SAS Company

From Community of Experts:

• Data profiling: Are you closing the loop?
• The Myth of the “Golden Record”
• Questions for your next data quality business case
• Identity Resolution, Cleansing, and Survivorship
• Why are there Duplicate Records?
• Data Quality Non-Believers
• Data Quality, Data Requirements, Data Use
• Thoughts on Identifying Attributes
• Warning: Data Quality Improvement Can Seriously Damage Your Business
• Data-driven customer churn
• Data Streams and Data Rules
• Distant Early Warnings 

Related Posts

Recently Read: May 15, 2010

Recently Read: March 22, 2010

Recently Read: March 6, 2010

Recently Read: January 23, 2010

The 2009 Data Quality Blogging All-Stars

Additional Resources

From the IAIDQ, read the 2010 issues of the Blog Carnival for Information/Data Quality:

• El Festival del IDQ Bloggers (January 2010)
• El Festival del IDQ Bloggers (February 2010)
• El Festival del IDQ Bloggers (March 2010)
• El Festival del IDQ Bloggers (April 2010)
• El Festival del IDQ Bloggers (May 2010) 

The Diffusion of Data Governance

Marty Moseley of Initiate recently blogged Are We There Yet? Results of the Data Governance Survey, and the blog post includes a link to the survey, which is freely available—no registration required.

The Initiate survey says that although data governance dates back to the late 1980s, it is experiencing a resurgence because of initiatives such as business intelligence, data quality, and master data management—as well as the universal need to make better data-driven business decisions “in less time than ever before, often culling data from more structured and unstructured sources, with more transparency required.”

Winston Chen of Kalido recently blogged A Brief History of Data Governance, which provides a brief overview of three distinct eras in data management: Application Era (1960-1990), Enterprise Repository Era (1990-2010), and Policy Era (2010-?).

As I commented on Winston’s post, I began my career at the tail-end of the Application Era, and my career has been about a 50/50 split between applications and enterprise repositories since history does not move forward at the same pace for all organizations, including software vendors—by which, I mean that my professional experience was influenced more by working for vendors selling application-based solutions than it was by working with clients who were, let’s just say, less than progressive.

Diffusion of innovations (illustrated above) is a theory developed by Everett Rogers for describing the five stages and the rate at which innovations (e.g., new ideas or technology) spread through markets (or “cultures”), starting with the Innovators and the Early Adopters, then progressing through the Early Majority and the Late Majority, and finally ending with the Laggards.

Therefore, the exact starting points of the three eras Winston described in his post can easily be debated because progress can be painfully slow until a significant percentage of the Early Majority begins to embrace the innovation—thereby causing the so-called Tipping Point where progress begins to accelerate enough for the mainstream to take it seriously. 

Please Note: I am not talking about crossing “The Chasm”—which as Geoffrey A. Moore rightfully discusses, is the critical, but much earlier, phenomenon occurring when enough of the Early Adopters have embraced the innovation so that the beginning of the Early Majority becomes an almost certainty—but true mainstream adoption of the innovation is still far from guaranteed.

The tipping point that I am describing occurs within the Early Majority and before the top of the adoption curve is reached. 

Achieving 16% market share (or “cultural awareness”) is where the Early Majority begins—and only after successfully crossing the chasm (which I approximate occurs somewhere around 8% market share).  However,  the difference between a fad and a true innovation occurs somewhere around 25% market share—and this is the tipping point that I am describing.

The Late Majority (and the top of the adoption curve) doesn’t begin until 50% market share, and it’s all downhill from there, meaning that the necessary momentum has been achieved to almost guarantee that the innovation will be fully adopted.

For example, it could be argued that master data management (MDM) reached its tipping point in late 2009, and with the wave of acquisitions in early 2010, MDM stepped firmly on the gas pedal of the Early Majority, and we are perhaps just beginning to see the start of MDM’s Late Majority.

It is much harder to estimate where we are within the diffusion of data governance.  Of course, corporate cultural awareness always plays a significant role in determining the adoption of new ideas and the market share of emerging technologies.

The Initiate survey concludes that “the state of data governance initiatives is still rather immature in most organizations” and reveals “a surprising lack of perceived executive interest in data governance initiatives.”

Rob Karel of Forrester Research recently blogged about how Data Governance Remains Immature, but he is “optimistic that we might finally see some real momentum building for data governance to be embraced as a legitimate competency.”

“It will likely be a number of years before best practices outnumber worst practices,” as Rob concludes, “but any momentum in data governance adoption is good momentum!”

From my perspective, data governance is still in the Early Adopter phase.  Perhaps 2011 will be “The Year of Data Governance” in much the same way that some have declared 2010 to to be “The Year of MDM.”

In other words, it may be another six to twelve months before we can claim the Early Majority has truly embraced not just the idea of data governance, but have realistically begun their journey toward making it happen.

What Say You?

Please share your thoughts about the diffusion of data governance, as well as your overall perspectives on data governance.

Related Posts

MacGyver: Data Governance and Duct Tape

The Prince of Data Governance

Jack Bauer and Enforcing Data Governance Policies

Follow OCDQ

If you enjoyed this blog post, then please subscribe to OCDQ via my RSS feed, my E-mail updates, or Google Reader.

You can also follow OCDQ on Twitter, fan the Facebook page for OCDQ, and connect with me on LinkedIn.

The Great Rift

I recently read a great article about social collaboration in the enterprise by Julie Hunt, which includes the excellent insight:

“Most enterprises have failed to engender a ‘collaboration culture’ based on real human interaction.  The executive management of many companies does not even understand what a ‘collaboration culture’ is.  Frankly, executive management of many companies is hard put to authentically value employees—these companies want to de-humanize employees with such terms as ‘resources’ and ‘human capital’, and think that it is enough if they sling around a few ‘mission statements’ claiming that they ‘value’ employees.”

Even though the article was specifically discussing the reason why companies struggle to effectively use social media in business, it reminded me of the reason that many enterprise initiatives struggle—if not fail—to live up to their rather lofty expectations.

The most common root cause for the failure of enterprise initiatives is what I like to refer to as The Great Rift.

The Great Rift

In astronomy, the Great Rift—also known as the Dark Rift—is a series of overlapping and non-luminous molecular dust clouds, which appear to create a dark divide in the otherwise bright band of stars and other luminous objects comprising our Milky Way.

Within the intergalactic empires of the business world, The Great Rift is a metaphor for the dark divide separating how most of these organizations would list and prioritize their corporate assets:

Please note that a list of things is on the left side of The Great Rift and on the right side is a list of people. 

Although the order of importance given to the items within each of these lists is debatable, I would argue what is not debatable is that the list of things is what most organizations prioritize as their most important corporate assets.

It is precisely this prioritization of the value of things over the value of people that creates and sustains The Great Rift.

Of course, the message delivered by corporate mission statements, employee rallies, and customer conferences would lead you to believe the exact opposite is true—and in fairness, some organizations do prioritize the value of people over the value of things.

However, the harsh reality of the business world is that the message “we value our people” is often only a Machiavellian illusion.

I believe that as long as The Great Rift exists, then no enterprise initiative can be successful—or remain successful for very long. 

The enterprise-wide communication and collaboration that is so critical to achieving and sustaining success on initiatives such as Master Data Management (MDM) and Data Governance, can definitely not escape the effects of The Great Rift. 

Eventually, The Great Rift becomes the enterprise equivalent of a black hole, where not even the light shining from your very brightest stars will be able to escape its gravitational pull.

“Returning to the human side of business won’t happen magically,” Julie Hunt concluded her article.  “It will take real work and real commitment, from the executive level through all levels of management and employee departments.”

I wholeheartedly agree with Julie and will therefore conclude this blog post by paraphrasing the lyrics from “Yellow” by Coldplay into a song I am simply calling “People” because repairing The Great Rift and “returning to the human side of business” can only be accomplished by acknowledging that every organization’s truly most important corporate asset is—their people.

Rumors have it that the The Rolling Forecasts might even add the song to their playlist for the Data Rock Star World Tour 2010.

People

Look at your people
Look how they shine for you
And in everything they do
Yeah, they’re all stars

They came along 
They wrote a song for you
About all the Things they do
And it was called People

So they each took their turn 
And sung about all the things they’ve done
And it was all for you

Your business
Oh yeah, your technology and your data too
They turned it all into something beautiful
Did you know they did it for you?
They did it all for you

Now what are you going to do for them?

They crossed The Great Rift
They jumped across for you 
Because all the things you do
Are all done by your people

Look at your stars
Look how they shine
And in everything they do
Look how they shine for you

They crossed the line
The imaginary line drawn by you
Oh what a wonderful thing to do
And it was all for you

Your business
Oh yeah, your technology and your data too
They turned it all into something beautiful
Did you know they did it for you?
They did it all for you

Now what are you going to do for them?

Look at your people, they’re your stars, it’s true
Look how they shine
And in everything they do
Look how they shine for you

Look at your people
Look at your stars
Look how they shine
And in everything they do
Look how they shine for you

Now what are you going to do for them?

Channeling My Inner Beagle: The Case for Hyperactivity

Phil Simon, who is a Bulldog’s best friend and is a good friend of mine, recently blogged Channeling My Inner Bulldog: The Case for Stubbornness, in which he described how the distracting nature of multitasking can impair our ability to solve complex problems.

Although I understood every single word he wrote, after three dog nights, I can’t help but take the time to share my joy to the world by channeling my inner beagle and making the case for hyperactivity—in other words, our need to simply become better multitaskers.

The beloved mascot of my blog post is Bailey, not only a great example of a typical Beagle, but also my brother’s family dog, who is striking a heroic pose in this picture while proudly sporting his all-time favorite Halloween costume—Underdog.

I could think of no better hero to champion my underdog of a cause:

“There’s no need to fear . . . hyperactivity!”

Please Note: Just because Phil Simon coincidentally uses “Simon Says” as the heading for all his blog conclusions, doesn’t mean Phil is Simon Bar Sinister, who coincidentally used “Simon Says” to explain his diabolical plans—that’s completely coincidental.

The Power of Less

I recently read The Power of Less, the remarkable book by Leo Babauta, which provides practical advice on simplifying both our professional and personal lives.  The book has a powerfully simple message—identify the essential, eliminate the rest.

I believe that the primary reason multitasking gets such a bad reputation is the numerous non-essential tasks typically included. 

Many daily tasks are simply “busy work” that we either don’t really need to do at all, or don’t need to do as frequently.  We have allowed ourselves to become conditioned to perform certain tasks, such as constantly checking our e-mail and voice mail. 

Additionally, whenever we do find a break in our otherwise hectic day, “nervous energy” often causes us to feel like we should be doing something with our time—and so the vicious cycle of busy work begins all over again.

“Doing nothing is better than being busy doing nothing,” explained Lao Tzu. 

I personally find that whenever I am feeling overwhelmed by multitasking, it’s not because I am trying to distribute my time among a series of essential tasks—instead, I was really just busy doing a whole lot of nothing.  “Doing a huge number of things,” explains Babauta, “doesn’t mean you’re getting anything meaningful done.”

Meaningful accomplishment requires limiting our focus to only essential tasks.  Unlimited focus, according to Babauta, is like “taking a cup of red dye and pouring it into the ocean, and watching the color dilute into nothingness.  Limited focus is putting that same cup of dye into a gallon of water.”

Only you can decide which tasks are essential.  Look at your “to do list” and first identify the essential—then eliminate the rest.

It’s about the journey—not the destination

Once you have eliminated the non-essential tasks, your next challenge is limiting your focus to only the essential tasks. 

Perhaps the simplest way to limit your focus and avoid the temptation of multitasking altogether is to hyper-focus on only one task at a time.  So let’s use reading a non-fiction book as an example of one of the tasks you identified as essential.

Some people would read this non-fiction book as fast as they possibly can—hyper-focused and not at all distracted—as if they’re trying to win “the reading marathon” by finishing the book in the shortest time possible. 

They claim that this gives them both a sense of accomplishment and allows them to move on to their next essential task, thereby always maintaining their vigilant hyper-focus of performing only one task at a time. 

However, what did they actually accomplish other than simply completing the task of reading the book?

I find people—myself included—that voraciously read non-fiction books often struggle when attempting to explain the book, and in fact, they usually can’t tell you anything more than what you would get from simply reading the jacket cover of the book. 

Furthermore, they often can’t demonstrate any proof of having learned anything from reading the book.  Now, if they were reading fiction, I would argue that’s not a problem.  However, their “undistracted productivity” of reading a non-fiction book can easily amount to nothing more than productive entertainment. 

They didn’t mind the gap between the acquisition of new information and its timely and practical application.  Therefore, they didn’t develop valuable knowledge.  They didn’t move forward on their personal journey toward wisdom. 

All they did was productively move the hands of the clock forward—all they did was pass the time.

Although by eliminating distractions and focusing on only essential tasks, you’ll get more done and reach your destination faster, in my humble opinion, a meaningful life is not a marathon—a meaningful life is a race not to run.

It’s about the journey—not the destination.  In the words of Ralph Waldo Emerson:

“With the past, I have nothing to do; nor with the future.  I live now.”

Hyperactivity is Simply Better Multitasking

Although I do definitely believe in the power of less, the need to eliminate non-essential tasks, and the need to focus my attention, I am far more productive when hyper-active (i.e., intermittently alternating my attention among multiple simultaneous tasks).

Hyperactively collecting small pieces of meaningful information from multiple sources, as well as from the scattered scraps of knowledge whirling around inside my head, is more challenging, and more stressful, than focusing on only one task at a time.

However, at the end of most days, I find that I have made far more meaningful progress on my essential tasks. 

Although, in all fairness, I often breakdown and organize essential tasks into smaller sub-tasks, group similar sub-tasks together, then I multitask within only one group at a time.  This lower-level multitasking minimizes what I call the plate spinning effect, where an interruption can easily cause a disastrous disruption in productivity.

Additionally, I believe that not all distractions are created equal.  Some, in fact, can be quite serendipitous.  Therefore, I usually allow myself to include one “creative distraction” in my work routine.  (Typically, I use either Twitter or some source of music.)

By eliminating non-essential tasks, grouping together related sub-tasks, and truly embracing the chaos of creative distraction, hyperactivity is simply better multitasking—and I think that in the Digital Age, this is a required skill we all must master.

The Rumble in the Dog Park

So which is better?  Stubbornness or Hyperactivity?  In the so-called Rumble in the Dog Park, who wins?  Bulldogs or Beagles? 

I know that I am a Beagle.  Phil knows he is a Bulldog.  I would be unhappy as a Bulldog.  Phil would be unhappy as a Beagle. 

And that is the most important point.

There is absolutely no better way to make yourself unhappy than by trying to live by someone else’s definition of happiness.

You should be whatever kind of dog that truly makes you happy.  In other words, if you prefer single-tasking, then be a Bulldog, and if you prefer multitasking, then be a Beagle—and obviously, Bulldogs and Beagles are not the only doggone choices.

Maybe you’re one of those people who prefers cats—that’s cool too—just be whatever kind of cool cat truly makes you happy. 

Or maybe you’re neither a dog person nor a cat person.  Maybe you’re more of a Red-Eared Slider kind of person—that’s cool too.

And who ever said that you had to choose to be only one kind of person anyway? 

Maybe some days you’re a Beagle, other days you’re a Bulldog, and on weekends and vacation days you’re a Red-Eared Slider. 

It’s all good. 

Just remember—no matter what—always be you.

The Prince of Data Governance

The difference between politics and policies was explained in the recent blog post A New Dimension in Data Governance Directives: Politics by Jarrett Goldfedder, who also discussed the need to consider the political influences involved, as they can often have a far greater impact on our data governance policies than many choose to recognize.

I definitely agree, especially since the unique corporate culture of every organization carries with it the intricacies and complexities of politics that Niccolò Machiavelli (pictured) wrote about in his book The Prince.

The book, even despite the fact it was written in the early 16th century, remains a great, albeit generally regarded as satirical, view on politics.

The Prince provides a classic study of the acquisition, expansion, and effective use of political power, where the ends always justify the means.

An example of a Machiavellian aspect of the politics of data governance is when a primary stakeholder, while always maintaining the illusion of compliance, only truly complies with policies when it suits the very purposes of their own personal agenda, or when it benefits the interests of the business unit that they represent on the data governance board.

Creating Accountability

In her excellent comment on my recent blog post Jack Bauer and Enforcing Data Governance Policies, Kelle O'Neal provided a link to the great article Creating Accountability by Nancy Raulston, which explains that there is a significant difference between increasing accountability (e.g., for compliance with data governance policies) and simply getting everyone to do what they’re told (especially if you have considered resorting to the use of a Jack Bauer approach to enforcing data governance policies).

Raulston shares her high-level thoughts about the key aspects of alignment with vision and goals, achieving clarity on actions and priorities, establishing ownership of processes and responsibilities, the structure of meetings, and the critical role of active and direct communication—all of which are necessary to create true accountability.

“Accountability does not come from every single person getting every single action item done on time,” explains Raulston.  “It arises as groups actively manage the process of making progress, raising and resolving issues, actively negotiating commitments, and providing direct feedback to team members whose behavior is impeding the team.”

Obviously, this is often easier said than done.  However, as Raulston concludes, “ultimate success comes from each person being willing to honestly engage in the process, believing that the improved probability of success outweighs any momentary discomfort from occasionally having to admit to not having gotten something done.”  Or perhaps more important, occasionally having to be comfortable with not having gotten what would suit their personal agenda, or benefit the interests of their group.

The Art of the Possible

“Right now, our only choice,” as Goldfedder concluded his post, “is to hope that the leaders in charge of the final decisions can put their own political goals aside for the sake of the principles and policies they have been entrusted to uphold and protect.”

Although I agree, as well as also acknowledge that the politics of data governance will always make it as much art as it is science, I can not help but be reminded of the famous words of Otto von Bismarck:

“Politics is the art of the possible.”

The politics of data governance are extremely challenging, and yes, at times rather Machiavellian in their nature. 

Although it is certainly by no means an easy endeavor for either you or your organization to undertake, neither is achieving a successful and sustainable data governance program impossible. 

Politics may be The Prince of Data Governance, but as long as Communication and Collaboration reign as King and Queen, then Data Governance is the Art of the Possible.

Please share your thoughts about the politics of data governance, as well as your overall perspectives on data governance.

Follow OCDQ

If you enjoyed this blog post, then please subscribe to OCDQ via my RSS feed, my E-mail updates, or Google Reader.

You can also follow OCDQ on Twitter, fan the Facebook page for OCDQ, and connect with me on LinkedIn.

Jack Bauer and Enforcing Data Governance Policies

In my recent blog post Red Flag or Red Herring?, I explained that the primary focus of data governance is the strategic alignment of people throughout the organization through the definition, and enforcement, of policies in relation to data access, data sharing, data quality, and effective data usage, all for the purposes of supporting critical business decisions and enabling optimal business performance.

Simply establishing these internal data governance policies is often no easy task to accomplish.

However, without enforcement, data governance policies are powerless to affect the real changes necessary.

(Pictured: Jack Bauer enforcing a data governance policy.)

Jack Bauer and Data Governance

Jill Wanless commented that “sometimes organizations have the best of intentions.  They establish strategic alignment and governing policies (no small feat!) only to fail at the enforcement and compliance.  I believe some of this behavior is due to the fact that they may not know how to enforce effectively, without risking the very alignment they have established.  I would really like to see a follow up post on what effective enforcement looks like.”

As I began drafting this requested blog post, the first image that came to my mind for what effective enforcement looks like was Jack Bauer, the protagonist of the popular (but somewhat controversial) television series 24.

Well-known for his willingness to do whatever it takes, you can almost imagine Jack explaining to executive management:

“The difference between success and failure for your data governance program is the ability to enforce your policies.  But the business processes, technology, data, and people that I deal with, don’t care about your policies.  Every day I will regret looking into the eyes of men and women, knowing that at any moment, their jobs—or even their lives—may be deemed expendable, in order to protect the greater corporate good. 

I will regret every decision and mistake I have to make, which results in the loss of an innocent employee.  But you know what I will regret the most?  I will regret that data governance even needs people like me.”

Although definitely dramatic and somewhat cathartic, I don’t think it would be the right message for this blog post.  Sorry, Jack.

Enforcing Data Governance Policies

So if hiring Jack Bauer isn’t the answer, what is?  I recommend the following five steps for enforcing data governance policies, which I have summarized into the following simple list and explain in slightly more detail in the corresponding sections below:

1. Documentation – Use straightforward, natural language to document your policies in a way everyone can understand.
2. Communication – Effective communication requires that you encourage open discussion and debate of all viewpoints.
3. Metrics – Truly meaningful metrics can be effectively measured, and represent the business impact of data governance.
4. Remediation – Correcting any combination of business process, technology, data, and people—and sometimes, all four. 
5. Refinement – You must dynamically evolve and adapt your data governance policies—as well as their associated metrics.

Documentation

The first step in enforcing data governance policies is effectively documenting the defined policies.  As stated above, the definition process itself can be quite laborious.  However, before you can expect anyone to comply with the new policies, you first have to make sure that they can understand exactly what they mean. 

This requires documenting your polices using a straightforward and natural language.  I am not just talking about avoiding the use of techno-mumbo-jumbo.  Even business-speak can sound more like business-babbling—and not just to the technical folks.  Perhaps most important, avoid using acronyms and other lexicons of terminology—unless you can unambiguously define them.

For additional information on aspects related to documentation, please refer to these blog posts:

• The Acronymicon
• Podcast: Business Technology and Human-Speak
• The Third Law of Data Quality
• Eternal September and Tacit Knowledge
• Not So Strange Case of Dr. Technology and Mr. Business 

Communication

The second step is the effective communication of the defined and documented data governance policies.  Consider using a wiki in order to facilitate easy distribution, promote open discussion, and encourage feedback—as well as track all changes.

I always emphasize the importance of communication since it’s a crucial component of the collaboration that data governance truly requires in order to be successful. 

Your data governance policies reflect a shared business understanding.  The enforcement of these policies has as much to do with enterprise-wide collaboration as it does with supporting critical business decisions and enabling optimal business performance.

Never underestimate the potential negative impacts that the point of view paradox can have on communication.  For example, the perspectives of the business and technical stakeholders can often appear to be diametrically opposed. 

At the other end of the communication spectrum, you must also watch out for what Jill Dyché calls the tyranny of consensus, where the path of least resistance is taken, and justifiable objections either remain silent or are silenced by management. 

The tyranny of consensus is indeed the antithesis of the wisdom of crowds.  As James Surowiecki explains in his excellent book, the best collective decisions are the product of disagreement and contest, not consensus or compromise.

Data Governance lives on the two-way Street named Communication (which, of course, intersects with Collaboration Road).

For additional information on aspects related to communication, please refer to these blog posts:

• The Point of View Paradox
• Shut Your Mouth
• Podcast: Open Your Ears
• Hailing Frequencies Open
• The Game of Darts – An Allegory

Metrics

The third step in enforcing data governance policies is the creation of metrics with tangible business relevance.  These metrics must be capable of being effectively measured, and must also meaningfully represent the business impact of data governance.

The common challenge is that the easiest ones to create and monitor are low-level technical metrics, such as those provided by data profiling.  However, elevating these technical metrics to a level representing business relevance can often, and far too easily, merely establish their correlation with business performance.  Of course, correlation does not imply causation. 

This doesn’t mean that creating metrics to track compliance with your data governance policies is impossible, it simply means you must be as careful with the definition of the metrics as you were with the definition of the policies themselves. 

In his blog post Metrics, The Trap We All Fall Into, Thomas Murphy of Gartner discussed a few aspects of this challenge.

Truly meaningful metrics always align your data governance policies with your business performance.  Lacking this alignment, you could provide the comforting, but false, impression that all is well, or you could raise red flags that are really red herrings.

For additional information on aspects related to metrics, please refer to these blog posts:

• DQ-Tip: “There is no point in monitoring data quality…”
• Oughtn’t you audit?
• The First Law of Data Quality
• Data Governance and Data Quality
• Adventures in Data Profiling 

Remediation

Effective metrics will let you know when something has gone wrong.  Francis Bacon taught us that “knowledge is power.”  However, Jackson Beck also taught us that “knowing is half the battle.”  Therefore, the fourth step in enforcing data governance policies is taking the necessary corrective actions when non-compliance and other problems inevitably arise. 

Remediation can involve any combination of business processes, technology, data, and people—and sometimes, all four. 

The most common is data remediation, which includes both reactive and proactive approaches to data quality. 

Proactive defect prevention is the superior approach.  Although it is impossible to truly prevent every problem before it happens, the more control that can be enforced where data originates, the better the overall quality will be for enterprise information.

However, and most often driven by a business triage for critical data problems, reactive data cleansing will be necessary. 

After the root causes of the data remediation are identified—and they should always be identified—then additional remediation may involve a combination of business processes, technology, or people—and sometimes, all three.

Effective metrics also help identify business-driven priorities that determine the necessary corrective actions to be implemented.

For additional information on aspects related to remediation, please refer to these blog posts:

• Hyperactive Data Quality (Second Edition)
• A Tale of Two Q’s
• What going to the dentist taught me about data quality
• Microwavable Data Quality 
• DQ-Tip: “Data quality is about more than just improving your data...” 

Refinement

The fifth and final step is the ongoing refinement of your data governance policies, which, as explained above, you are enforcing for the purposes of supporting critical business decisions and enabling optimal business performance.

As such, your data governance policies—as well as their associated metrics—can never remain static, but instead, they must dynamically evolve and adapt, all in order to protect and serve the enterprise’s continuing mission to survive and thrive in today’s highly competitive and rapidly changing marketplace.  

For additional information on aspects related to refinement, please refer to these blog posts:

• The HedgeFoxian Hypothesis
• The Circle of Quality
• Is DG a D-O-G?
• Data Quality and The Middle Way
• Days Without A Data Quality Issue  

Conclusion

Obviously, the high-level framework I described for enforcing your data governance policies has omitted some important details, such as when you should create your data governance board, and what the responsibilities of the data stewardship function are, as well as how data governance relates to specific enterprise information initiatives, such as master data management (MDM). 

However, if you are looking to follow a step-by-step, paint-by-numbers, only color inside the lines, guaranteed fool-proof plan, then you are going to fail before you even begin—because there are simply NO universal frameworks for data governance.

This is only the beginning of a more detailed discussion, the specifics of which will vary based on your particular circumstances, especially the unique corporate culture of your organization. 

Most important, you must be brutally honest about where your organization currently is in terms of data governance maturity, as this, more than anything else, dictates what your realistic capabilities are during every phase of a data governance program.

Please share your thoughts about enforcing data governance policies, as well as your overall perspectives on data governance.

Follow OCDQ

If you enjoyed this blog post, then please subscribe to OCDQ via my RSS feed, my E-mail updates, or Google Reader.

You can also follow OCDQ on Twitter, fan the Facebook page for OCDQ, and connect with me on LinkedIn.

Red Flag or Red Herring?

A few weeks ago, David Loshin, whose new book The Practitioner's Guide to Data Quality Improvement will soon be released, wrote the excellent blog post First Cuts at Compliance, which examines a challenging aspect of regulatory compliance.

David uses a theoretical, but nonetheless very realistic, example of a new government regulation that requires companies to submit a report in order to be compliant.  An associated government agency can fine companies that do not accurately report. 

Therefore, it’s in the company’s best interest to submit a report because not doing so would raise a red flag, since it would make the company implicitly non-compliant.  For the same reason, it’s in the government agency’s best interest to focus their attention on those companies that have not yet reported—since no checks for accuracy need to be performed on non-submitted reports.

David then raises the excellent question about the quality of that reported, but unverified, data, and shares a link to a real-world example where the verification was actually performed by an investigative reporter—who discovered significant discrepancies.

This blog post made me view the submitted report as a red herring, which is a literacy device, quite common in mystery fiction, where the reader is intentionally misled by the author in order to build suspense or divert attention from important information.

Therefore, when faced with regulatory compliance, companies might conveniently choose a red herring over a red flag.

After all, it is definitely easier to submit an inaccurate report on time, which feigns compliance, than it is to submit an accurate report that might actually prove non-compliance.  Even if the inaccuracies are detected—which is a big IF—then the company could claim that it was simply poor data quality—not actual non-compliance—and promise to resubmit an accurate report.

(Or as is apparently the case in the real-world example linked to in David's blog post, the company could provide the report data in a format not necessarily amenable to a straightforward verification of accuracy.)

The primary focus of data governance is the strategic alignment of people throughout the organization through the definition, and enforcement, of policies in relation to data access, data sharing, data quality, and effective data usage, all for the purposes of supporting critical business decisions and enabling optimal business performance.

Simply establishing these internal data governance policies is often no easy task to accomplish.  Just as passing a law creating new government regulations can also be extremely challenging. 

However, without enforcement and compliance, policies and regulations are powerless to affect the real changes necessary.

This is where I have personally witnessed many data governance programs and regulatory compliance initiatives fail.

Red Flag or Red Herring?

Are you implementing data governance policies that raise red flags, not only for implicit, but also for explicit non-compliance? 

Or are you instead establishing a system that will simply encourage the submission of unverified—or unverifiable—red herrings?

Related Posts

Days Without A Data Quality Issue

Data, data everywhere, but where is data quality?

The Circle of Quality

DQ-Tip: “Don't pass bad data on to the next person...”

The Only Thing Necessary for Poor Data Quality

Data Governance and Data Quality

Mind the Gap

Photo via Flickr (Creative Commons License) by: futureshape

For many people, the phrase “mind the gap” conjures up images of a train platform, and perhaps most notably one used by the London Underground.  I’ll even admit to buying the T-shirt during my first business trip to England more than a decade ago.

However, lately I have been thinking about this phrase in a completely different context, specifically in relation to a recurring thought that was provoked by two blog posts, one written by James Chartrand in February, the other by Scott Berkun in May.

The gap I have in mind is the need to coordinate our acquisition of new information with its timely and practical application.

Information Acquisition

The Internet, and even more so, The Great Untethering (borrowing a phrase from Mitch Joel) provided by mobile technology, has created a 24 hours a day, 7 days a week, 365 days a year, world wide whirlwind of constant information flow, where the very air we breath is literally teeming with digital data streams—continually inundating us with new information. 

Of course, until they start embedding the computer chips directly into our brains at birth (otherwise known as the top secret iBaby experiment at Apple), we always have the choice of turning off all the devices and giving our full undivided attention to a single source of new information—such as a printed book or, even better, an in-person conversation with another human being.

However, when we are confronted by information overload, its accompanying stress is often caused by the sense that we have some obligation to acquire this new information—as if we were constantly cramming for a perpetually looming pop quiz. 

Contrast this perspective with Albert Einstein, who was known for not remembering even some of the most basic equations.  He argued why would he waste time memorizing something he could just look up in a book—when he needed it. 

This allowed Einstein to focus on problems nobody else could solve, as well as problems nobody had even thought of before, instead of learning what everyone else already knew.  He acquired more of his new information from his thought experiments than he did from books or other sources.

Filter Failure

As Clay Shirky famously stated, “it’s not information overload, it’s filter failure.”  I agree, but setting our filters is no easy task. 

Defending ourselves against information overload has become more difficult precisely because we now have greater individual responsibility for our own filters.  Not only are there more published books than ever before, but blogs, and other online sources of new information, have virtually eliminated the “built-in filter” that was provided by publishers, editors, and other gatekeepers. 

Please don’t misunderstand me—I am the complete opposite of Andrew Keen—I believe that this is a truly great thing. 

However, our time is a zero-sum game, meaning for every book, blog, or other new information source that we choose, others are excluded.  There’s no way to acquire all available information.  Additionally, cognitive load, a scientific theory that, in part, examines the limitations of our memory, explains why we often don’t remember much of the new information we do acquire.

Limiting ourselves to the few books and blogs we currently have the time to read, still requires filtering a much larger selection in order to make those choices—or we could simply choose to read only bestselling books and the blogs with the highest PageRank. 

However, can that approach guarantee access to the most valuable sources of new information?  Can any approach do this?

Information Application

Although acquiring new information is always potentially useful, it is when—and if—we can put it to use that makes it valuable. 

The distinction between useful and useless information is largely one of applicability.  If the gap in time between the acquisition and application of information is too great, then we would need to reacquire it, rendering the previous acquisition a wasted effort.

Perhaps the key point could be differentiating the type of potential knowledge provided by the information.  At a very high level, there are two broad categories of knowledge—explicit and tacit.

Explicit Knowledge

Explicit knowledge is relatively easily to acquire from either verbal or written information, and is often easily understood without extensive explanation.  Explicit knowledge can be based on a straightforward set of facts, or a specific set of instructions to follow, which after being repeatedly put to practical use just a few times, becomes easy to internalize and later recall when necessary. 

The information required for explicit knowledge is often best coordinated around when the knowledge gained would be used. 

One example is software training classes.  As an instructor, I always recommend minimizing the gap in time between when a training class is taken, and when the students would actually start using the software.  Additionally, an introductory class should focus on the most commonly used software features so students can master the basics before approaching advanced concepts.

Tacit Knowledge

Tacit knowledge is not only more difficult to acquire, but it is often not even easily recognizable.  Some lessons can simply not be taught, they can only be learned from experience, which is why tacit knowledge is sometimes alternatively defined as wisdom.

One of my favorite quotes about wisdom is from Marcel Proust:

“We do not receive wisdom, we must discover it for ourselves, after a journey through the wilderness, which no one can make for us, which no one can spare us, for our wisdom is the point of view from which we come at last to regard the world.”

Thought-provoking or paradigm-shifting information is often required to get us started on our journey through the wilderness of tacit knowledge, but we can easily lose sight of the deep forest it represents because we are far more immediately concerned with the explicit knowledge provided by the trees.

Whereas explicit knowledge is often more tactical in nature, tacit knowledge is often more strategic.  In general, we tend to prioritize short-term tactics over long-term strategy, thereby developing a preference for explicit, and not tacit, knowledge.

With tacit knowledge, the gap in time between information acquisition and application is much wider.  You require this time to assess the information before attempting to apply it.  You also need to realize that you will fail far more often when applying this type of information—which is to be expected since failure is a natural and necessary aspect of developing tacit knowledge.

Mind the Gap

As the growing stack of unread books on my nightstand, as well as the expanding list of unread blog posts in my Google Reader, can both easily attest, neither filtering nor acquiring new information is an easy task.

I have read many books—and considerably more blog posts—containing new information, which in retrospect, I can not recall. 

Obviously, in some cases, their information was neither valuable nor applicable.  However, in many cases, their information was both valuable and applicable, but I didn’t find—or more precisely, I didn’t make—the time to either put it to an immediate use, or to use it as inspiration for my own thought experiments.

I am not trying to tell you how to manage your time, or what new information sources to read, or even when to read them.

I simply encourage you to mind the gap between your acquisition of new information and its timely and practical application.

As always, your commendable comments are one of my most valuable new information sources, so please share your thoughts.

Related Posts

Eternal September and Tacit Knowledge

Podcast: Open Your Ears

The War of Word Craft

Will people still read in the future?

The Fragility of Knowledge

Hailing Frequencies Open

Recently Read: May 15, 2010

Recently Read is an OCDQ regular segment.  Each entry provides links to blog posts, articles, books, and other material I found interesting enough to share.  Please note “recently read” is literal – therefore what I share wasn't necessarily recently published.

Data Quality

For simplicity, “Data Quality” also includes Data Governance, Master Data Management, and Business Intelligence.

• Something happened on the way to better data quality – Rich Murnane discusses facing the challenging reality that around 80% of data quality “issues” at his organization were not “technology” problems, but instead “social” (i.e., human) issues.
• Data Profiling with SQL is Hazardous to Your Company’s Health – Stephen Putman explains that implementing a robust data profiling system is an essential part of an effective data management environment.
• How to deliver a Single Customer View – Ken O’Connor previews his e-book (available via Data Quality Pro free download)  on how to cost effectively deliver a Single Customer View that satisfies the UK Financial Services Authority requirements.  The process steps in the e-book would also be more generally applicable to anyone planning a major data migration project.
• Nerd Appeal or Boardroom Fare? – Marty Moseley explains data quality professionals generally do a very poor job in relaying the business value of data quality, and therefore we must strive to define meaningful, business relevant metrics.
• Blind Vendor Allegiance Trumps Utility – Evan Levy examines the bizarrely common phenomenon of selecting a vendor without gathering requirements, reviewing product features, and then determining the best fit for your specific needs.
• When Data Governance Turns Bureaucratic – Dan Power describes what he calls “reactive data governance” and how it can prevent organizations from realizing the full value of MDM.
• Data Quality: The Movie – Henrik Liliendahl Sørensen explains although you can learn data quality from courses, books, and articles, it’s a bit like watching a movie and then realizing that the real world isn’t exactly the same as the movie’s world.
• Why you should data profile – James Standen explains that initial data profiling provides crucial insight necessary for accurate estimates of the effort required on your business intelligence or data migration project.
• How are you Executing your Data Quality Strategy? – Phil Wright examines the high level characteristics of three different approaches to executing your data quality strategy—by power, by process, and by promise.
• Who’ll stop the rain – Frank Harland approaches the pervasive challenge of Business-IT alignment and collaboration from a new angle—by using data to form a divine triangle of Business, IT, and Data.
• “Dirty Harry” was right, “You've got to know your limitations” – Jim Whyte explains that MDM requires a deployment strategy that chunks up organizational and business process changes into small, manageable initiatives.
• Have you built your DQ trust today? – Thorsten Radde explains that a “blame and shame” approach, although somewhat cathartic, is not an effective tool for improving an organization’s data quality.
• The Data Accident Investigation Board – Julian Schwarzenbach outlines a “no blame” approach that would result in more data quality issues being reported, as well as leading to the true root causes of those problems being identified.
• I have a dream – Graham Rhind shares his dream of a revolution in data management, where the focus is on prevention of data quality problems, rather than on trying to resolve them only after their detrimental effect becomes obvious.
• My Data Governance Hero: A True Story – Amar Ramakrishnan shares a great story about encountering an unexpected hero who demonstrated an understanding of data governance and MDM challenges without using “industry speak.”
• Attributes of a Data Rock Star – Jill Wanless provides a great summary of the attributes of a “data rock star” based on an excellent online magazine article recently written by Elizabeth Glagowski.
• Three Conversations to Have with an Executive - the Only Three – Steve Sarsfield discusses how “data champions” must be prepared to talk about the value they bring to the organization in terms that will resonate with executives.
• Demarcating The Lines In Master Data Governance Turf Battles – Judy Ko explains a common challenge, namely how different groups within an organization often argue about master data—what it is, how it is defined, and who “owns” it.
• Data profiling: Are you closing the loop? – Dylan Jones explains how only using data profiling results to drive data cleansing efforts is missing the other part of the equation, namely also capturing and implementing defect prevention rules.
• Data Management Best Practices for Today's Businesses – Tony Fisher uses the Three R's of enterprise data management (Reduce, Reuse, Recycle) to explain how data is the one asset that every company has, but not every company exploits.

Social Media

For simplicity, “Social Media” also includes Blogging, Writing, Social Networking, and Online Marketing.

• Blogging: The Good, the Bad, and the Really, Really Bad – Brenda Somich provides a brief blog post succinctly conveying a few key points and providing some useful general advice regarding the art of effective blogging.
• The need for social media training is larger than ever – John Moore recaps a recent talk about extending thought leadership positions via social media, especially by leveraging it for professional networking—and while you are still happily employed.
• Information as Theater – The Power of Humanized Description – Jay Baer relates the story of Randy Lauson, the best flight attendant that he has ever seen, as a great story about how information isn’t boring by accident—you make it that way.
• New Adventures in Wi-Fi – Track 2: Twitter – Peter Thomas applies his very comprehensive but not overwhelming blogging style to the subject of Twitter, and thereby provides us with an excellent overview of my favorite social networking service.
• The 4 Es of Social Media Strategy – Jill Dyché explains that although over time your social media strategy can incorporate each of the 4 Es (Expose, Engage, Entertain, Educate), a single prevailing need will likely drive your initial efforts.
• What Role For The CMO In Social? – Mary Beth Kemp examines the possible roles that a Chief Marketing Officer (CMO), and the marketing department, could play in an organization’s social media strategy.  Includes a very useful diagram.
• Is Social Media a Fad? – On Day 6 of her 28 day blogging challenge, Tamara Dull shared a great video about social media, which includes some very compelling statistics provided by the Erik Qaulman book Socialnomics.
• Social Media Resistance: Déjà Vu All Over Again – Phil Simon compares the current resistance to social media adoption shown by many organizations, with their similar reluctance in the 1990s regarding the creation of a corporate website.
• Can you have a social system without a community or a collective? – Mark McDonald explains that not only can you have a social system without a community, approaching social media from this perceptive expands its true potential.

• Social Media and BI – Kelly Pennock explains that the newest frontier for data collection is the vast universe of social media, which you need to incorporate into your company’s overall business intelligence strategy.

Related Posts

Recently Read: March 22, 2010

Recently Read: March 6, 2010

Recently Read: January 23, 2010

Recently Read: December 21, 2009

Recently Read: December 7, 2009

Recently Read: November 28, 2009

Recently Read Resources

Data Quality via My Google Reader

Blogs about Data Quality, Data Governance, Master Data Management, and Business Intelligence

Books about Data Quality, Data Governance, Master Data Management, and Business Intelligence

Social Media via My Google Reader

Blogs and Websites about Social Media, Social Networking, and Online Marketing

Books about Social Media, Blogging, Social Networking, and Online Marketing

Podcast: Business Technology and Human-Speak

An excellent recent Marty Moseley blog post called for every one of us, regardless of where we sit within our organization chart, to learn conversational business-speak. 

This common call to action, perhaps first sounded by the George Colony blog post in August of 2006, rightfully emphasizes that “business is technology and technology is business” and therefore traditional IT needs to be renamed BT (Business Technology) and techies need to learn how to “engage in a discussion of process, customers, and operations, not esoteric references to SOA, Web services, and storage management.” 

Therefore, we need to always frame enterprise information initiatives (such as data governance and master data management) in a business context by using business language such as mitigated risks, reduced costs, or increased revenue, in order to help executives understand, as the highly recommended Tony Fisher book details, the need to view data as a strategic corporate asset.

While I do not disagree with any of these viewpoints, as I was reading the latest remarkable Daniel Pink book, I couldn’t help but wonder if what we really need to do is emphasize both Business Technology and (for lack of a better term) Human-Speak.

In this brief (approximately 9 minutes) OCDQ Podcast, I share some of my thoughts on this subject:

You can also download this podcast (MP3 file) by clicking on this link: Business Technology and Human-Speak

Related Posts

Not So Strange Case of Dr. Technology and Mr. Business

Podcast: Open Your Ears

Shut Your Mouth

Hailing Frequencies Open

The Game of Darts – An Allegory

Follow OCDQ

If you enjoyed this blog post, then please subscribe to OCDQ via my RSS feed, my E-mail updates, or Google Reader.

You can also follow OCDQ on Twitter, fan the Facebook page for OCDQ, and connect with me on LinkedIn.

Commendable Comments (Part 6)

Last September, and on the exact day of the sixth mensiversary (yes, that’s a real word, look it up) of my blog, I started this series as an ongoing celebration of the truly commendable comments that I regularly receive from my heroes—my readers.

Commendable Comments

On The Circle of Quality, Kelly Lautt commented:

“One of the offerings I provide as a consultant is around data readiness specifically for BI.  Sometimes, you have to sneak an initial data quality project into a company tightly connected to a project or initiative with a clear, already accepted (and budgeted) ROI.  Once the client sees the value of data quality vis a vis the BI requirements, it is easier to then discuss overall data quality (from multiple perspectives).

And, I have to add, I do feel that massive, cumbersome enterprise DQ programs sometimes lose the plot by blindly ‘improving’ data without any value in sight.  I think there has to be a balance between ignoring generalized DQ versus going overboard when there will be a diminishing return at some point.

Always drive effort and investment in any area (including DQ) from expected business value!”

On The Poor Data Quality Jar, Daragh O Brien commented:

“We actually tried to implement something like this with regard to billing data quality issues that created compliance problems.  Our aim was to have the cost of fixing the problem borne by the business area which created the issue, with the ‘swear jar’ being the budget pool for remediation projects.

We ran into a few practical problems:

1) Many problems ultimately had multiple areas with responsibility (line-of-business workers bypassing processes, IT historically ‘right-sizing’ scope on projects, business processes and business requirements not necessarily being defined properly resulting in inevitable errors)

2) Politics often prevented us from pushing the evidence we did have too hard as to the weighting of contributions towards any issue.

3) More often than not it was not possible to get hard metrics on which to base a weighting of contribution, and people tended to object to being blamed for a problem that was obviously complex with multiple inputs.

That said, the attempt to do it did help us to:

1) Justify our ‘claims’ that these issues were often complex with multiple stakeholders involved.

2) Get stakeholders to think about the processes end-to-end, including the multiple IT systems that were involved in even the simplest process.

3) Ensure we had human resources assigned to projects because we had metrics to apply to a business case.

4) Start building a focus on prevention of defect rather than just error detection and fix.

We never got around to using electric shocks on anyone.  But I’d be lying if I said it wasn’t a temptation.”

On The Poor Data Quality Jar, Julian Schwarzenbach commented:

“As data accuracy issues in some cases will be identified by front line staff, how likely are they going to be to report them?  Whilst the electric chair would be a tempting solution for certain data quality transgressions, would it mean that more data quality problems are reported?

This presents a similar issue to that in large companies when they look at their accident reporting statistics and reports of near misses/near hits:

* Does a high number of reported accidents and near hits mean that the company is unsafe, or does it mean that there are high levels of reporting coupled with a supportive, learning culture?

* Does a low number of reported accidents and near hits mean that the company is safe, or does it mean that staff are too scared of repercussions to report anything?

If staff risk a large fine/electric shock for owning up to transgressions, they will not do it and will work hard to hide the evidence, if they can.

In organizational/industrial situations, there are often multiple contributing factors to accidents and data quality problems.  To minimize the level of future problems, all contributory causes need to be identified and resolved.  To achieve this, staff should not be victimized/blamed in any way and should be encouraged to report issues without fear.”

On The Scarlet DQ, Henrik Liliendahl Sørensen commented:

“When I think about the root causes of many of the data quality issues I have witnessed, the original data entry was actually made in good faith by people trying to make data fit for the immediate purpose of use.  Honest, loyal, and hardworking employees striving to get the work done.

Who are the bad guys then?  Either it is no one or everyone or probably both.

When I have witnessed data quality problems solved it is most often done by a superhero taking the lead in finding solutions.  That superhero has been different kinds of people.  Sometimes it is a CEO, sometimes a CFO, sometimes a CRM-manager, sometimes it is anyone else.”

On The Scarlet DQ, Jacqueline Roberts commented:

“I work with engineering data and I find that the users of the data are not the creators of data, so by the time that data quality is questioned the engineering project has been completed, the engineering teams have been disbanded and moved on to other projects for other facilities. 

I am sure that if the engineers had to put the spare part components on purchasing contracts for plant maintenance, the engineers would start to understand some of the data quality issues such as incomplete part numbers or descriptions, missing information, etc.”

On The Scarlet DQ, Thorsten Radde commented:

“Is the question of ‘who is to blame’ really that important?

For me, it is more important to ask ‘what needs to be done to improve the situation.’

I don’t think that assigning blame helps much in improving the situation.  It is very rare that people cooperate to ‘cover up their mistakes.’  I found it more helpful to point out why the current situation is ‘wrong’ and then brainstorm with people on what can be done about it - which additional conventions are required, what can be checked automatically, if new functionality is needed, etc.

Of course, to be able to do that, you’ve got to have the right people on board that trust each other - and the blame game doesn’t help at all.  Maybe you need a ‘blame doll’ that everyone can beat in order to vent their frustrations and then move on to more constructive behavior?”

On Can Enterprise-Class Solutions Ever Deliver ROI?, James Standen commented:

“Fantastic question.  I think the short answer of course as always is ‘it depends’.

However, what’s important is exactly WHAT does it depend on.  And I think while the vendors of these solutions would like you to believe that it depends on the features and functionality of their various applications, that what it all depends on far more is the way they are installed, and to what degree the business actually uses them.

(Insert buzz words here like: ‘business process alignment’, ‘project ownership’, ‘Business/IT collaboration’)

But if you spend Gazillions on a new ERP, then customize it like crazy to ensure that none of your business processes have to change and none of your siloed departments have to talk to each other (which will cost another gazillion in development and consulting by the way), which will then ensure that ongoing maintenance and configuration is more expensive as well, and will eliminate any ability to use pre-built business intelligence solutions etc., etc.  Your ROI is going to be a big, negative number.

Unfortunately, this is often how it’s done.  So my first comment in this debate is - If enterprise systems enable real change and optimization in business processes, then they CAN have ROI.  But it’s hard. And doesn't happen often enough.”

On Microwavable Data Quality, Dylan Jones commented:

“Totally agree with you that data cleansing has been by far the most polarizing topic featured on our site since the launch.  Like you, I agree that data governance is a marathon not a sprint but I do object to a lot of the data cleansing bashing that goes on.

I think that sometimes we should give people who purchase cleansing software far more credit than many of the detractors would be willing to offer.  In the vast majority of cases data cleansing does provide a positive ROI and whilst some could argue it creates a cost base within the organization it is still a step in the direction of data quality maturity.

I think this particular debate is going to run and run however so thanks for fanning the flames.”

On The Challenging Gift of Social Media, Crysta Anderson commented:

“This is the biggest mindshift for a lot of people.  When we started Social Media, many wanted to build our program based only on the second circle - existing customers.  We had to fight hard to prove that the third circle not only existed (we had a hunch it did), but that it was worth our time to pursue.  Sure, we can't point to a direct sales ROI, but the value of building a ‘tribe’ that raises the conversation about data quality, MDM, data governance and other topics has been incredible and continues to grow.”

Thank You

Thank you all for your comments.  Your feedback is greatly appreciated—and truly is the best part of my blogging experience.

Since there have been so many commendable comments, please don’t be offended if one of your comments wasn’t featured. 

Please keep on commenting and stay tuned for future entries in the series.

Related Posts

Commendable Comments (Part 5)

Commendable Comments (Part 4)

Commendable Comments (Part 3)

Commendable Comments (Part 2)

Commendable Comments (Part 1)

Follow OCDQ

For more blog posts and commendable comments, subscribe to OCDQ via my RSS feed, my E-mail updates, or Google Reader.

You can also follow OCDQ on Twitter, fan the Facebook page for OCDQ, and connect with me on LinkedIn.