This Month’s Pattern *

53 Data Qualty

Data quality often sucks. A woefully common approach to this problem is to seek better software to process the data.

It’s not unusual for the quality of database software to exceed the quality of the data it processes, yet from the end-user’s viewpoint, system quality is limited by the lesser of the two. Companies everywhere are faced with databases full of inaccuracies and out-of-date or missing information. The problem is as obvious as the nose on your face, but like your own nose, it can be difficult to see. It’s hard for companies to come directly to grips with their own data-quality problems, though nobody has trouble seeing the other guy’s. What companies tend to see instead is a problem in the aggregate of software + data. Since the software is always easier to fix than the data (there is just so awfully much data), companies set out to fix or replace the software.

As none of this makes much sense, the essential thing to discuss here is not why we shouldn’t do it, but why we do it even though we shouldn’t. Part of the reason is a special instance of news improvement (see Pattern 45): The bad news that 2.4 percent of this month’s invoices were returned as undeliverable makes its way up the hierarchy, being greeted at each level with the angry question, “Well, what the hell are you going to do about this and damn quick?”

The damn-quick part immediately precludes extensive manual fixing. The vague answer is that a serious “data cleansing” effort will be started pronto. This charming little phrase means different things as it moves up toward the CEO level. At the bottom of the hierarchy, data cleansing means getting on the phone and Internet and poring over correspondence files to research and correct each separate bad datum. At the top, it means working smarter, somehow teasing out the right data by cleverly processing the bad data. Since funding comes from the top, the funds that are allocated are typically tied to the working smarter approach rather than to a small army of clerks to do the real work.

It’s worth pointing out that data can be corrupted (for example, by incorrect computing), and in this case, there are some at least partially automated ways to undo the damage by retrieving earlier backed-up versions. Similarly, when the same data are separately recorded in multiple systems, some automated data cleansing can help to isolate the better variant. In both cases, automated data cleansing depends on an ability to exploit data redundancy. While it’s easy to imagine an example of redundancy coming to our rescue (System A has an old address, but here’s a break: System B has the new one), real instances of poor data quality that can be automated away are few and far between.

The major cause of declining data quality over time is change. This spoilage in the asset we call “corporate data” can only be repaired by manual fix. Imagining otherwise just puts off the day of reckoning.



* Each month we plan to publish here one of the patterns from our Jolt Award book, Adrenaline Junkies and Template Zombies — Understanding Patterns of Project Behavior. (Watch this space for a mere 86 months and you'll have read the whole thing.) The book is published by Dorset House Publishing, in the US and Hanser Verlag in Germany. It is available at Amazon and also as a Kindle book.

events

DeMarco Keynote at OOP Konferenz 2015 in Munich
29-Jan-2015 to 29-Jan-2015

Tom DeMarco gives the keynote at the OOP Konferenz 2015 in Munich, January 29, 2015.

Brussels, Mastering the Requirements Process
24-Feb-2015 to 26-Feb-2015

James Robertson teaches Mastering the Requirements Process. Please contact I.T.Works for details.

Oslo, Mastering the Requirements Process
24-Feb-2015 to 26-Feb-2015

Suzanne Robertson teaches Mastering the Requirements Process. For more information on this popular course, contact Den Norske Dataforeningen

Oslo, ITSMF
3-Mar-2015 to 5-Mar-2015

Oslo, March 3-5. Suzanne Robertson presents a talk entitled Understanding and Communicating the Real Business Problem at the ITSMF Conference. She also runs a one-day workshop to explore The Requirements Food Chain. Details here

Hilversum, Mastering Business Analysis
17-Mar-2015 to 18-Mar-2015

James Robertson teaches the popular Mastering Business Analysis. Details from Adept Events in English or Dutch.

London, Mastering the Requirements Process
24-Mar-2015 to 26-Mar-2015

The Spring edition of Mastering the Requirements Process with James Archer. Please contact IRM UK for details.

Rome, Mastering the Requirements Process
13-Apr-2015 to 15-Apr-2015

Rome, Mastering Business Analysis
16-Apr-2015 to 17-Apr-2015

James Robertson teaches Mastering Business Analysis. Contact Technology Transfer for details of this new course.  

London, Mastering Business Analysis
23-Apr-2015 to 24-Apr-2015

James Archer presents Mastering Business Analysis. Please contact IRM UK for details of this course.

Brussels, Mastering Business Analysis
28-Apr-2015 to 29-Apr-2015

James Robertson and James Archer teach Mastering Business Analysis. Contact IT Works for details of this course.  

Oslo, Mastering the Requirements Process
28-Apr-2015 to 30-Apr-2015

Mastering the Requirements Process with Suzanne Robertson. Contact Den Norske Dataforeignen for details. 

Stockholm, Mastering the Requirements Process
5-May-2015 to 7-May-2015

Oslo, Mastering Business Analysis
12-May-2015 to 13-May-2015

James Robertson teaches Mastering Business AnalysisContact Den Norske Dataforeignen for details. 

Hilversum, Mastering the Requirements Process
19-May-2015 to 21-May-2015

James Robertson presents Mastering the Requirements Process for Adept Events. Details and registration: English - Dutch.

Brussels, Mastering the Requirements Process
16-Jun-2015 to 18-Jun-2015

James Robertson teaches Mastering the Requirements Process. Please contact I.T.Works for details. 

Oslo, Mastering the Requirements Process
15-Sep-2015 to 17-Sep-2015

Mastering the Requirements Process with Suzanne Robertson. Contact Den Norske Dataforeignen for details. 

Brussels, Mastering the Requirements Process
15-Sep-2015 to 17-Sep-2015

James Robertson teaches Mastering the Requirements Process. Please contact I.T.Works for details.  

London, BA Conference Europe
21-Sep-2015 to 23-Sep-2015

Business Analysis Conference Europe. Details and registration a Details and registration at Business Analysis Conference Europe 2015.

Stockholm, Mastering the Requirements Process
29-Sep-2015 to 1-Oct-2015

Hilversum, Mastering Business Analysis
5-Oct-2015 to 6-Oct-2015

James Robertson teaches the popular Mastering Business Analysis. Details from Adept Events in English or Dutch.

Brussels, MRP part 2
7-Oct-2015 to 8-Oct-2015

Brussels, Mastering Business Analysis
7-Oct-2015 to 8-Oct-2015

James Robertson teaches Mastering Business Analysis. Contact IT Works for details of this course.  

Rome, Mastering the Requirements Process
19-Oct-2015 to 21-Oct-2015

Rome, Mastering Business Analysis
22-Oct-2015 to 23-Oct-2015

James Robertson teaches Mastering Business Analysis. Contact Technology Transfer for details of this course.  

Oslo, Mastering the Requirements Process part 2
2-Nov-2015 to 3-Nov-2015

Suzanne Robertson teaches Mastering the Requirements Process part 2. Details for this advanced class at Den Norske Dataforeningen.

Hilversum, Mastering the Requirements Process
3-Nov-2015 to 5-Nov-2015

James Robertson teaches Mastering the Requirements Process. For details please contact Adept Events. Dutch description, or in English.

Oslo, Mastering the Requirements Process
4-Nov-2015 to 6-Nov-2015

Mastering the Requirements Process with Suzanne Robertson. Contact Den Norske Dataforeignen for details. 

London, Mastering the Requirements Process
11-Nov-2015 to 13-Nov-2015

James Archer teaches Mastering the Requirements Process. For details and registration, please contact IRM UK.

Wellington, Mastering the Requirements Process
24-Nov-2015 to 26-Nov-2015

The ever popular Mastering the Requirements Process. For details please contact Software Education.  

Melbourne, Mastering the Requirements Process
30-Nov-2015 to 2-Dec-2015

Mastering the Requirements Process. Please contact Software Education  for details and registration. 

Sydney, Mastering the Requirements Process
30-Nov-2015 to 2-Dec-2015

James Robertson teaches the popular Mastering the Requirements Process sponsored by Software Education.

in depth

Als auf der Welt das Licht Ausging, the German edition of Tom DeMarco's science fiction epic, Andronescu's Paradox, has now been published by Hanser Verlag in Munich.  Translation by Andreas Brandhorst.



James Robertson’s webinar for Software Education explains how agile stories are best used to ensure the right solution. Writing the Right Agile Stories on YouTube. Download the webinar slides.


Suzanne and James Robertson’s article The Requirements Food Chain explores how the originators and consumers of requirements interact with each other as the requirement matures.


Read Tom DeMarco's article from the July/August edition of IEEE Software: Sigil, BlueGriffon, and the Evolving Software Market.



Suzanne Robertson is one of the Agile Experts who discuss the subject of Scrum versus Kanban. The report is published by the Cutter Consortium and they have kindly made it available to readers of our web site. The lead author, Johanna Rothman, sets forth her argument that one is not necessarily better than the other; they are just different and it's up to the organization to figure out which method is best under which circumstance. In response, seven of Cutter's Agile experts discuss their views on Crossing the Agile Divide.


Complete Systems Analysis - the Workbook, the Textbook, the Answers by Suzanne Robertson and James Robertson is now in e-book format. It is available as a Kindle book, from InformIT, or you can download a sample chapter.


Tim Lister was one of the keynote presenters during Agile 2013. Tim’s talk “Forty Years of Trying to Play Well With Others” was a big hit. In this interview Tim shares some of the highlights of his talk.
Lister, Keynote at Agile 2013Tim Lister was one of the keynote presenters during Agile 2013. Tim’s talk “Forty Years of Trying to Play Well With Others” was a big hit. In this interview Tim shares some of the highlights of his talk
was one of the keynote presenters during Agile 2013. Tim’s talk “Forty Years of Trying to Play Well With Others” was a big hit. In this interview Tim shares some of the highlights of his talk - See more at: http://podcasts.bigvisible.com/?p=255#sthash.G1hS2edS.dp
was one of the keynote presenters during Agile 2013. Tim’s talk “Forty Years of Trying to Play Well With Others” was a big hit. In this interview Tim shares some of the highlights of his talk - See more at: http://podcasts.bigvisible.com/?p=255#sthash.G1hS2edS.dpuf
Tim Lister, Keynote at Agile 2013


Announcing the publication of the third edition of Tom DeMarco and Tim Lister's iconic text, Peopleware: Productive Projects and Teams.  The book is available now from Amazon or directly from Addison Wesley. See press release on Business Wire.



Tom DeMarco's speculative novel, Andronescu's Paradox is now available from Amazon and Barnes and Noble.  

"This war isn't going to blow anything up, only turn everything off."



Read Tom DeMarco's essay from the July/August issue of Software Magazine.  It's entitled, Bells, Whistles, Power, and the Requirements Process.



In this podcast James speaks of his experience in the profession of architecture and how it provides inspiration for his work on innovation and creativity. He also discusses the role of the business analyst in agile teams. Listen to the podcast.


There are now seven books by Guild authors available as ebooks, either as Nook Books or as Kindle Books.  



See what all the fuss is about. Tom DeMarco's article in IEEE Software seems to have annoyed practically everyone: "Software Engineering, an idea whose time has come and gone?"


Shane Hastie's interview with the authors and book excerpt: Mastering the Requirements Process on InfoQ.


The preparation course for the IREB "Certified Professional for Requirements Engineering" is now available as video training. Learn at home or any other place. Including questionnaires to prepare you for the multiple choice test.



Mastering the Requirements Process, third edition Getting Requirements Right is now available as a Kindle Book, Nook Book, or in traditional paper.



The Guild's Jolt Award book, Adrenaline Junkies and Template Zombies is now available in a Kindle edition.


Tom DeMarco and Tim Lister's perennially popular book Peopleware is now available in a Kindle Edition.


Tom DeMarco provides a video commentary about the Adrenaline Junkies book project. Learn how this Jolt Award-winning book came about.