DataFlux

The recent Wilshire Data Governance conference opened up some new debates, rehashed some old ones, and definitely got people thinking.

As I’ve said before, much of what’s been written about data governance up until now confuses the term “data governance” with either “data management” or “data quality.” Add data stewardship to that list for good measure. In some subsequent blogs, I’ll discuss what I consider some hot debates in the fields of data governance and data stewardship, and throw in my own two cents for good measure. - Read the rest of this entry »

The other day, I was discussing with our new “Quality Dude”, exactly what quality metrics could be measured in our environment. You may remember, from earlier blog entries, that I work for an ASP business. Customers upload their data to us nightly, weekly, monthly or quarterly. So, we got the CEO, President, and a couple of people from the development staff together in a room, and I asked, “What metrics about quality do we measure?” Here is what we found during our discussion: - Read the rest of this entry »

One of my favorite clients, Matt, the Manager of the BI Competency Center at an insurance company, just got a significant promotion. Matt’s now Director of Data Services and BI. Matt’s promotion is a testimony to the business’ newfound data awareness. The semantics of the title actually indicate a newly-mature view of data as broader than just BI. - Read the rest of this entry »

The other day, I got a phone call from an old friend who had read an article I wrote in DMReview magazine. The article was called Data Profiling Made Easy – Part 1. We discussed some open source programs that do a simplified analysis of the data source, something I would call structural analysis. Since I work for an open source company, I thought this might be a viable solution to consider. Open source is readily available from many internet sites, and it’s FREE! Programs found on these sites seem to be centered-around a specific product like PHP or Perl. - Read the rest of this entry »

My kids are addicted to “tween-age” social networking web sites, specifically Club Penguin and Webkinz. These sites are relatively simple, they provide hours of gaming entertainment, and they provide an interesting virtual interface for interaction, as each “player” is represented via an avatar that the player can house, dress, feed, etc., using virtual money earned through either playing games or taking on “work tasks. - Read the rest of this entry »

We recently had two different client experiences with data profiling. In one, the customer spent some time attempting to understand what profiling was and how it would be used as part of both a proactive assessment and a preventative monitor. We crafted a “run-book” for their use of the profiler, listing a number of different strategies for using the tool for specific purposes. As they piloted their use of a profiling tool, they created scenarios that would test these different strategies, which provided them with feedback as to the tool’s best use. - Read the rest of this entry »

The other day, one of the guys I work with asked me, “Joyce, what makes a ‘good’ data profiling and data quality architecture?” I thought, “Wow, what a great question.” Like any good consultant, my answer was “It depends!”

Deciding where to do data profiling and data quality always depends on what we are creating. For instance, if I am building a traditional data warehouse, data profiling checks and cleansing with the data quality tool can usually be completed before the data is loaded into the data warehouse. Data can also ‘land’ in the staging area and be integrated prior to load into the data warehouse. Some companies integrate it in the staging area (from all the data sources), do one more data profiling check, and then load into the data warehouse. While this may cause more ‘touching’ of the data, it works very well for many companies. - Read the rest of this entry »

What is a business rule?

I have been doing a lot of pondering recently regarding the notion of a business rule, something I do every time I encounter the use of the term at a client site where it seems to have lost all meaning. The way the term is used with this client is more to describe a procedural data transformation as information is extracted from a source and delivered into a target. But I have always thought of business rules as declarative, meaning that they assert a truth, expressing what should happen without presupposition of how it happens. - Read the rest of this entry »

As with all topics, there are good data quality presentations and there are bad data quality presentations.

You’ve probably seen the bad ones. The speaker stands before a genuinely engaged group and starts haranguing audience members about why it’s their fault that executives in the C-suite don’t understand that data quality matters. And how it’s their job to do the heavy lifting and instill a set of procedures on the company, and how the company can’t be competitive until everyone from the management ranks to the mailroom understands the risks of bad data. - Read the rest of this entry »

I can’t believe our project manager is leaving us! For those of you who know me, IT WAS NOT ME! This is definitely not going to be easy on any of us at the office. I wish him the best of luck with all of his future employment opportunities, and I hope we stay in touch. So, where does this leave us? Well, we are definitely going to need to get a handle on where we stand with getting better quality data from our customers for the data warehouse. - Read the rest of this entry »