DataFlux

After our last data quality metrics meeting, I set up new SLA’s (service level agreements) with our customers. I set up specific options of measurement — for instance, query performance time based on the complexity of a query, and how to protect customers from themselves. I ran into the funniest and weirdest thing today. As an ASP model, in the past this company pretty much accepted data the way it came in. Oh, we check to make sure that identifiers were unique within a customer, that numbers are numbers and dates are dates — but, the one measure we didn’t do was to profile the customer’s data and alert them to any defective data that may have come from their source system. - Read the rest of this entry »

Apparently, what is now being called “Predictive Analytics” is what I use to know as “data mining.” I guess the term “data mining” has been co-opted by so many different applications that it has lost its original meaning. The definition of data mining that I was familiar with had to do with using algorithms for learning and extracting “knowledge” from data sets - another term, KDD (”knowledge discovery in databases”) has also been used. - Read the rest of this entry »

In the words of Livy, a gadfly of the emperor Augustus and author of the history of Rome: “An empire remains powerful so long as its subjects rejoice in it.” This means that the DataFlux User Event should be around for a while since it makes me and my colleagues in the data quality and integration communities rejoice, and happily herald its return next month, from September 10-13. - Read the rest of this entry »

I just got off the phone with the panelists for the upcoming DataFlux Users Event - September, 2007 in Orlando. There, I will be hosting a panel on propagating data governance throughout the enterprise. I find this subject particularly interesting because I think the definition of data governance is misunderstood, and misinterpreted by most organizations. Most people seem to be overwhelmed by what it takes to set a good data governance program. As you will see in this session, I don’t think it takes a 10 person team or a ton of money. - Read the rest of this entry »

For those of you who don’t remember, I work for a software company that uses an ASP model for data collection. The other day, I found out that the data in our two databases was out of sync again. Database number 1 holds the real-time data, while database number 2 holds the data warehouse. In the data warehouse, we track all the different stages of an object through its life cycle. Very cool stuff! - Read the rest of this entry »

I have a confession to make. I saw something I wasn’t meant to see. And I kept looking. You would have, too. C’mon, you would have! Before you pass judgment, let me explain.

You see, on a flight from Charlotte to Atlanta, I was stationed in a middle seat in a 737 (3+3 configuration). I was wearing khaki shorts, my broken-in Keen sandals and cap that said “Cabo.” You get the picture: I looked like a slacker who’d spent the weekend hiking in the Blue Ridge Mountains, which indeed I was. - Read the rest of this entry »

The new system is an ASP model for customer information. It will provide analytics at a lower level of granularity. As the new system is implemented, and our customer’s data is brought into the system, we are seeing more and more data quality issues. For instance, customer information should be mapped correctly to our fields in the database management system, but as the program ran, we found certain fields that needed more than just a check to see if they were numeric, alphanumeric, or dates. Thus, our acquisition programs with the quality validations (left over from a previous system) are failing miserably. - Read the rest of this entry »

At an early point in the development of data profiling tools, the objective was to count things and present those counts back to the user. Most profiling tools do this handily - they count the number of distinct items in each column, then summarize details about the value set. But at what point do profiling tools better serve the user when counting things then traditional approaches to counting? For example: some of the same summary data profilers provide can be achieved using SQL queries (into databases), awk/sed and/or perl scripts (if you know what those are, then you are definitely a techie), C/C++/Java code, or even dumping files into excel and using the provided functionality. - Read the rest of this entry »

The other day I was chatting with the network ‘dude’ at one of my clients about the data coming into the company from the customers. I was thinking about the possibilities of what we could do with the right data — you know, analytics on cubes and data marts and the metrics we could keep over time. He said, “We own the data, we can do anything we want!” I sat there for a moment, and thought, “Do a lot of information technology people believe if you are in possession on the data, that the data belongs to you?” I think so! Which brings me to the idea of governance. - Read the rest of this entry »