Monday, 21 July 2014

Data Cleaning-How Important is it?

Data cleaning in itself is never the end goal. Instead it is the key step towards one. Organizations spend a lot of time, money and resources into gathering data through a variety sources. The primary reason for this effort besides legal, compliance and customer service requirements is to help make quality management decisions. If the database for whatever reason is not error free, the resultant analysis and interpretation is most likely to be of poor quality. And when this recorded data is unable to support this crucial requirement, the huge investment of time, effort and manpower remains sadly largely wasted.

So what really constitutes the ‘dirty data’? While not necessarily always incorrect, any data that suffers from errors related to formatting or is out dated, partially captured, captured once too many times, or no longer relevant is labeled ‘dirty’. The reasons for this can generally be traced to how data is entered ( is the staff trained, is the system user friendly?) to how it is stored (are there regular audits, is it formally recorded and maintained?)

Most organizations maintain several databases that may or may not talk to each other. In order to interpret insights however they need to merge at some point. This becomes a huge challenge as each source may carry different representation of the data, many duplication errors and also redundant information. The job of data cleaning includes ensuring clear consistency across these different types of sources  to allow for easy merging when needed and hence a comprehensive and easy overall analysis.

An interesting but lesser known (or appreciated) fact in the process of analyzing data is that a major chunk of the overall allotted time is used to simply clean the  data and beat it into the shape and state that  is conducive for interpretation. It could be possible, for example, that the data available in a company may be factually correct but difficult to process through the analytics systems. Data cleaning then has to ensure that such data is captured in a manner that makes it easy to be used in data analytics.

Deduplication, column segmentation and matching of records are the basic methods of data cleaning. However depending on the data type, the methodology is likely to vary. On a basic process level we begin with inspecting data samples to get a handle of the kinds of errors that need attention. This will then lead to creating detailed workflows for improving data quality, laying down thought out processes of error correction, listing rules for future data capturing etcetera. Once this is done, the corrected data is retested for errors missed and its usability for analytics.

Laborious and at times, time consuming though it may seem, data cleaning certainly is the most crucial step towards procuring in depth  insights that empower the management with all the relevant information to improve the quality and speed of  decision making.

1 comment:

  1. I'M NOW FULFILL FINANCIALLY BECAUSE OF THE LOAN I GOT FROM LFDS .I would like to bring this to the notice of the public about how i came in contact with LFDS after i lost my job and being denied loan by my bank and other financial institution due to my credit score. I could not pay my children's fees. I was behind on bills, about to be thrown out of the house due to my inability to pay my rent, It was during this period my kids were taken from me by foster care. Then i set out to seek for funds online where i lost $3,670 that i borrowed from friends which i was rip off by two online loan companies. Until i read about:Le_Meridian Funding Service  ( / somewhere on the internet, Still wasn't convince because of what i have been through until a relative of mine who is a clergy also told me about the ongoing loan scheme of LFDS at a very low interest rate of 1.9%% and lovely repayment terms without penalty for default of payment. I have no choice than to also contact them which i did through text +1-989-394-3740 and Mr Benjamin responded back to me That day was the I'M best and greatest day of my life which can never be forgotten when i receive a credit alert of $400,000.00 Usd loan amount i applied for. I utilized the loan effectively to pay up my debts and to start up a business and today i and my kids are so happy and fulfill. You can as well contact them through email: ( /  WhatsApptext  helpline: +1-989-394-3740 Why am i doing this? I am doing this to save as many that are in need of a loan not to be victim of scams on the internet. Thanks and God bless you all, I'm Oleksander Artem from Horizon Park BC , Ukrain.