As data is frequently updated within an organization, achieving quality data requires constant monitoring and measurement as well as improvement actions. In addition to defect correction for quality data, it also requires prevention and reporting. The following table shows four important reasons why the quality of data in an organization has deteriorated in the past few years.
| Cause | Explanation |
|---|---|
| External data sources | Much of the organizational data originates outside, where there is less control over the data sources to comply with the expectations of the receiving organization. |
| Redundant data storage and inconsistent metadata | Many organizations have allowed redundant, inconsistent, and incompatible data through the uncontrolled proliferation of spreadsheets, desktop databases, legacy databases, data marts, data warehouses, and other repositories of data. |
| Data entry problems | User interfaces that do not take advantage of integrity controls—such as automatically filling in data, providing drop-down selection boxes, and other improvements in data entry control— are tied for the number-one cause of poor data. |
| Lack of organizational commitment | Not recognizing poor data quality as an organizational issue because of budget and commitment issues. |
How to Improve the Data Quality?
Implementing a successful quality improvement program will require the active commitment and participation of all members of an organization.
Some of the key steps which can help improve data quality in an organization are given below.
- Get the Business Buy-In
An organization should view data quality as a business imperative rather than an IT project. Most organizations need to obtain the appropriate level of executive sponsorship for a good business case related to data quality. It is also important to identify and define the key performance indicators and metrics that can quantify the results for improving data quality.
- Conduct a Data Quality Audit
An organization without any established data quality project should begin with an audit of data to understand the extent and nature of data quality problems. Data quality audits can include many procedures such as statistically profiling the files and keeping track of all the records of data at the table level. With a data audit, we can identify the obscure and unexpected extreme values.
Statistical analysis helps to analyze patterns of data using their distribution, outliers, and frequencies. Data can be checked against certain relevant business rules (counts, means, and variance) and can send an alert (email, reports) if those rules are broken.
- Maintain a data stewardship program
Data steward/Data Governance must ensure that the data that are captured within the organization are accurate and consistent throughout the organization so that users within that organization can rely on that data.
- Improve the Data Capture Process
Improving the data capture process is a fundamental step in a data quality improvement program. Critical points of data entry related to where data are:
- Originally captured
- Pulled into a data integration process
- Loaded into an integrated data store (e.g. Data warehouse)
- Apply Modern Data Management Principles and Technology
There are different powerful software available that can assist users with technical aspects of data quality improvement. This software employs advanced techniques such as pattern matching, fuzzy logic, and other expert systems to analyze data for quality problems, identify and eliminate redundant data, and integrate data from multiple sources.
- Apply Total Quality Management(TQM) Principles and Practices
Data quality improvement is a continuous effort that should not be treated like a one-time project. With this in mind, many leading organizations apply Total Quality Management (TQM) to improve data quality. Even though TQM has many principles, some principles are used to prevent defects (rather than a correction), continuous improvement of the processes that touch the data, and use enterprise data standards.