Data has been managed for centuries, millennia even, with modern data management making up the last 100 years or so just a snippet of this journey. Following this history, today’s data issues—such as quality, security and trust—are not new. They have simply not been attended to over the years, resulting in decades of underinvestment in data due to behaviors such as short-termism and a lack of data expertise and awareness of data’s value.
Understanding the evolution of data management helps provide insights into current challenges and guides future strategies to resolve them. Each data management generation has had its strengths and weaknesses, and organizations can learn from past experiences to shape more effective data management approaches that better enable future generations of data management.
Looking back through 100 years of data management, quality, security and trust are recurring themes.
Quality
Despite advancements in digital data management over the last quarter of a century, merely a fraction of an organization’s critical data manages to meet basic quality standards. Poor data quality leads to mistrust and compromises an enterprise’s ability to make informed decisions and meet its objectives.
Organizations should prioritize data quality by implementing robust data governance practices, ensuring data accuracy, completeness and reliability. A structured way of starting a data quality journey is to identify the data-based pain points that an organization experiences and demonstrate to leadership how data management can resolve those pains. The goal of this engagement is to get the organization to support the resolution of its data pain points.
However, no matter how good the quality of a particular data set is, it is compromised by data duplication across an organization. The concept of a data mesh emphasizes federated governance and the creation of data products owned by domain experts, which should help reduce duplication when data product owners encourage data sharing rather than copying. Trustworthy data products that meet defined operational needs and undergo regular quality checks for consistency across the organization are key outcomes of effective data management.
Security and Access Control
Many employees have access to data they should not have, posing a potential security risk. Organizations need to establish or review their access control mechanisms to prevent unauthorized access and potential data breaches.
Regular security audits and monitoring are crucial to identify and address vulnerabilities. Also, many breaches occur because the basics of cybersecurity—such as password management, patch management, firewall configuration and security awareness training— are not being practiced well, if at all. There is no point pursuing advanced security practices if the basics of cybersecurity are not being effectively implemented.
Data governance and data management practices around metadata and privacy complement the security conversation, the processes of many of which are automatable. However, careful consideration must be given to security risk introduced by automation, with security as an inherent part of complementary and supporting data management practices and data governance by design.
Trust
Trust in the stewardship of those organizations that capture data about the public has eroded due to numerous high-profile failures and abuses of personal data. Organizations, both in the public and private sectors, need to prioritize rebuilding trust by being transparent about their data management practices, protecting user privacy and transparently and speedily addressing concerns related to data misuse and accuracy.
The decline in trust in the public sector contributes to the rise of the spread of misinformation across the political spectrum, a destructive force for everyone involved. Building trust in data management practices to help combat these issues begins by putting the appropriate data management practices and their supporting governance in place and by enhancing transparency around these with the goal of providing accurate and reliable information to the public to reverse the trend.
Finding the Will to Improve Data Management
Of these three issues—quality, security and trust—all are controllable. What is missing is whether there is a will and a supporting budget to address them. But senior technical leadership should not wait too long to decide, as the demand for trustworthy data will increase, while data management technologies continue to become more sophisticated, leaving ever greater data investment gaps.
Data management has undergone significant transformation over the last 100 years or so, from paper records to punched cards, magnetic tape, data warehouses, data lakes, data fabrics and data meshes, and it is set to continue. Organizations must stay informed about emerging technologies and data management paradigms, and regularly evaluate their applicability based on their specific data management needs.
Ultimately, insights from the past highlight the importance of prioritizing data quality, establishing robust security measures, and building trust by leveraging emerging technologies, implementing effective data governance, and adopting innovative data management approaches to sustainably and consistently improve data management practices.
Editor’s note: For further insights on this topic, read Guy Pearce’s recent Journal article, “Three Lessons From 100 Years of Data Management,” ISACA Journal, volume 4, 2023.