There are many modern tool options in the data catalog segment, but new leaders are emerging that harness the advanced capabilities needed for the next generation in cataloging. The future catalog is not just a list of assets, but the combination of, at the minimum, features that enable companies to operate seamlessly and scale up and out, effectively and efficiently.

An enumerated list of data sources and usually, their business context (i.e., a repository for HR or Finance business process data)
Explicit, current details, or metadata, about data assets
The semantic context and understanding of what those assets mean and how they are related to other assets, concepts, and data.
Tools to automate and enable work on large amounts of data (AI/ML, scripting, etc.).
A transport or integration layer that enables the automation of advanced analytics required for next-generation data workloads, such as a knowledge graph or a large language transformer model.

A carefully implemented data catalog with these next-generation capabilities enables organizations to establish a single point of reference, and vectors to select data assets while maintaining Positive Data Control with complete lineage and provenance.

The data catalog, while generally not a repository or warehouse and system of record for the Data itself, ingests metadata that supports corporate automation requirements, operational activities and decision-making processes to meet business operations, data processing, regulatory compliance and reporting, and advanced analytics to the myriad of stakeholders.

This metadata can be used by management, staff, auditing officials, and data scientists and researchers to enable decision-support by way of useful, reliable, high-quality evidence that is timely, relevant, and detailed enough to inform policy and procedures.

The data catalog further enables policymakers with self-service data access, provided safe in the knowledge that authenticated users are accessing trusted data that is accessible under permitted circumstances.

There are many types of metadata, but most practitioners are focused on delivering data organized by business, technical, operational, lineage, transformations and calculations, and provenance, or rights management. Regardless of the category, metadata enriches the value of the data to consumers.

Metadata serves several purposes, each of which contributes to business value proposition through the data platform:

Transparency is vital to realizing the promise of evidence-based policymaking, where “evidence-based” means including information as to what data mean and why they should be trusted. Transparency, in turn, requires that enough of this information is provided. Loosely speaking then, transparency is achieved when sufficient documentation is provided.
Metadata Automation is the ability to facilitate collection, visibility, and control of metadata. Data professionals across the spectrum spend a great deal of time and energy manually exploring, discovering, and understanding metadata. Automation, on the other hand, turns several weeks of manual labor into minutes and allows the business to move more quickly and efficiently.
Optimized Metadata Search makes it possible for analysts and technologists to find the data, whatever that data may be, within and across the broad data ecosystem, often across business units, agencies, or partnering organizations.

This provides value because it reduces duplication of effort and resources by agencies, supports change management opportunities, makes it easier to track who is responsible for the quality and timeliness of that data, identifies potentially marketable content, and identifies gaps in the data strategy that can be filled with investment in quality data sources.

Rich Analytics resulting from significant data preparation effort. Most data scientists spend more than 90% of their effort finding, cleansing, and refactoring data from one-to-many sources to begin analysis processes. The process of managing metadata also directly impacts the quality and interoperability of the data that the metadata describes.

In other words, the information that is generated by your operations, data scientists and business analysts is likely to be more accurate, timely, and those same people can spend more time mining that data for meaningful insights rather than simply spending their time compensating for bad data. And that means better information for future planning and decision-making.

Effective Data Governance and Metadata Management have a symbiotic relationship. Most Data Governance is focused on interpreting provenance (where data comes from), defining authoritative sources (what constitutes a primary or golden record), establishing stewardship (who is responsible for ensuring the operational integrity), validating definitions (how exactly the information is defined, often including citational context), temporality (when was the data first known and how often does it change), and business purpose (why was the data captured in the first place). Governance is largely involved with telling the "story" of data and its relevance to the organization.
Understanding and managing interactions with the external data environment is critical and often overlooked. Metadata management requires understanding and integrating data from the external environment. This includes data from constituents, vendors, counterparts, agencies, government, and regulatory organizations. Using emerging metadata standards and protocols such as ISO 8000, NIST, FISD and SIIA, prov-o, schema.org, and others, there's a growing consensus about how to use and manage common structures - organizations, individuals, contracts and transactions, vehicles, scientific, biological, and medical information, and other subject areas that increasingly ensures at least an 80% solution for data interchange with the outside world.

However, making it easier to find data, that data must be trusted and endorsed by reliable stewards with controls established through data governance boards, working groups, and data governance operations.

OneTrust, Chief Strategy Officer: Leveraging the Data Catalog

Related Stories