- ABOUT
- VIDEOS
- TOPICS
- TESTIMONIALS
- COMMUNITY
- EVENTS
- NOMINATIONS & LISTS
- SPONSORSHIP
- ABOUT
- VIDEOS
- TOPICS
- TESTIMONIALS
- COMMUNITY
- EVENTS
- NOMINATIONS & LISTS
- SPONSORSHIP
(US and Canada) Phil Miller, Co-CEO and Co-Founder, Solidatus, talks with Baz Khuti, President Modak USA, about the rationale behind starting Solidatus, his approach to work as an entrepreneur, and the importance of data lineage in data management.
Miller realized during previous jobs that a lot of the regulatory issues he was addressing were similar, giving rise to the need for organizations to understand how their data can work better. This manifested itself as a problem involving metadata — the actual context in which data operates and is processed as a part of an ecosystem.
“Organizations are not looking after their blueprint as well as they look after their codebase. They're not looking after the back-office functions, which enable them to get their license to operate,” Miller says. “Every year I was having to relearn an organization and I thought, what if we didn't have to relearn the organization every day? What if the organization was self-documenting, a bit like a codebase that looks after itself in source control?”
Regarding the importance of data lineage, Miller explains that even the most organized of organizations have multiple sources of data and multiple ways of using it. Not understanding the lineage leads to inefficient, slow project turnaround, among other problems.
“We think about data lineage as relationships,” he says. “Data is born somewhere, it's used in some places and transformed; it's transferred verbatim around an organization and it eventually goes somewhere as an output. So, it's inputs and outputs with a whole bunch of connectivity between — sort of the blueprint of the organization. If you draw the lines of lineage together, you can link not only data processing facilities to data processing facilities or databases to databases, but also the people who use that data, the processes for which they use that data, and the places they use that data.”
Even the various concerns around data usage, such as data privacy, environmental impact, etc., are linked, Miller notes. This makes it impossible to pull one strand of an organization without pulling others at the same time. So, data lineage is, in a way, the weave of the organization, Miller adds.
“That's how an organization is built. Not recognizing that is bad because your organization cannot be conceptually designed. It's physically built, and not understanding how the physical and the conceptual are linked — that's a risk,” Miller concludes.