Data lineage is an important function of data governance that tracks the journey of data from its origin to its final destinations via various hops.
Codes written in these procedural languages lack inherent data lineage capabilities.
You need not only a technical expert but a functional expert to establish data lineage.
Like Ab Initio and SAS DI studio, can generate their data lineage.
An organization having a mix of technologies may have partial/fragmented lineage available but not the complete lineage.
Such lineage will be disconnected as Ab Initio will hold lineage for codes written in AB initio, and SAS will hold lineage for codes written in SAS. So even lineage is present - it's a disconnected lineage.
The presence of these EUDAs often disrupts the data lineage, creating gaps and inconsistencies within the lineage records.
The organizations regulated by governments and independent bodies have been mandated to demonstrate the data lineage.
The same analogy can be applied to data lineage, where we need different types and levels of data lineage for different users.
High-level business lineage helps in data governance and compliance Process Level.
Technical lineage at this level focuses on the details of technical data flow, including the source of the data and the high-level transformations it undergoes.
This lineage is one level below entity level lineage where the grain of the lineage is at individual attribute in a table.
In lineage terms, these are called direct contributors or direct lineage.
In addition to direct lineage, attribute-level lineage also covers indirect lineage.
Indirect lineage is also referred to as dependency lineage or conditional lineage.
Adding indirect lineage to lineage analysis may complicate the lineage and, in some cases, would add huge attributes to the analysis.
A good data lineage tool must provide a filter to include or exclude indirect lineage from presentation/lineage output.
Every data movement - where data changes - is a lineage contributor.
Data lineage serves as an important tool in tracing the complete journey of an attribute from its inception to its final destination within the data flow.
Data lineage plays a pivotal role across various aspects of data management, data governance, and system evolution.
This Cyber News was published on feeds.dzone.com. Publication date: Fri, 08 Dec 2023 15:13:05 +0000