Apache NiFi - Data Provenance
Apache NiFi logs and store every information about the events occur on the ingested data in the flow. Data provenance repository stores this information and provides UI to search this event information. Data provenance can be accessed for full NiFi level and processor level also.
The following table lists down the different fields in the NiFi Data Provenance event list have following fields −
| S.No. |
Field Name |
Description |
| 1 |
Date/Time |
Date and time of event. |
| 2 |
Type |
Type of Event like ‘CREATE’. |
| 3 |
FlowFileUuid |
UUID of the flowfile on which the event is performed. |
| 4 |
Size |
Size of the flowfile. |
| 5 |
Component Name |
Name of the component which performed the event. |
| 6 |
Component Type |
Type of the component. |
| 7 |
Show lineage |
Last column has the show lineage icon, which is used to see the flowfile lineage as shown in the below image. |
To get more information about the event, a user can click on the information icon present in the first column of the NiFi Data Provenance UI.
There are some properties in nifi.properties file, which are used to manage NiFi Data Provenance repository.
| S.No. |
Property Name |
Default Value |
Description |
| 1 |
nifi.provenance.repository.directory.default |
./provenance_repository |
To specify the default path of NiFi data provenance . |
| 2 |
nifi.provenance.repository.max.storage.time |
24 hours |
To specify the maximum retention time of NiFi data provenance. |
| 3 |
nifi.provenance.repository.max.storage.size |
1 GB |
To specify the maximum storage of NiFi data provenance. |
| 4 |
nifi.provenance.repository.rollover.time |
30 secs |
To specify the rollover time of NiFi data provenance. |
| 5 |
nifi.provenance.repository.rollover.size |
100 MB |
To specify the rollover size of NiFi data provenance. |
| 6 |
nifi.provenance.repository.indexed.fields |
EventType, FlowFileUUID, Filename, ProcessorID, Relationship |
To specify the fields used to search and index NiFi data provenance. |
