Introduction of Watermarking in Mule 4

Watermarking in Mule 4 is a powerful technique that ensures efficient data processing by marking the state of a data source at a specific point in time. This method prevents reprocessing already handled data, focusing only on new or updated information. Through this article, we explore how watermarking functions within Mule 4 connectors, along with practical examples and the advantages of using Object Store V2 for persistent data handling.

How Watermarking Works

1. Initial State: When data processing starts, an initial watermark is set. Watermarks could be a timestamp, identifier, or any other marker representing the data’s state at that moment.

2. Data Retrieval: As data is retrieved or polled, it gets compared against the existing watermark. This comparison helps identify new or updated records since the last recorded state.

3. Processing: Only files/data newer than the existing watermark are processed, ensuring efficient and up-to-date data handling.

4. Update Watermark: After processing, the watermark gets updated to reflect the current state of the data. This updated watermark serves as the reference for the following data retrieval cycle.

Exhibiting Watermarking

Here’s a simplified example of how you might set up watermarking with the File Connector:

Initial Watermark: Store the timestamp of the last processed file.

Polling and Comparison:

Use the File Connector to list files in a directory.

Compare file timestamps or names with the stored watermark.

Process and Update:

Process files that are newer than the stored watermark.

Update the watermark with the latest timestamp or filename.

Implementing Watermarking Before Data Retrieval

The above implementation works with remarkable efficacy when fetching files from a folder or rows from a database. However, the goal is to implement watermarking before retrieving data or files from the respective connectors.

This article edifies how Watermarking can be achieved using various Mule 4 connectors.

Default Watermark Features in Mule 4 Connectors

Mule 4 connectors have built-in watermarking features that can be used for efficient data processing. Let’s dive into some specific connectors:

File- On New or Updated File

SFTP- On New or Updated File

FTP- On New or Updated File

Database- On Table Row
Let’s dive into each connector to learn how watermarking is achieved.

File – On New or Updated File

In a File connector, there are two modes for watermarking strategies: “CREATION_TIMESTAMP” or “MODIFIED_TIMESTAMP”. The strategy depends on whether you want to filter files based on their creation or modification time.

In the File Connector, there are three watermarking options:

DISABLED

CREATED_TIMESTAMP – Picks only newly created files in the file location or directory. It can also identify newly created sub-directories and the files within them when used in recursive mode (which can be enabled by selecting the “recursive” checkbox)

MODIFIED_TIMESTAMP – Picks both the newly created and modified files. The recursive mode works here as well as in the CREATED_TIMESTAMP mode.