I made a pipeline that connects to my workday with an integration connector as the source and created another connector for GCS as the destination. also created application integration under that added workday connector after data mapping rules and added a GCS connector at the end. I published this integration and created a while loop task to trigger the above integration and perform a full load of data to my GCS bucket.
*** This application integration works well and I’m able to get full data but I want to add incremental logic to the while loop to get daily updated data.***
Please reach out to me if any suggestions are any referred documents to implement this incremental logic.
Hope this finds you well, thank you for your question. I agree that a way to only process new data rather than a full load is a better pattern in some cases. One option that you can try is the high-water mark approach.
The High-Water Mark Strategy
This approach, also known as watermarking or state management, involves three key steps:
Read State: At the beginning of a run, retrieve the timestamp of the last record you successfully processed. This is your “high-water mark.”
Filter Source: In your Workday query, ask for records that have been created or updated after that high-water mark timestamp.
Write State: After successfully writing the new data to GCS, save the timestamp of the current run as the new high-water mark for the next execution.
This ensures that if a run fails, the high-water mark isn’t updated, and the next run will re-process the same data, preventing data loss.
The new integration logic should look like this:
Trigger (Scheduler) -> Read last_run_timestamp.txt from GCS -> Get Current Time -> Query Workday (using last timestamp as filter) -> Write Data to GCS Destination -> (On Success Only) -> Write Current Time to last_run_timestamp.txt -> End