How to write an efficient GCP Workflow Pipeline with 2 Cloud Run jobs

Google Cloud Data Analytics
1

Hi everyone,

Hi would like to create a GCP Workflow pipeline with :

  • step1 : CloudRun Job number 1
  • Return error message of step1
  • If “Return error message of step1” is OK then
    • step2 : CloudRun Job number 2
    • Return error message of step2
  • If “Return error message of step2” is OK then my GCP Workflow pipeline returns OK

Currently, my “CloudRun Job number 2” does not wait the “Return error message of step1”.

  • What is the technical solution to do this :
    • Callback ?
    • Polling ?
  • How can I implement this ?

My code :

main:
    steps:
    - google-asset-inventory:
        call: http.post
        args:
            url: https://europe-west1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/paj-report-dev-prj-resti/jobs/google-asset-inventory:run
            timeout: 900.0
            auth:
                type: OAuth2
        result: statusmessage1
    - returnOutput1:
        return: ${statusmessage1}
    - dbt-bigquery-monitoring:
        call: http.post
        args:
            url: https://europe-west1-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/paj-report-dev-prj-resti/jobs/dbt-bigquery-monitoring:run
            timeout: 900.0
            auth:
                type: OAuth2
        result: statusmessage
    - returnOutput2:
        return: ${statusmessage2}
2

Hi,

Finally, problem is solved following this tutorial :

https://cloud.google.com/workflows/docs/tutorials/execute-cloud-run-jobs

Have a good work !

# This is a sample workflow to test or replace with your source code.
#
# This workflow passes the current day of the week to the Wikipedia API and
# returns a list of related Wikipedia articles.
# The current day of the week (in GMT) is retrieved from a Cloud Function
# unless you input your own search term (for example, {"searchTerm": "Monday"}).
main:
    steps:
        - init:
            assign:
                - project_id: paj-report-dev-prj-resti
                - job_name1: google-asset-inventory
                - job_name2: dbt-bigquery-monitoring
                - job_location: europe-west1
        - google-asset-inventory:
            call: googleapis.run.v1.namespaces.jobs.run
            args:
                name: ${"namespaces/" + project_id + "/jobs/" + job_name1}
                location: ${job_location}
            result: statusmessage1
        - dbt-bigquery-monitoring:
            call: googleapis.run.v1.namespaces.jobs.run
            args:
                name: ${"namespaces/" + project_id + "/jobs/" + job_name2}
                location: ${job_location}
            result: statusmessage2
        - returnOutput:
            return: ${statusmessage2}