For the latest version - refer to this post
Introduction
Application Integration is an Integration-Platform-as-a-Service (iPaaS) solution in Google Cloud that offers a comprehensive set of core integration tools to connect and manage the multitude of applications and data required to support various business operations.
Application Integration is a low-code/no-code platform that allows developers to create integration flows. Integration flows are sequences of tasks or activities that connect and coordinate the exchange of data between different systems. It is a common practice to develop these assets and promote them to various environments just like source code.
The best practice for CI/CD with Application Integration is to use a GCP project for each phase of SDLC. In this example, we will demonstrate how to automate the promotion of Application Integration and Integration Connectors artifacts between two SDLC phases - development (dev) and qa environments.
Prerequisites
- Application Integration and Integration Connectors are enabled and provisioned. If you have not done so already, please follow this guide
- Basic working knowledge of Application Integration and Integration Connectors.
- Github is used as the example source code repository. A Github account will be necessary to complete the instructions
Download gcloud and configure the default project
gcloud config set project $project
Principles
There are a few (opinionated) principles followed in this example:
- An integration flow developed in the development environment must be deployed unchanged in production. Treat the integration flow like one would source code.
- The portions of an integration flow that do change from one environment to another are externalized. Only those properties change between environments. For example, the HTTP URL configured for a REST Task.
- There must be traceability between what is stored in a source code repository and an integration flow version deployed in an environment.
- Maintain a single repository for each deployable service
- Automate the deployment between environments
- Test in a clone of the production environment
Steps to Automate Application Integration Deployments
In this example you will see how to store integration and connector assets to a source code repository, promote those assets from one SDLC environment to another and finally, how such deployments can be automated.
Sample Integration
We will build a sample integration with minimal complexity to demonstrate this use case. This Integration flow calls an API and publishes the response from the API to a Pub/Sub topic. This sample is meant to illustrate the use of REST and Connector tasks.
Create a topic :
gcloud pubsub topics create mytopic
Create a connection for Pub/Sub :
In the GCP console, navigate to “Integration Connectors” from the left menu and click the “+ CREATE NEW” button to create a new connector.
Please change the Service Account and Project ID appropriately. Ensure the service account has privileges to publish to the PubSub topic.
Create an Integration Flow
- Add an API Trigger and wire to the REST task
- Define a variable for the environment. It is important the name of the variable starts with “"****. Variables with the prefix "” are exported to the overrides file
- Add the variable to a http header in the REST task as shown below
- Add a Connectors Task, do not wire it yet. Select the PubSub connection created previously
- Select the connection
- Add a data mapper task and wire the tasks as shown below:
- Map the output variable of the REST task to the input variable of the Connector task
Test the integration to ensure the flow works successfully. In this example, the following details may change between environments:
- The URL used in the REST task
- The project where the PubSub connection exists
- The topic name
- The variable content
Prepare Github
Create a new repository. It is recommended that every integration flow uses a different repository.
mkdir app-integration-demo && cd app-integration-demo
git init
git checkout -b dev
In this step, the dev branch is created. The recommended folder structure for Integration and Connectors are as follows:
├── connectors
│ └── <connector-name>.json #there is one file per connector. the connector name is the file name.
├── authconfig
│ └── <authconfig-name>.json #there is one file per authconfig. the authconfig name is the file name.
├── overrides
│ └── overrides.json #always name this overrides.json. there is only one file in this folder
└── src
└── <integration-name>.json #there only one file in the folder. the integration name is the file name.
Introduction to integrationcli
integrationcli is a tool that lets you interact or manage (create, delete, get, list integrations and connections) with Application Integration, Integration Connectors or Apigee Integration/Connector APIs. This example uses this tool to automate deployments. You can see other examples and options here.
Install the CLI with the following command:
curl -L https://raw.githubusercontent.com/GoogleCloudPlatform/application-integration-management-toolkit/main/downloadLatest.sh | sh -
Set integrationcli preferences
token=$(gcloud auth print-access-token)
project=<set DEV project here>
region=<set DEV region here>
integrationcli prefs set -p $project -r $region -t $token
Create a scaffold for the Integration
integrationcli integrations scaffold -n sample -s 1 -f app-integration-demo
Where “-n sample” is the name of the integration, “-s 1” is the snapshot number and “-f app-integration-demo” is the folder to generate the artifacts. This command downloads the integration, connections used by the integration, authconfigs, sfdc instances & channels in the folder structure mentioned previously.
Overrides file
Integration flows contain values that can change between environments. For example,
- The Connector task needs to be changed as you migrate from one project to another
- The REST task & Cloud Function tasks may have different values between environments
integrationcli makes it easy to generate an overrides file for values that typically change between environments.
The file should look like this (./app-integration-demo/overrides/overrides.json )
{
"task_overrides": [
{
"task": "GenericRestV2Task",
"taskId": "1",
"parameters": {
"url": {
"key": "url",
"value": {
"stringValue": "https://httpbin.org/get"
}
}
}
}
],
"connection_overrides": [
{
"taskId": "2",
"task": "GenericConnectorTask",
"parameters": {
"connectionName": "pubsub"
}
}
],
"param_overrides": [
{
"key": "_env",
"defaultValue": {
"stringValue": "dev"
}
}
]
}
The items marked in red are the values that can be overridden when the integration is promoted. Other overrides not automatically captured from the CLI may be added to the file at this time. Examples like retries, default values etc. may be added.
Check in all the content to the dev branch.
git add --all
git commit -m 'first draft'
git push
The final folder structure will look like this:
├── connectors
│ └── pubsub.json
├── overrides
│ └── overrides.json
└── src
└── sample.json
Create the QA branch
This branch will contain the overrides needed for the QA environment.
git checkout -b qa
Make changes to the overrides.json and pubsub.json files that are specific to the qa environment. In this example, we will change the variable to qa.
"param_overrides": [
{
"key": "_env",
"defaultValue": {
"stringValue": "qa"
}
}
]
Save changes to the QA branch
git add --all
git commit -m 'add overrides to qa'
git push
Manual deployments to QA
The artifacts in the qa branch can now be deployed to the QA environment (GCP project). Set integrationcli preferences for the QA environment:
token=$(gcloud auth print-access-token)
project=<set QA project here>
region=<set QA region here>
integrationcli prefs set -p $project -r $region -t $token
Create the connection
name=$(ls -A connectors | sed 's/\.json'$//)
integrationcli connectors create -n $name -f ./connectors/$name.json --wait=true
Publish the Integration with the overrides
name=$(ls -A src | sed 's/\.json'$//)
integrationcli integrations create -n $name -f ./src/$name.json -o ./overrides/overrides.json
Manually Apply Changes
integrationcli can be used to apply changes generated by scaffold.
integrationcli integrations apply -f .
Integration with Cloud Build
To promote changes into upper environments like QA, Stage, you would need to set up DevOps pipelines for example, Jenkins, Gitlab, or CloudBuild, etc. For this demo purpose, we are going to use Cloud Build, you can learn more about cloud build here.
This example uses a custom cloud builder called integrationcli-builder:
us-docker.pkg.dev/appintegration-toolkit/images/integrationcli-builder:latest
Please follow the instructions here to set up Cloud Build to deploy Application Integration and Integration Connectors.
The cloudbuild file has three parts:
- Loop through the connectors folder and create any connections
- Loop through the authconfigs folder and create any authconfigs (the example in this tutorial does not have any)
- Create the Integration and Publish the revision
There are few options in the cloud build YAML to consider:
substitutions:
_CREATE_SECRET: "false" # do not create Secret Manager secrets
_GRANT_PERMISSIONS: "true" # grant service account permissions
_ENCRYPTED: "false" # use encryption
_DEFAULT_SA: "false" # use default service account
_SERVICE_ACCOUNT_NAME: # the service account for connectors
_KMS_RING_NAME: # name of the KMS key ring
_KMS_KEY_NAME: # the name of the cloud kms key
Please change them to appropriate values esp. if using Cloud KMS for encryption.
Manual triggering of builds
gcloud builds submit --config=cloudbuild.yaml --project=$project --region=$region
NOTE: The integration revision is labeled (userLabel field) with the SHORT_SHA of the commit code in github. This is to provide traceability between what is stored in the source code repository and the published revision.
Automate deployments with triggers
Setup a Cloud Build Trigger as shown in the screenshot below
This will trigger deployments to the qa environment automatically as soon as changes are merged with the qa branch.
Similar to QA, you can create a “prod” branch and merge the qa branch to promote it using a pipeline to your “prod” environment.
Merge changes between dev and qa
Create a new branch from prod, for each feature in dev (ex: dev-feature1). Download artifacts from dev environment to feature branch. These may include:
- New and/or modified connections
- New and/or modified authconfigs
- Modified integration version
- and finally, a new/modified overrides file created for dev-feature1
Create a qa feature branch from the dev feature branch (ex: qa-feature1). Modify the overrides files for qa-feature1. If you have automated deployments through Cloud Build triggers, then the changes are automatically published to the QA environment.
Advanced Options
When developing integrations, there may be a need to store sensitive information. For example,
- An authconfig may contains api keys or passwords
- Connections to databases, FTP servers etc. require username and password
Read more about advanced options here to use Cloud KMS to encrypt sensitive information.
Assets
All the assets used in this tutorial are found here.
Note : Special Thanks to Nandan Sridhar for co-authoring the article and @ssvaidyanathan for his feedback.
: you can also find here same kind of implementations but using different pipeline solutions (currently: GitLab-CI and Azure-Pipeline, more to come) :