Skip to main content

CI/CD

Data Engine is using Docker builds integrated with Turbo Monorepo. Built images are pushed to Azure Container Registry.

Dockerfiles​

To build each application, we use Turbo it's prune feature which creates an out directory containing 1) all JSON dependencies and 2) all apps/packages required for our docker container. This allows us to split dependency installation from source code installation and speed up & minimize the docker build.

Example: creating out directory for signaling-http-socket-io app​

RUN turbo prune --scope=signaling-http-socket-io --docker

After installing all the necessary dependencies, we can run turbo build which will translate to pnpm run build for the dependency chain:

RUN pnpm turbo run build --filter=signaling-http-socket-io

⚠️ It's important to set the current directory to where turbo.json is located, to ensure correct start of the application

Azure DevOps​

In order to deploy/change resources in DEV or higher environments, it's necessary to use Azure DevOps. There are existing multiple Pipelines for different purpose.

DB init​

Pipeline is connecting to the Key Vault and fetching Connection String to the Postgres DB, then installing pnpm dependencies. In the end is recreating the DB schema and seeding it with a data.

Pulumi​

Pipeline is creating necessary resources for Pulumi state files, and running pulumi up command.

Pulumi unlock​

It happens that when pulumi up finishes unexpectedly, the Pulumi state files are in the lock state, which makes impossible to run any Pulumi operation. This pipeline would run pulumi cancel in order to unlock the Pulumi state file.

Docker pipelines​

There two types of pipelines which are building docker images:

  • build-app-name : triggered on Pull Request to main branch - will build a docker image
  • build-deploy-app-name : triggered merge main branch - will build a docker image and push it to the ACR of indicated subscription

Azure DevOps Pipeline Templates​

In order to avoid boilerplate code and a lot of repetition across all the pipeline files, the templates have been implemented. You can find them in .devops/templates/ directory. All templates have tmpl prefix.

Build Deploy Templates​

Every pipeline for build and deploy to Azure has following steps:

  • login to ACR in the correct Subscription
  • install Docker
  • login to the ACR
  • execute docker build command in order to build a docker image

First 3 steps are always the same for every application (only Subscription changes depending on the environment). Therefore this step is set in tmpl-acr-docker-login.yml file.

The last step is different for each application, as it's using different docker build command, therefore there is a separate template for each, which is then reused on each of the environments.

To sum up, application templates are using the tmpl-acr-docker-login one to execute logic necessary to initialize Docker context and then they are executing a custom step for building the Docker Image.

Build Test Templates​

Those templates are used to build and test an application. The application is built by its dependency manager (for node it's npm, for rust is cargo, for java is maven). Pipelines that are using those templates are using for the Build Validation in Azure DevOps, in order to ensure and improve our CI/CD flow.

Pulumi templates​

Pulumi templates are used in a similar way as templates described above. There are 2 templates, one for Pulumi and one for Pulumi LA (Location Analytics Resource Group).