Skip to main content
Version: Next

Development and Deployment process

Olympe, is a development platform with Saas and On-premise offers. It provides specific steps to operate the development process in order to save, build and deploy what is developed by your team.

Here is a schema that summarizes the process of deploying work (from CODE and DRAW) done in one environment to another environment:

Deployment from git repository to Olympe cloud

It is all based on docker images that can be built on 3 different basis:

  1. Nginx to deploy new code for the frontend applications
  2. NodeJS for backends (Service application)
  3. CodeAsData to deploy the compositions and development done in Draw.

The deployment steps are the following

  1. Saving Code as Data in git (Optional)
  2. Build your project
  3. Build docker images
  4. Deploy docker images

All the steps described below can be automated in a CI/CD process like Jenkins, Gitlab CI, Azure or Github Actions. Please refer to the official documentation of these tools.

Saving Code as Data in git​

After developing libraries or projects in Draw, developers need to package that work to be able to:

  1. Deploy the project and changes to another environment (e.g.: from development to production environments)
  2. Package the project with its JavaScript code into an NPM artifact to be pushed to a registry

To achieve that, it is convenient to create a snapshot of the code as data of your project and generate the patches files to be added into your git repository.

To save the code as data from Draw to patches files, use the toolkit snapshot command. That command requires a configuration file you could store in your git repository. It must contain a list of projects/folders to save, a destination directory.

res/snapshot-config.json
[
  {
    "snapshooter": {
      "name": "My Projects",
      "rootTags": ["10000000000000000000"],
      "exclude": ["10000000000000000001"],
      "outputDir": "snapshot"
    }
  }
]

This is a typical example of a snapshooter configuration file: it tells the toolkit to snapshot all the projects from Home (10000000000000000000) except what is inside the Olympe folder (10000000000000000001), and to store the patches in the snasphot folder of your project.

Once the code as data files are generated, they can be commit to the git repository like standard code source files.

Automatically commit and push snapshot to git​

The snapshooter is designed to be used in jobs or cron jobs in your infrastructure to automatically save and commit the patches to the git repository. In order to tell the snapshooter to commit and push the changes to you repository, you need to complete the documentation like this:

res/snapshot-config.json
[
  {
    "snapshooter": { ... },
    "git": {
      "repo": "https://<token>@<repository_url>.git",
      "branch": "master",
      "commitMessage": "Snapshot at {date} in {folder}"
    }
  }
]

This will tell the snapshooter to clone the specified repository on the given branch. Do the snapshot in the specified outputDir based on the repository folder, and finishes the process by committing and pushing in case of changes, with the specified commit message. The commit message can be formatted:

  • {date}: write the date with the specified format: yyyy.MM.dd hh:mm:ss.
  • {folder}: the outputDir value from the configuration.

Build your project​

info

Olympe requires NodeJS version 18 or above (download)

Like any modern Javascript project, you can use the javascript code bundler/builder of your choice (eg: webpack, esbuild, vite, etc.) to build your source code file.

tip

In the example below, the code as data will be built as a final project, but you can also build it as a library of bricks, serving as an Olympe component to be used in multiple place.

Run the following commands to build your code:

npm install
#build:draw to build your frontend
npm run build:draw
# build:node to build your backend
npm run build:node
#buildCodeAsData to build code as data
npx olympe buildCodeAsData

This will generate all the necessary files and patches with the appropriate configuration files for the project to be deployed on an environment in the following directories:

  • dist/web for the frontend.
  • dist/node for the backend.
  • dist/codeAsData for the codeAsData.

These folder will be used as base directory to build the docker images.

Build docker images​

info

Olympe requires Docker version 24 or above: Download

As described in the introduction, an Olympe environment deployed on the Olympe cloud relies on 3 types of docker images: Frontend, Backend and CodeAsData. All of them should already be present in the docker directory of your project.

the $SOURCES_PATH variable should match the correct build directory for each image as explained in the previous step.

It will contain the files built and required by the UI Applications and delivered by the web server. The olympeio/frontend-base image is based on Nginx:

FROM olympeio/frontend-base:stable
COPY --chown=nginx:root $SOURCES_PATH /usr/share/nginx/html

Deploy docker images​

In order to deploy the newly built docker images to the Olympe Cloud, you will have to do a call to the API with the correct payload.

First, contact your Olympe referent and ask for the webhook URL and authorization token. Once you have this information, you can then send a request (using curl or tools like Postman or Insomnia) with the following information:

  • URL: URL provided by Olympe
  • Method: POST
  • Auth: Bearer <token provided by Olympe>
  • Headers:
    • Content-Type: application/json
  • Body: the JSON formatted payload. Example:
{
	"backends": {
		"images": [
			"my-registry/image-for-backend-1:tag",
			"my-registry/image-for-other-backends:tag"
		],
		"deployRules": [
			{
				"source": "^my-registry\/image-for-other-backends:tag.*$",
				"target":"^(?!backend-1).*$"
			},
			{
				"source": "^my-registry\/image-for-backend-1:tag.*$",
				"target": "^backend-1.*$"
			}
		]
	},
	"codeAsData": {
		"image": "my-registry/image-for-codeasdata:tag",
	},
	"frontend": {
		"image":"my-registry/image-for-frontend:tag"
	}
}

Here is the detail of the available keys and values:

KeyRequiredDefault valueDescription
backends.imagesif backends is presentN/AArray of backend images to deploy (if applicable), following the deploy rules
backends.deployRulesif backends.images is presentN/ADeploy rules for the backend deployment
backends.deployRules[].sourceyesN/ADeploy rules for the backend deployment
backends.deployRules[].targetyesN/ADeploy rules for the backend deployment
codeAsData.imagenoN/ACode As Data image to deploy (if applicable)
codeAsData.updateModenoautoWhether to deploy code as data manually (via an approval step) or automatically. can be auto or manual
frontend.imageif frontend is presentN/ACode As Data image to deploy (if applicable)
noDowntime.promotionModeif noDowntime is presentautoWhether to promote deployments manually (via an approval step) or automatically. Only applicable if no downtime mode is enabled on the instance. can be auto or manual
skipUpdateVersionsnofalseWheter to skip automatic update of Orchestrator and Toolkit based on stable-versions. Should not be used, only when working with dedicated snapshot versions of the orchestrator or toolkit. Can be true or false
extraArgsno[]Array of additional arguments to be added to the deploy process. Currently only --force-sync can be provided, which will force the codeasdata to be synced to the orchestrator

Deploy Rules​

As you might require to build and deploy multiple backends with different code base within a single deployment, you will have to set deploy rules to define which image will be deployed to which backend. For each deploy rule you will have to set a source and target as shown in the example above.

  • source is a regular expression that must match the source image (example for my-registry/image-for-other-backends:tag)
  • target is a regular expression must match the target backend(s) name (example for backend-2 or backend-3)

You can define as many deploy rules as you want. To help you out, you can use tools like regex101.

If you provided the correct information, you should receive a success message:

{
	"message": "success"
}

You can then monitor your CI/CD.

Integrate with your CI​

There are many different ways to integrate this with your CI, as long as you call the right API with the correct payload as shown above. Below are some examples you can use for inspiration:

For this example to work, you will have to define a WEBHOOK_TOKEN variable as a secret variable in Gitlab CI/CD

variables:
  REGISTRY: <docker registry>
  IMAGE_NAME: <image-name>-$DOCKERFILE
  DOCKER_DIRECTORY: docker
  BUILT_BRANCHES_REGEX: master
  TAG_SUFFIX: $CI_COMMIT_SHORT_SHA
  DEPLOY_CODE_AS_DATA:
    value: "false"
    description: "If set to 'true', build and deploy a code as data image"

stages:
  - build-src
  - build-docker
  - deploy

.build-docker-image:
  stage: build-docker
  extends: 
   - .rules
  image:
    name: gcr.io/kaniko-project/executor:v1.20.1-debug
    entrypoint: [""]
  dependencies:
    - build
    - init-requirements
  variables:
    AWS_DEFAULT_REGION: "eu-central-1"
  script:
    - echo "{\"auths\":{\"https://index.docker.io/v1/\":{\"username\":\"$DOCKER_REGISTRY_USER\",\"password\":\"$DOCKER_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - IMAGE_NAME=$(echo $IMAGE_NAME | sed 's/professional-services-//g')
    - destination="$REGISTRY/$IMAGE_NAME:latest $REGISTRY/$IMAGE_NAME:$CI_COMMIT_REF_SLUG-$TAG_SUFFIX $REGISTRY/$IMAGE_NAME:$CI_COMMIT_REF_SLUG"
    - |
      for tag in $ADDITIONAL_TAGS; do
        destination="${destination} $REGISTRY/$IMAGE_NAME:${tag}"
      done
    - |
      if [ ! -z "$CI_COMMIT_TAG" ]; then
        destination="${destination} $REGISTRY/$IMAGE_NAME:$(echo "$CI_COMMIT_TAG" | tr _/:\  ----)"
      fi
    - kaniko_destination=$(echo $destination | sed 's/ / --destination /g')
    - echo "[INFO] building image..."
    - /kaniko/executor --context . --dockerfile $DOCKER_DIRECTORY/$DOCKERFILE.Dockerfile  --build-arg="SOURCES_PATH=$SOURCES_PATH" $ADDITIONAL_ARGS --label draw_version=$DRAW_VERSION --destination $kaniko_destination
  artifacts:
    reports:
      dotenv: build.env

.rules:
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
      when: never
    - if: $CI_COMMIT_BRANCH =~ $BUILT_BRANCHES_REGEX
      when: on_success
    - when: never

.codeAsData-rules:
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
      when: never
    - if: $DEPLOY_CODE_AS_DATA == 'true'
      when: on_success
    - if: $CI_COMMIT_BRANCH =~ $BUILT_BRANCHES_REGEX
      when: manual
      allow_failure: true # Avoid this manual job to block the rest of the pipeline
    - when: never

build:
  stage: build-src
  script:
    - apt-get install -y jq
    - npm update --cache .npm
    - sh ./bumpVersion.sh
    - npm run build:node
    - npm run build:draw
    - npx olympe buildCodeAsData
    - echo "DRAW_VERSION=$(jq -r '.version' $VERSION_FILE_PATH)" >> build.env
  variables:
    VERSION_FILE_PATH: node_modules/@olympeio/draw/package.json
  artifacts:
    expire_in: 1 day
    paths:
      - dist
    reports:
      dotenv: build.env

build-frontend-image:
  extends:
    - .build-docker-image
  after_script:
    - echo "FRONTEND_IMAGE=$REGISTRY/$IMAGE_NAME:$CI_COMMIT_REF_SLUG-$TAG_SUFFIX" >> build.env
  variables:
    DOCKERFILE: frontend
    SOURCES_PATH: dist/web

build-backend-image:
  extends:
    - .build-docker-image
  after_script:
    - echo "BACKEND_IMAGE=$REGISTRY/$IMAGE_NAME:$CI_COMMIT_REF_SLUG-$TAG_SUFFIX" >> build.env
  variables:
    DOCKERFILE: backend
    SOURCES_PATH: dist/node

build-codeAsData-image:
  extends:
    - .build-docker-image
    - .codeAsData-rules
  after_script:
    - echo "CODEASDATA_IMAGE=$REGISTRY/$IMAGE_NAME:$CI_COMMIT_REF_SLUG-$TAG_SUFFIX" >> build.env
  variables:
    DOCKERFILE: codeasdata
    SOURCES_PATH: dist/codeAsData

deploy:
  stage: deploy
  environment:
    name: $INSTANCE
  extends: 
   - .rules
  image: argoproj/argocli:latest
  variables:
    URL:
    PAYLOAD: |
      # Here put your json payload without the images reference
  before_script: |
  - json_data=$(echo $PAYLOAD)
    - |
      # If CODEASDATA_IMAGE variable exists, then it will add it to the payload
      if [ ! -z "$CODEASDATA_IMAGE" ] ; then
        json_data=$(echo $json_data | jq ".codeAsData.image = \"$CODEASDATA_IMAGE\"")
      fi
    - |
      # For any variable that contains BACKEND_IMAGE, it will add it to the payload
      for var_name in $(compgen -v | grep 'BACKEND_IMAGE'); do
        json_data=$(echo $json_data | jq ".backends.images += [\"${!var_name}\"]")
      done
    - |
      # If FRONTEND_IMAGE variable exists, then it will add it to the payload
      if [ ! -z "FRONTEND_IMAGE" ] ; then
        json_data=$(echo $json_data | jq ".frontend.image = \"$FRONTEND_IMAGE\"")
      fi
    - status_code=$(curl -s -w "%{http_code}" -o response.json -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $WEBHOOK_TOKEN" -d "$json_data" $URL)
    - |
      if [ "$status_code" -ne 200 ]; then
        echo "Error: API returned status code $status_code";
        cat response.json;
        exit 1;
      fi