Skip to main content
Version: 2.8

Development and Deployment process

Olympe, is a development platform with Saas and On-premise offers. It provides specific steps to operate the development process in order to save, build and deploy what is developed by your team.

Here is a schema that summarizes the process of deploying work (from CODE and DRAW) done in one environment to another environment:

Deployment from git repository to Olympe cloud

It is all based on docker images that can be built on 3 different basis:

  1. Nginx to deploy new code for the frontend applications
  2. NodeJS for backends (Service application)
  3. CodeAsData to deploy the compositions and development done in Draw, the code as data.

Once the docker images are built, the docker registry can call appropriate web hooks in order to run a deployment workflow on the Olympe cloud side.

tip

When using CI/CD pipelines on git repositories (eg: Github actions, Gitlab CI, Azure DevOps pipelines, etc.), it can be important to avoid automatic build of code as data image. This can cause a brief downtime of the composition database while the update process is in progress.

Code as data

When developing applications on the Olympe platform, developers combine and compose pieces of code, known as coded bricks, using DRAW to create processes and applications. The system generates data in a graph database, referred to as the composition database. While applications composed of brick assemblies offer many advantages over traditional source code files, they require tools to extract these assemblies, create library packages, and deploy them to other environments.

The term Code as Data designates everything that developers assemble in Draw and store in the database. This is code, but stored like data instead of being stored in text files. Hence, that code must be considered as part of the source code of projects built on Olympe.

The files that are used to store the code as data are json files and named patches.

Olympe Toolkit

The Olympe toolkit (@olympeio/toolkit npm package) is the tool blended to handle operations with Olympe environments, in particular operations related to the code as data.

The toolkit is generally used as npm devDependency in the Olympe projects to build the appropriate code as data files (also named patches) and install or deploy them to your environment. The toolkit is also available in DockerHub as docker image to be used directly in jobs on your kubernetes cloud (for Saas or on-premise Olympe environments).

Execute toolkit commands

The toolkit is a CLI tool written in JavaScript that can either be installed globally or locally in your project, or be deployed on a docker/kubernetes environment.

Use the help command to know about existing commands:

To install the toolkit locally in your project, execute the following command in a terminal:

npm install --save-dev @olympeio/toolkit

or globally on your computer, available as a global binary:

npm install --global @olympeio/toolkit

To execute a command, use the following syntax:

npx olympe <command> [--<param> <value> ...]

For instance, use this to list the available commands:

npx olympe help

The list of commands and parameters can be found in the reference section.

Saving Code as Data in git

After developing libraries or projects in Draw, developers need to package that work to be able to:

  1. Deploy the project and changes to another environment (e.g.: from development to production environments)
  2. Package the project with its JavaScript code into an NPM artifact to be pushed to a registry

To achieve that, it is convenient to create a snapshot of the code as data of your project and generate the patches files to be added into your git repository.

To save the code as data from Draw to patches files, use the toolkit snapshot command. That command requires a configuration file you could store in your git repository. It must contain a list of projects/folders to save, a destination directory.

res/snapshot-config.json
[
{
"snapshooter": {
"name": "My Projects",
"rootTags": ["10000000000000000000"],
"exclude": ["10000000000000000001"],
"outputDir": "snapshot"
}
}
]

This is a typical example of a snapshooter configuration file: it tells the toolkit to snapshot all the projects from Home (10000000000000000000) except what is inside the Olympe folder (10000000000000000001), and to store the patches in the snasphot folder of your project.

Once the code as data files are generated, they can be commit to the git repository like standard code source files.

Automatically commit and push snapshot to git

The snapshooter is designed to be used in jobs or cron jobs in your infrastructure to automatically save and commit the patches to the git repository. In order to tell the snapshooter to commit and push the changes to you repository, you need to complete the documentation like this:

res/snapshot-config.json
[
{
"snapshooter": { ... },
"git": {
"repo": "https://<token>@<repository_url>.git",
"branch": "master",
"commitMessage": "Snapshot at {date} in {folder}"
}
}
]

This will tell the snapshooter to clone the specified repository on the given branch. Do the snapshot in the specified outputDir based on the repository folder, and finishes the process by committing and pushing in case of changes, with the specified commit message. The commit message can be formatted:

  • {date}: write the date with the specified format: yyyy.MM.dd hh:mm:ss.
  • {folder}: the outputDir value from the configuration.

Build your project to an NPM package

info

Olympe requires NodeJS version 18 or above (download)

Like any modern Javascript project, you can use the javascript code bundler/builder of your choice (eg: webpack, esbuild, vite, etc.) to build your source code file. There are two possibilities, either your project is aimed to be a library of bricks, serving as an Olympe component to be used in multiple places, or it is a final project that uses libraries and need to be deployed to an environment.

In general, we use two configurations with different entry points to transpile the bricks for 2 targets (browsers and NodeJS). When building a project, the bundler will generate two folders to be used when building the docker images for **frontend** and **backend** web hooks.

For the code as data, build the snapshot of your project aggregated with the code as data of all the npm dependencies. Use the buildCodeAsData command from the toolkit.

npx olympe buildCodeAsData

This is equivalent to use the following command:

npw olympe buildCodeAsData --codeAsData.mode all

This will generate all the patches with the appropriate configuration files for the project to be deployed on an environment in the dist/codeAsData directory.

`CodeAsData` folder

That folder must be used as base directory to build the code as data docker image, to deploy the code as data to your environment via the web hook.

In the end, you typically end up with the following directories structure:

`dist` folder

Build docker images to be deployed

info

Olympe requires Docker version 24 or above: Download

As described in the introduction, an Olympe environment deployed on the Olympe cloud relies on 3 types of docker images:

It will contain the files built and required by the UI Applications and delivered by the web server. That image typically is based on Nginx and its dockerfile is:
FROM olympeio/frontend-base:stable
COPY --chown=nginx:root $SOURCES_PATH /usr/share/nginx/html

Configure deployment triggers

In order to deploy the newly built docker images to the Olympe Cloud, you must now configure your container registry to trigger the appropriate webhook. The Olympe Cloud is compatible with any container registry that accepts an authentication using standard token in authorization header of the request.

In order to configure your registry, first contact your Olympe referent and ask for the webhook URL and authorization token. The URL should use the following syntax:

https://<client-name>.webhook.olympe.io/<project-name>/<instance-name>

Your Olympe referent will require some information to configure the webhook and deployment workflow:

  1. The repository, name and tag of the images to be pulled, that should be pulled from the registry. For instance, on Github registry it could look like ghcr.io/<registry name>:<project-name>-frontend:master-42bd534f.
    1. In the example, the suffix frontend is used in the image name. There could be any kind of name. By default, we recommend to use one of these suffixes: frontend, backend or codeasdata for the three types of image to deploy. It is simpler to identify where to deploy it.
    2. In the example, the tag master-42bd534f is used. It corresponds to the branch name followed by the commit id on which the image has been built. We recommend to use such a pattern to tag images to make it simple to know from what commit and branch the image has been built.
      tip

      When a same project has multiple environments (e.g.: codev, pre-production and production), this type of image tag can be used with a regular expression to specify on what environment to deploy an image. For instance, images with master-* tag would be configured to only deploy to production environment and the ones with develop-* tag would be deployed on codev.

  2. Provide the authentication credentials for the system to be able to pull the image from your container registry:
     {
    "auths": {
    "<url>": {
    "username": "<username>",
    "password": "<password>",
    "auth": "<auth>"
    }
    }
    }
    • the url field is the repository url
    • the username and password fields are the user's credentials
    • auth is computed from username and password of the user used to pull the image. It corresponds to the base64 representation of username:password use this in your terminal to get it.
    echo -n 'username:password' | base64`
  3. (Optional): Provide the patterns and regular expressions you need to apply to trigger the deployment pipelines.

When this is done, try to build and publish the images of your project and validate that the environment gets updated using your monitoring dashboard (see: managing environments)

Summary - Deployment steps

  1. Build a project/library in Draw in environment A
  2. Snapshot from environment A to json files (save in git)
  3. Build and gather patches to build and publish npm package / docker image
  4. Deploy the patches to the orchestrator of the environment B
  5. Install/Update the graph database with patches to setup the environment B

Toolkit reference

To run a command using the toolkit locally installed as npm dev-dependency use this in a terminal:

npx olympe <command> [--<parameter name> <parameter value>, ...]

For each command, use the -h, --help parameter to see the detailed list of parameter to use in that command.

CommandTypical useDescription
helpN/ADisplay the list of available commands with the toolkit, as well as the associated parameters
snapshotDocker: scheduled CI/CDDownload and save the code as data into patches (json files) from a deployed environment
buildCodeAsDataNPM: within CI/CDFrom the project snapshot files and its dependencies, build the dist/codeAsData folder containing all the patches required to build/deploy the project to an environment
updateDocker: Olympe cloud jobUpdate the graph database containing the code as data of an environment to be aligned with the patches deployed to the orchestrator.
installDocker: Olympe cloud jobClean and Install the content of patches deployed to the orchestrator. /!\ This command starts by clearing the whole composition database, so it must be used with caution.
snapshotUsersSub-command of installSave users and their Draw permissions of an environment to patches (mainly as sub-command in process of install)
restoreUsersSub-command of installRestore patches generated with command snapshotUsers to the database (mainly as sub-command in process of install)
snapshotBusinessDataSub-command of installSave business data (business objects visible in Runtime Data Set of projects), stored in the "Embedded graph database" of an environment to patches (mainly as sub-command in process of install)
restoreBusinessDataSub-command of installRestore patches generated with command snapshotBusinessData to the database (mainly as sub-command in process of install)
dependenciesNPM: manuallyList all npm dependencies from node_modules folder which contain some code as data
updateUserDocker: Olympe cloud jobUpdate the username and password of the admin user
maintenanceDocker: Olympe cloud jobActivate/Deactivate the maintenance mode of the orchestrator. When activated, ensures that all connected clients to the orchestrator are disconnected.
getUpgradeConfigDocker: during deploymentGenerate the configuration for the upgrade tool to apply upgrade script on the composition database. Used by the code as data deployment workflow to automatically upgrade the language version from one major version to another.
startGCDocker: Olympe cloud jobRequest the orchestrator to garbage collect useless files in file and backup services
statsDBReturn information about the composition database

Toolkit command parameters

Shell parameterEnvironment variableTypeDefault valueDescription
--logLevel, -lLOG_LEVELstringinfologging level, one of debug, verbose, info, warn, or error. (Eg. olympe install --logLevel warn)
--help, -hbooleanfalsedisplay help for the current command
--saveLogsSAVE_LOGSbooleanfalsesave logs to a file under directory logs/
--codeAsData.workingDirstring./dist/codeAsDatarelative (to the project's root) path to the temporary directory used to store generated code as data
--codeAsData.modestringallspecify if the code as data to process is aimed to be used in a library package (only the code as data of the current project) or in a project to be deployed (including all the dependencies). Accepted values dependencies, project, all
--snapshot.excludeUsersSNAPSHOTER_EXCLUDE_USERSbooleanfalsewhether to exclude users from dump of business data
--snapshot.dumpBinariesbooleanfalsewhether to dump binary files while dumping business data
--snapshot.dataRootTagstring10000000000000000000what root tag to use to dump business data
--snapshot.userDataDirSNAPSHOT_USER_DATA_FOLDERstringuserssnapshot directory sub-folder to dump users
--snapshot.businessDataDirSNAPSHOT_BUSINESS_DATA_FOLDERstringdatasnapshot directory sub-folder to dump business data
--snapshot.repositoryDirstring/tmp/repositoriesdefine directory where the snapshooter can clone repositories if config contains git configuration
--snapshot.commitMessageSNAPSHOT_COMMIT_MESSAGEstringSnapshot at {date} in {folder}define a commit message for snapshot
--snapshot.dirstring./snapshotrelative (to the project's root) path to the directory containing the project's snapshots
--versionFileNamestringversion.jsonName of the file containing the project's Draw/Framework versions
--runningInDockerbooleanfalseglobal option to set whether the Olympe Toolkit is running inside a docker image. Primarily used when calling the install command on local environment to determine whether to copy the patches inside the docker container or not .
--jsonConfigjsonset the commands configuration via a JSON config
--downloadFolderstring/tmp/toolkitpath to the temporary download folder used during update command.
--user.tagDRAW_USERTAGstring014831d95fd7d12b8568tag of the user to update (command updateUser)
--user.nameDRAW_USERNAMEstringadminnew username to set for the specified user.tag (command updateUser)
--user.passwordDRAW_PASSWORDstringadminnew password to set for the specified user.tag (command updateUser)
--import.debugLOAD_PATCHES_DEBUGbooleanfalsewhether the orchestrator will import patches in debug mode
--import.ignoreRelationFailIGNORE_RELATION_FAILbooleanfalsewhen importing patches, ignore relations that cannot be created because of the nodes missing
--import.backupUsersBACKUP_USERSbooleantruewhether to backup users before resetting the database when using install command
--import.backupBusinessDataBACKUP_BUSINESS_DATAbooleantruewhether to backup business data before resetting the database when using install command
--import.timeoutRESET_DB_TIMEOUTnumber3600000the timeout for importing all patches when using install command, in ms.
--orchestrator.hostORCHESTRATOR_HOSTstringlocalhostorchestrator http host
--orchestrator.portORCHESTRATOR_SERVICE_PORTnumber8080orchestrator http port
--orchestrator.userNameDRAW_USERNAMEstringadminuser for Orchestrator connection
--orchestrator.passwordDRAW_PASSWORDstringadminuser password for Orchestrator connection
--orchestrator.importOrderFilestringimport_order.jsonName of the file that specifies the import order of modules
--orchestrator.containerUserstringvertxuser which is used to copy patches inside of the orchestrator container
--orchestrator.codeAsDataDirstring/patchesthe path to copy code as data folder in the orchestrator container. Used exclusively when the toolkit is not running in a container.
--bus.protocolstringamqpprotocol used for bus communication
--bus.hostnameRABBITMQ_HOSTstringlocalhosthost for bus communication
--bus.portRABBITMQ_PORTnumber5672port for bus communication
--bus.usernameRABBITMQ_ORCHESTRATOR_USERstringguestusername used to connect to the bus
--bus.passwordRABBITMQ_ORCHESTRATOR_PASSWORDstringguestuser password used to connect to the bus
--bus.vhostRABBITMQ_ORCHESTRATOR_VHOSTstring/virtual host used for any communication sent on the bus
--bus.commandsQueueRABBITMQ_ORCHESTRATOR_COMMAND_QUEUEstringCOMMAND_QUEUErabbitmq queue used for commands sent to the orchestratr
--bus.commandsExchangeRABBITMQ_ORCHESTRATOR_COMMAND_EXCHANGEstring""rabbitmq exchange used for commands sent to the orchestrator
--bus.deploymentQueueRABBITMQ_ORCHESTRATOR_DEPLOYMENT_QUEUEstringSERVICE_DEPLOYMENTqueue used to send Orchestrator deployment commands
--bus.deploymentExchangeRABBITMQ_ORCHESTRATOR_DEPLOYMENT_EXCHANGEstringORCHESTRATOR_BROADCAST_EXCHANGEexchange for Orchestrator deployment commands
--bus.commandTimeoutRABBITMQ_ORCHESTRATOR_COMMAND_TIMEOUTnumber600000timeout applied for standard commands sent to the orchestrator (in ms)
--update.useChangeLogUPDATE_USE_CHANGELOGbooleantruewhether to use a changelog file to accelerate the database update process (can be negated using --no-update.useChangeLog)
--update.changeLogNamestringchange_log.txtname of the file giving the patches change log
--update.excludeTagsUPDATE_EXCLUDED_TAGSstring[]list of tags of projects/folders to exclude from the check of the update command. In a development environment, the appropriate value is to put all the tags of projects that are snapshot in that environment.
--update.includeTagsUPDATE_INCLUDED_TAGSstring[]list of tags of projects/folders should be included from the check of the update command, even though one of their ancestor in the Draw hierarchy has been excluded using the --update.excludeTags option.