Skip to main content

Integrate Web Modeler into CI/CD

note

Some features in this guide are only available starting in 8.3, however the Web Modeler API is already available.

IntermediateTime estimate: 1 hour

Web Modeler serves as a powerful tool for the development and deployment of processes and process applications. While Web Modeler simplifies one-click deployment for development, professional teams often rely on continuous integration and continuous deployment (CI/CD) pipelines for automated production deployments. The Web Modeler API facilitates integration of Web Modeler into these pipelines, aligning with team practices and organizational process governance.

Continuous integration and deployment are pivotal for rapid and reliable software development, testing, and delivery. These practices automate the building, testing, and deployment processes, leading to shorter development cycles, enhanced collaboration, and higher-quality releases.

Integrating Web Modeler into your CI/CD pipelines can significantly enhance process application development and deployment workflows. By automating process application deployment, changes can be promptly and accurately reflected in the production environment. This agility empowers teams to swiftly respond to evolving business needs, fostering a flexible and adaptable process orchestration approach.

Prerequisites​

Each pipeline is unique. The Web Modeler API offers flexibility to tailor integrations according to your pipelines. To get started, there are a few prerequisites based on your setup:

Setup​

While a pipeline for process application integration and deployment resembles general software CI/CD pipelines, key distinctions exist. Consider the following:

  • Web Modeler uses milestones to indicate specific process states, such as readiness for developer handover, review, or deployment.
  • A process application comprises main processes and diverse resources, such as subprocesses, forms, DMN decision models, Connectors, job workers, and orchestrated services. Some applications bundle these resources, while others focus on a single process for deployment.
  • Process reviews differ from code reviews, occurring on visual diagrams rather than XML.

Sample CI/CD setup with Web Modeler

Obtain API clients and tokens​

Before getting started, obtain API clients and tokens for integrating Web Modeler and accessing the process engine via API:

Disable manual deployments from Web Modeler​

To enforce pipeline-driven deployments to your environments, consider disabling manual deployments.

Disable manual deployments for any user by configuring environment variables ZEEBE_BPMN_DEPLOYMENT_ENABLED and ZEEBE_DMN_DEPLOYMENT_ENABLED as documented here.

Triggering CI/CD​

You need triggers to initiate the pipeline for files or projects. Choose between manual pipeline start or automatic background triggers based on events. Common approaches include:

  • Initiating the pipeline manually from your CI/CD tool/platform by uploading the file intended for deployment.
  • Starting the CI pipeline by creating a pull/merge request in the version control system.
  • Triggering pipelines by listening to milestones with certain characteristics.

Sync files with version control​

Synchronize files between Web Modeler and version control systems (VCS) and vice versa. Manage both files and projects by using a complete set of CRUD (create, read, update, delete) operations provided by the Web Modeler API. By syncing files from Web Modeler to your VCS, you benefit from full file ownership and avoid duplicated data housekeeping.

For automatic file synchronization, consider maintaining a secondary system of record for mapping Web Modeler projects to VCS repositories. This system also monitors the project-to-repository mapping and update timestamps.

To listen to changes, starting from 8.3 Web Modeler offers a polling approach that compares the update dates with the last sync dates recorded, see the 8.3 documentation.

Real-time synchronization isn't always what you need. Consider Web Modeler as a local repository, and update your remote repository only after files are committed and pushed. This aligns with the concept of milestones.

Listening to milestone creation​

A milestone reflects a state of a file in Web Modeler with a certain level of qualification, such as being ready for deployment. You can use this property to trigger deployments when a certain milestone is created.

Starting from 8.3, Web Modeler offers a polling approach to listen to milestone creation, see the 8.3 documentation.

To retrieve the content of this particular milestone, use the GET /api/beta/milestones/:id endpoint. To obtain the latest edit state of the file, use the GET /api/beta/files/:id endpoint. This endpoint also provides the projectId necessary for the POST /api/beta/projects/search endpoint offered from Web Modeler 8.3 on if you want to push the full project via the pipeline.

Progress is underway to introduce webhook registration or event subscription for milestone creation monitoring.

Combine these two approaches and listen to milestones to sync files to your version control, create a pull/merge request, and trigger pipelines.

Pipeline stages​

The following examples illustrate setting up build, test, review, and publish stages within a pipeline.

Build stage​

While there is no distinct concept for a build package in Camunda 8, artifact structuring depends on your overall software architecture. The build stage should primarily focus on acquiring dependencies and deploying them to a preview environment.

Set up preview environments​

Offering an automatically testable and review-ready process preview mandates a dedicated preview cluster. Numerous options exist, varying with software development lifecycle design, preferences, and Camunda 8 deployment type (SaaS, self-managed, or hybrid). This guide proposes a setup with lightweight local self-managed preview clusters (or embedded engines) and full-fledged staging and production clusters (Self-Managed or SaaS).

For local preview environments, you can deploy a comprehensive Zeebe cluster including Operate and Tasklist. Options include using docker-compose or Kubernetes via Helm. All necessary endpoints and UIs are available for thorough process/application testing. Opt for a cluster version aligned with your production cluster to ensure process compatibility.

Using embedded Zeebe engines​

If you don't need to spawn all apps such as Operate or Tasklist, you can use the lightweight embedded Zeebe engine, which is a community-maintained project, to set up a cost-effective solution with an in-memory database. Together with the Zeebe Hazelcast exporter (community-maintained as well), you can consume data generated from your process for reporting or testing.

In the build stage, deploy your process or project to a cluster or embedded engine. Post-pipeline completion, such as deployment to staging or production, preview environments can be discarded.

tip

For GitLab users, consider using GitLab Review Apps to provide preview environments.

Deploy resources using the zbctl CLI in this pipeline step, compatible with both SaaS and Self-Managed clusters. Alternately, utilize the Java or Go client library or any community-built alternatives.

Feature branches and Web Modeler installations

To maintain a single source of truth, avoid multiple Web Modeler instances for different feature branches. Instead, maintain a single Web Modeler installation for all environments, utilizing milestones to signify versioning and pipeline stages. Feature branches can be managed by cloning and merging files or projects, ensuring synchronization using VCS.

Automate deployment of linked resources/dependencies​

Pipeline-driven deployment can be executed for a single file or an entire project. A separate system of record, maintained outside Web Modeler, can handle finer-grained dependency management. Fetch the full project for a file using the GET /api/beta/files/:id endpoint to acquire the project's projectId. Subsequently, use the POST /api/beta/files/search offered starting from Web Modeler 8.3 on to retrieve all project files, see the 8.3 documentation.

To retrieve the actual file content, iterate over the response and fetch it via GET /api/beta/files/:id. Parse the XML of the diagram for the zeebe:taskDefinition tag to retrieve job worker types. Utilizing a job worker registry mapping, deploy these workers along with the process if required.

If you are running Connectors in your process or application, you need to deploy the runtimes as well. Parse the process XML for zeebe:taskDefinition bindings to identify the necessary runtimes (in addition to job workers). To learn how to deploy Connector runtimes, read more here for Self-Managed, or here for SaaS.

Deploy resources in this pipeline step using the zbctl CLI, compatible with both SaaS and Self-Managed clusters. Alternatively, utilize the Java or Go client library or any community-built alternatives.

Add environment variables via secrets​

If you are running Connectors, you need to provide environment variables, such as service endpoints and API keys, for your preview environment. You can manage these via secrets. Read the Connectors configuration documentation to learn how to set these up in SaaS or Self-Managed.

Test stage​

Keep strict quality standards for your processes with automatic testing and reporting.

Lint your diagrams​

Add a step to your pipeline for automatic process verification using the bpmnlint and dmnlint libraries. Maintained by the bpmn-io team at Camunda, these open source libraries provide a default set of verification rules, as well as the option to add custom rules. They provide reporting capabilities to report back when the verification fails. These are the same libraries Web Modeler uses to verify diagrams during modeling.

You could even report the wrong diagram patterns together with examples to resolve it using this extension.

Unit and integration tests​

For unit tests, select a test framework suitable for your environment. If working with Java, the zeebe-process-test library is an excellent option. Alternatively, employ the Java client with JUnit for testing your BPMN and DMN diagrams in dev or preview environments. Similar testing can be performed using community-built clients in Node.js, Python, or Go.

Review stage​

During the review stage, stakeholders and team members access the built and tested environment for review purposes. Both the deployed process/application and a visual diagram diff are available for examination.

Use milestones to indicate a state for review. Use the POST /api/beta/milestones endpoint to create a new milestone, and provide a description to reflect the state of this milestone using the name property. The current content of the file is copied over on milestone creation.

While it is possible to do a diff of your diagrams by comparing the XML in your VCS system, this is often not very convenient, and lacks insight into process flow changes. This approach is also less effective when involving business stakeholders in the review.

The Web Modeler API addresses this by providing an endpoint to generate visual diff links for milestones. Utilize the GET /api/beta/milestones/compare/{milestone1Id}...{milestone2Id} endpoint to compare two milestones. Obtain IDs for the latest milestones via the POST /api/beta/milestones/search endpoint available from Web Modeler 8.3, utilizing the fileId filter to identify the file to review. The resulting URL leads to a visual diff page similar to this:

Visual diff of two milestones

Example review flow​

The following process diagram demonstrates an example flow of how to run a preview using milestones and a diff link in GitHub:

Review a running process application​

If deployed in a review environment, processes/applications can be shared with peers for interactive review. For comprehensive review, full clusters inclusive of Operate and Tasklist can be used for process execution. This closely simulates the final experience. To integrate the preview environment with custom applications, leverage the Operate and Tasklist APIs and deploy them within the review environment.

In case you use an embedded Zeebe engine, or want to provide a lightweight, focused review experience, you can use Zeebe Simple Monitor, which is a community-maintained Web App similar to the Play mode in Web Modeler. Deploying Zeebe SimpleMonitor allows for thorough process testing and review.

Publish stage​

Push approved changes to staging or production by deploying them to the respective clusters. You can use the zbctl CLI to deploy via your pipeline, which works both for a SaaS or Self-Managed cluster. Deployments work slightly different on SaaS and Self-Managed, since there are differences in the cluster connection. Read more about deployments here.

Define resource authorizations​

For clusters with resource authorizations enabled (via the RESOURCE_PERMISSIONS_ENABLED feature flag), use the Identity API to assign the necessary authorizations through the pipeline. This step ensures appropriate accessibility for process/application stakeholders or updating existing authorizations.

Monitoring and error handling​

As with any CI/CD integration, it's crucial to set up monitoring and error handling mechanisms. These can include:

  • Monitoring the CI/CD pipeline execution for errors and failures.
  • Using Operate to catch incidents and send alerts to the pipeline in the test stage.
  • Sending notifications or alerts in case of deployment issues in both the build and publish stages.
  • Implementing rollback mechanisms in case a faulty BPMN diagram gets deployed.

FAQ​

Can I do blue-green deployments on Camunda 8?​

Blue-green deployments are possible with Camunda 8 with limitations. While switching clusters is quick for new process instances, audit logs and existing process instances remain tied to the previous cluster. Consider exporting audit logs from Elasticsearch or OpenSearch to your own streams if needed. If you don't have to migrate running process instances, keeping them running on the previous cluster alongside new instances on the new cluster is also an option.

Can I implement blue-green deployments with Camunda 8 SaaS?​

While blue-green deployments are more straightforward with Self-Managed setups, you can implement similar deployment strategies with Camunda 8 SaaS. Keep in mind the limitations and differences between clusters when planning your deployment approach.

How can I prevent manual deployments from Web Modeler?​

To enforce CI/CD pipelines and restrict manual deployments, you can disable manual deployments. For Self-Managed setups, set environment variables ZEEBE_BPMN_DEPLOYMENT_ENABLED and ZEEBE_DMN_DEPLOYMENT_ENABLED. In Camunda 8 SaaS, only the Developer role allows deployments from Web Modeler. Assigning any other role effectively removes deployment privileges.

How can I sync files between Web Modeler and version control?​

Use the Web Modeler API's CRUD operations to sync files between Web Modeler and your version control system. Consider maintaining a second system of record to map Web Modeler projects to VCS repositories and track sync/update dates.

What is the purpose of the build stage in my pipeline?​

The build stage focuses on preparing dependencies and deploying them to a preview environment. This environment provides a preview of your process that can be tested and reviewed by team members.

Can I lint my process diagrams for verification?​

Yes, you can use the bpmnlint and dmnlint libraries to automatically verify your process diagrams against predefined rules. These libraries provide reporting capabilities to identify and fix issues during the build stage.

How can I perform unit and integration tests on my processes?​

You can use the zeebe-process-test library for Java-based unit tests or community-built clients for other programming languages. These libraries allow you to execute your BPMN and DMN diagrams with assertions in your development or preview environments.

How do I provide environment variables to Connectors in preview environments?​

You can manage environment variables for Connectors using secrets. This can be set up in both Camunda 8 SaaS and Self-Managed. Refer to the Connectors configuration documentation for details.

How can I monitor and handle errors in my CI/CD pipeline?​

Implement monitoring mechanisms in your CI/CD pipeline to catch errors and failures during the deployment process. Additionally, consider implementing rollback mechanisms in case a faulty BPMN diagram is deployed.

Additional resources and next steps​