Skip to main content

Importer and archiver

Operate consists of three modules:

  • Webapp: Contains the UI and operation executor functionality.
  • Importer: Responsible for importing data from Zeebe.
  • Archiver: Responsible for archiving "old" data (finished process instances and user operations.) See data retention.

Configuration​

Modules can be run together or separately in any combination and can be scaled. When you run an Operate instance, by default, all modules are enabled. To disable them, use the following configuration parameters:

Configuration parameterDescriptionDefault value
camunda.operate.importerEnabledWhen true, Importer module is enabled.True
camunda.operate.archiverEnabledWhen true, Archiver module is enabled.True
camunda.operate.webappEnabledWhen true, Webapp module is enabled.True

Scaling​

Additionally, you can have several importer and archiver nodes to increase throughput. Internally, they will spread their work based on Zeebe partitions.

For example, if your Zeebe runs 10 partitions and you configure two importer nodes, they will import data from five partitions each.

Each single importer/archiver node must be configured using the following configuration parameters:

Configuration parameterDescriptionDefault value
camunda.operate.clusterNode.partitionIdsArray of Zeebe partition ids this Importer (or Archiver) node must be responsible for.Empty array, meaning all partitions data is loaded.
camunda.operate.clusterNode.nodeCountTotal amount of Importer (or Archiver) nodes in the cluster.1
camunda.operate.clusterNode.currentNodeIdId of current Importer (or Archiver) node, starting from 0.0

It's enough to configure either partitionIds or a pair of nodeCount and currentNodeId. If you provide nodeCount and currentNodeId, each node will automatically guess the Zeebe partitions they're responsible for.

note

nodeCount always represents the number of nodes of one specific type.

For example, the configuration of a cluster with one Webapp node, two Importer nodes, and one Archiver node could look like the following:

Webapp node

camunda.operate:
archiverEnabled: false
importerEnabled: false
#other configuration...

Importer node #1

camunda.operate:
archiverEnabled: false
webappEnabled: false
clusterNode:
nodeCount: 2
currentNodeId: 0
#other configuration...

Importer node #2

camunda.operate:
archiverEnabled: false
webappEnabled: false
clusterNode:
nodeCount: 2
currentNodeId: 1
#other configuration...

Archiver node

camunda.operate:
webappEnabled: false
importerEnabled: false

You can further parallelize archiver and/or importer within one node using the following configuration parameters:

Configuration parameterDescriptionDefault value
camunda.operate.archiver.threadsCountNumber of threads in which data will be archived.1
camunda.operate.importer.threadsCountNumber of threads in which data will be imported.3
note

Parallelization of import and archiving within one node will also happen based on Zeebe partitions, meaning only configurations with (number of nodes) * (threadsCount) <= (total number of Zeebe partitions) will make sense. Too many threads and nodes will still work, but some of them will be idle.

Archive period​

The time between a process instance finishing and being archived can be set using the following configuration parameter:

Configuration parameterDescriptionDefault value
camunda.operate.archiver.waitPeriodBeforeArchivingAmount of time before data will be archived.1h

By default, the archive period is set to "1h" (one hour). This means the data for the finished process instances will be kept in the "main" index for one hour after the process instance has finished, and then it will be moved to a "dated" index.

The syntax for the parameter uses Elasticsearch date math. See the table below for reference:

ValueDescription
yYears
MMonths
wWeeks
dDays
hHours
mMinutes
sSeconds