Configure the number of concurrent jobs executed by the application

This procedure explains how to modify the number of asynchronous tasks run simultaneously by the Cyberwatch application.

Understanding the tasks queues in Cyberwatch

The execution of applications tasks is done asynchronously in order to improve performances and allow for parallelization of tasks.

Pending tasks are put in different waiting queues depending on the type of the task:

• the cbw_api_client queue is dedicated to synchronization tasks with the Cyberwatch central security database;
• the default queue is for the tasks relative to the automatic actions of the software (scheduled reports, asset rules…);
• the servers queue is relative to the scans results analyses of monitored assets;
• other queues are named after the Cyberwatch nodes registered in the application, these tasks are exclusively executed by the corresponding scanners (for agentless connections for example).

Number of tasks concurrently running

By default, the application is able to execute:

• 6 concurrent tasks relative to the specific scanner for each Cyberwatch scanner;
• 4 concurrent tasks for the other queues mentioned (cbw_api_client, default and servers) for each Cyberwatch scanner.

These values can be modified through the application base configuration.

They are respectively defined by the value of environment variables SIDEKIQ_NODE_CONCURRENCY and SIDEKIQ_CONCURRENCY defined in the containers.

These tasks are respectively executed by the sidekiq_node and sidekiq services.

Adding sidekiq replicas

By default, sidekiq will only use one CPU for managing tasks.

Default value for the number of tasks (4) is generally enough in most configurations for sidekiq to use a whole CPU for running these tasks.

If there is a latency in the applications due to a slow execution of tasks, the first recommended approach is to allow sidekiq to use both CPU of the Cyberwatch server.

This will double the number of tasks that can be ran concurrently by the Cyberwatch scanner.

These modifications should only be done following the recommendation of a Cyberwatch engineer.

To apply these modifications, follow the procedure below:

1. Open the file /etc/cyberwatch/config.env and edit or add the following lines:

   # Number of Sidekiq replicas to start.
   CBW_SIDEKIQ_REPLICAS=1
   

Replace 1 with the desired number of replicas.

1. Restart Cyberwatch using the standard procedure:

   sudo cyberwatch restart
   

Modifying the values of SIDEKIQ_CONCURRENCY and SIDEKIQ_NODE_CONCURRENCY

It can be useful to modify these default values when the hardware configuration of the Cyberwatch scanner allows it.

These modifications will allow a bigger increase in the number of parallel tasks running concurrently and will ideally allow to remove or to reduce potential latencies in the Cyberwatch application.

These modifications should only be done following the recommendation of a Cyberwatch engineer.

To apply these modifications, follow the procedure below:

1. Create the file /etc/cyberwatch/configs-enabled/99-sidekiq-concurrency.yml using the following command:

   cat >> /etc/cyberwatch/configs-enabled/99-sidekiq-concurrency.yml <<EOL
   version: '3.7'
   
   services:
     sidekiq:
       environment:
         SIDEKIQ_CONCURRENCY: "\${SIDEKIQ_CONCURRENCY}"
     sidekiq_node:
       environment:
         SIDEKIQ_NODE_CONCURRENCY: "\${SIDEKIQ_NODE_CONCURRENCY}"
   EOL
   

2. Update the file /etc/cyberwatch/containers.env and add the following lines:

   SIDEKIQ_CONCURRENCY=N
   SIDEKIQ_NODE_CONCURRENCY=N
   

   where the N variable should take the value of the number of concurrent tasks for each execution thread.

3. Restart Cyberwatch using the standard procedure:

   sudo cyberwatch restart