search close

Performance & Reliability

access_time Updated Sep 21, 2021

Performance

How does your architecture ensure high performance and reliability?

One of the key reasons for the architectural split between the module and the agent is to optimize for maximum performance and reliability. If the agent ever crashes, your application does not go down because the module fails open if it doesn’t hear back from the agent within a set time limit. This claim is simple to verify in a deployment, as the module can be enabled without running the agent, and the site will continue to load as normal. From the performance side, this set time limit is also the worst case latency that Signal Sciences could introduce to a request.

Can I see the actual performance impact of Signal Sciences on my systems?

Yes. We provide graphs and data on resources used by the agent in the agent details page in the console.

How much memory does Signal Sciences consume?

Most clients see median memory usage of 1024mb (1GB) in production deployments.

How much CPU does Signal Sciences consume?

CPU varies by machine size. By default the number of available cores determines the maximum cores the agent can use.

The agent scales using the following by default (overridable - see below):

Available Cores Agent Core Limit
1 1
2 - 3 2
4 or more 50% of available

Agent and Module

How much time does the agent spend processing a request?

Most clients see a median time of 0.6ms to 2.0ms in production deployments.

How often does the agent poll for new decisions?

By default every 30 seconds the agent will poll for any new decisions made by the back end, and this value is configurable via the agent command line.

What measures are in place to ensure agent updates are from an authorized source?

Agent updates, such as new decisions made by the back end, are encrypted by the back end and then decrypted by the agent using the agent keys.

What impact does the agent to backend communication have on my egress bandwidth?

Impact to egress bandwidth is minimal. Every 30 seconds, we compress any data we have collected and send it to our backend. In other words, it’s a ratio of n inbound attacks to one outbound request to our backend.

Are my production systems impacted if the Signal Sciences backend goes down?

No. All agent communication with the backend is asynchronous. Should the agent lose the ability to communicate with the Signal Sciences cloud backend the agent will continue to function with the following caveats:

  • The agent will continue to perform detections of attacks, anomalies and any custom rules/signals
  • The agent will continue to enforce existing blocking decisions
  • The agent will not queue request logs and there will be an outage of data shown in the console, ability to look at individual requests or aggregate data will be lost until the connection is reestablished.
  • The agent will not receive updates for new detections or enforcement decisions

How can I disable the agent?

The agent can be disabled in two ways: 1) by clicking the agent mode toggle at the top of the consoles and selecting “off,” and 2) by stopping agents via configuration management.

How do I increase the number of CPUs available to the agent?

By default the agent is configured to scale proportionally based on the number of available CPUs on a system (see above). This is typically a reasonable number, but cases of extremely high throughput can lead to resource contention, which manifests as higher latency and increased memory utilization with a slightly elevated decision time.

To change the number of cores available to the agent, edit the agent configuration file (typically /etc/sigsci/agent.conf) to include the line max-procs = n where “n” is the number of CPU cores to use. You must then restart the agent for this change to take effect.

What’s the difference between the “Host CPU” and “Agent CPU” metrics?

The “Host CPU” metric indicates the CPU percentage for the full host wherein 100% is all cores.

The “Agent CPU” metric on the other hand doesn’t use a scale of 100%. The Agent CPU metric is the CPU by core.

For example, take a machine with 8 cores: the maximum Agent CPU percentage would be 800%. However, if the agent has been configured to be limited to only 4 cores, the maximum Agent CPU percentage would instead be 400%. In this example, if the agent is shown to be taking about 50% CPU, it’s actually only using 6% CPU (50% Agent CPU / 800% total CPU).

Updates

How frequently do you release updates to the agent and module?

See:

How are updates to the agent/module tested?

Our testing process includes:

  • Unit tests
  • Integration tests
  • Security tests
  • Signal detection tests (quality test)
  • Module correctness tests
  • Packaging / install tests
  • Performance tests (load tests)

Most of these are completely automated and run regularly, if not constantly.