search close

Error Response Codes

access_time Updated Jun 20, 2021

What do “-2”, “-1”, and “0” agent response codes mean?

The -2, -1, and 0 response codes are error codes applied to requests that weren’t processed correctly. There are a few reasons why this can happen but they tend to fall into two major categories:

  • The post/response couldn’t be matched to the request
  • The module timed out waiting for a response from the agent

Request and response mismatch

Error response codes can occur when a post/response couldn’t be matched to any actual requests. This is typically the result of NGINX redirecting before the request is passed to the Signal Sciences module.

Specific server response codes

The following server response codes cause NGINX to skip the phases that normally run. Due to their nature, they cause NGINX to finish processing the request without it being passed to the Signal Sciences module:

  • 400 (Bad Request)
  • 405 (Not Allowed)
  • 408 (Request Timeout)
  • 413 (Request Entity Too Large)
  • 414 (Request URI Too Large)
  • 494 (Request Headers Too Large)
  • 499 (Client Closed Request)
  • 500 (Internal Server Error)
  • 501 (Not Implemented)

Look for NGINX return directives

Look for custom NGINX configurations or Lua code that could be redirecting the request. This is almost always due to return directives in an NGINX configuration file. There could be return directives used to redirect specific pages to www, https, or a new URL. The return directive stops all processing, causing the request to not be processed by the Signal Sciences module. For example:

location /oldurl {
     return 302 https://newurl.com/
}

These would need to be updated to force the request to be processed by our agent first. Calling the rewrite_by_lua_block directly allows you to force the Signal Sciences module to run first and then perform the return statement for NGINX:

location /oldurl {
     rewrite_by_lua_block {
          sigsci.prerequest()
          return ngx.exit(302 "https://newurl.com/")
     }
     #return 302 https://newurl.com/
}

Agent restarted

Request and response mismatches can also be due to restarting the agent. If the agent is restarted after the request is processed, but before the response is processed, the agent will not see the response and fail to attribute it to the request, resulting in an error response code.

Module timing out

When the module receives a request, it sends it to the agent for processing. The module then waits for a response from the agent (whether or not to block) for a set amount of time (typically 100ms). If the agent doesn’t process the request within that time, the module will time out and default to failing open, allowing the request through. These requests that failed open will have error response codes applied to them.

Module timeouts are most commonly due to insufficient resources allocated to the agent. This can be a result of host or agent misconfiguration, such as the agent being limited to too few CPU cores.

This can also be due to a high volume of traffic to the host. If requests are coming in faster than the agent can process them subsequent requests will be queued for processing. If a queued request reaches the timeout limit, then the module will fail open and allow the request through.

Similarly, certain rules designed specifically for penetration testing can take longer to run than traditional rules. This can result in requests queueing and timing out due to the increased processing time per request.

Look at Response Time

Requests that are timing out will have a high response time, exceeding the default timeout of 100ms:

Look at Agent metrics

Metrics for each agent can be viewed by going to Agents > Metrics. There are a few metrics that can indicate error response codes are occurring due to the module timing out.

Connections dropped

The “Connections dropped” metric indicates number of requests that were allowed through (or “dropped”):

CPU usage

The CPU metrics can indicate the host is overloaded, preventing it from processing requests quickly enough.

  • The “Host CPU” metric indicates the CPU percentage for all cores together (100% is maximum).
  • The “Agent CPU” metric indicates the total CPU percentage for the number of cores in use by the agent. For example, if the agent were using 4 cores, then 400% would be the maximum.

CPU allocation and containerization

There are known issues with agents running within containers. It’s possible for agents to have insufficient CPU to process requests, due to a low number of CPUs (cores) allocated to the container by the cgroups feature.

We recommend the container running the agent should be given at least 1 CPU. If both NGINX and the agent are running in the same container, then we recommend allocating at least 1.5 CPUs.

Further help

If you’re unable to troubleshoot or resolve this issue yourself, generate an agent diagnostic package by running sigsci-agent-diag, which will output a .tar.gz archive with diagnostic information. Reach out to our support team to explain the issue in detail—including console links to the requests and agents affected—and provide the diagnostic .tar.gz archive.