Troubleshooting

  Last updated 2023-06-14

Apache module fails to load

(The following information has been confirmed for RHEL/CentOS deployments using the default yum module installation.)

The default install location for the SigSci Apache module is /etc/httpd/modules but some systems may have Apache loading it's config from a non-standard directory. When this happens the yum installer will not install mod_signalsciences.so to /etc/httpd/modules but instead to the following path:

/usr/lib64/httpd/modules/mod_signalsciences.so

If Apache fails to restart after the module installation because it cannot locate mod_signalsciences.so change the LoadModules line in httpd.conf to reflect the correct location on the target system.

How do I configure the agent to use a proxy for egress traffic?

The agent can be configured to use a local proxy for egress traffic to the Signal Sciences cloud infrastructure by setting the HTTPS_PROXY environment variable. Add the following line to /etc/default/sigsci-agent, replacing IP-OR-HOST-NAME with the IP address or hostname to proxy traffic to:

export HTTPS_PROXY=IP-OR-HOSTNAME

Restart the agent and verify the configuration.

How can I view requests that have been blocked or allowed by rules?

When configuring rules with a block or allow action, you can use the Request logging menu to select whether a sample of matched requests are logged or not logged.

When a request is logged, the individual request data and time series data for that request will be available throughout the web interface (e.g., on the Requests page). When a request is not logged, only time series data for that request will be available in the web interface.

Changing hostname for web servers

By default, the agent asks the OS for the hostname configuration. The agent can be configured to instead use a custom hostname in one of two ways:

Command line

Add the -server-hostname="HOSTNAME" flag when starting the sigsci-agent process via command line:

sigsci-agent -server-hostname="HOSTNAME"

Config file

Add the following line to your agent configuration file (by default at /etc/sigsci/agent.conf):

server-hostname = "HOSTNAME"

Agent or module is not detected

When the module and agent have been successfully installed you will be able to see them reporting within the Agents page of the console. In many cases, customers first realize there may be a problem with their configuration when they have started the agent and everything appears to be running normally but the agent or module are not listed correctly.

Agent is not detected

Although the agent appears to be running, it's possible for the agent to not be listed in the Agents page of the console. This is typically due to either the agent being misconfigured or a connection issue between the agent and our cloud-hosted backend. Run through the following troubleshooting steps:

  1. Check if the agent is running:

    $ ps -aef | grep sigsci-agent

  2. Try restarting the agent with:

    $ sudo restart sigsci-agent

  3. If the agent is running, ensure communication between the agent and the cloud-hosted backend isn't blocked by your firewall. The Signal Sciences agent communicates with the following endpoints outbound via port 443/TCP:

    • c.signalsciences.net
    • wafconf.signalsciences.net
    • sigsci-agent-wafconf.s3.amazonaws.com
    • sigsci-agent-wafconf-us-west-2.s3.amazonaws.com

    Additional information about firewall restrictions can be found in Architecture.

  4. Review any log files for error messages:

    $ ls -l /var/log/sigsci-agent
    $ tail -n 20 /var/log/sigsci-agent

  5. If the agent is not starting and nothing is written to the log files, check what messages are displayed if you run the agent manually:

    $ stop sigsci-agent
    $ /usr/sbin/sigsci-agent

  6. Run the debug tool and send the output, along with a detailed description of the issue and all log files, to our Support team.

    $ /usr/sbin/sigsci-agent-diag

Module is not detected

Alternatively, although the console may show that the agent is reporting, the module may be listed as "undetected". There are a few possible causes to this scenario and the following steps are intended to help troubleshoot this condition:

  1. It is necessary to send a request through the system in order for the module to report to the agent. Generating a manual 404 to the server in question by requesting a page that doesn't exist is the easiest way to start seeing traffic validated on the console. Allow up to 30 seconds from the time of the request for the module to report and the console to display the anomaly.

  2. Confirm the steps for module installation specific to your web server, and any optional configuration changes, have been made correctly.

  3. Restart the web server after module installation.

  4. If the module is still not reporting and no data is showing in the console, check for issues related to domain socket permissions. By default, the agent and module are configured to use /var/run/sigsci.sock as the local domain socket under Linux operating systems and will require sufficient privileges to run properly:

    • If using Red Hat/CentOS, check for SELinux:

      $ sestatus

      If SELinux is enabled refer to the SELinux support guide.

    • If using Ubuntu check for AppArmor and adjust security profiles if necessary:

      $ sudo apparmor_status

  5. If the module is still not reporting, reach out to our Support team with a detailed description of the issue and the following logs:

    • NGINX or Apache error.log, IIS error logs (default %SystemDrive%\inetpub\logs\LogFiles)

    • If NGINX is your web server, capture the output of:

      $ /opt/sigsci/bin/check-nginx

    • Collect the configuration files /etc/sigsci/agent.conf and if running NGINX /etc/nginx/nginx.conf or if running Apache your httpd.conf normally located in /etc/httpd/conf/httpd.conf.

Agent not receiving request data when integrated with Ambassador

The Ambassador configuration may not have AuthService defined, which is required for the Signal Sciences agent to receive request data. AuthService is enabled by default; if the agent is not receiving requests, run kubectl get authservice to check on the status of this service.

Why are my F5 load balancer health checks failing when going through the Signal Sciences reverse proxy?

F5 load balancer health checks use HTTP/0.9 by default. However, the SigSci reverse proxy does not support HTTP/0.9 because Go---which the Signal Sciences agent is written in---does not support it. This results in the F5 health checks failing with 400 Bad Request response codes.

To resolve this, force the F5 health checks to use HTTP/1.0 or HTTP/1.1 instead. Specify the HTTP version in the send string, which will force the monitor to send an HTTP/1.0 or 1.1 request instead.

Below is an example of an HTTP/0.9 GET request:

GET /index.html

By specifying HTTP/1.0, it will instead become an HTTP/1.0 GET request:

GET /index.html HTTP/1.0

For additional information about altering the F5 health check requests, see F5's official documentation.

What flags are available for configuring the agent?

The following options were derived from running the command sigsci-agent -help and can be used as command line flags, set in /etc/sigsci/agent.conf or set as ENV vars.

Refer to our Configuration Options to view all flags.

Generated environment variables:

1SIGSCI_RPC_ADDRESS
2SIGSCI_RPC_VERSION
3SIGSCI_ACCESSKEYID
4SIGSCI_SECRETACCESSKEY
5SIGSCI_MAX_CONNECTIONS
6SIGSCI_MAX_BACKLOG
7SIGSCI_MAX_PROCS
8SIGSCI_MAX_RECORDS
9SIGSCI_SAMPLE_PERCENT
10SIGSCI_UPLOAD_URL
11SIGSCI_UPLOAD_INTERVAL
12SIGSCI_UPLOAD_SEND_EMPTY
13SIGSCI_DOWNLOAD_URL
14SIGSCI_DOWNLOAD_INTERVAL
15SIGSCI_SERVER_HOSTNAME
16SIGSCI_CLIENT_IP_HEADER
17SIGSCI_REVERSE_PROXY
18SIGSCI_REVERSE_PROXY_LISTENER
19SIGSCI_REVERSE_PROXY_UPSTREAM
20SIGSCI_DEBUG_LISTENER
21SIGSCI_DEBUG_RPC_SERIAL
22SIGSCI_DEBUG_GC_PERCENT
23SIGSCI_DEBUG_DELAY
24SIGSCI_DEBUG_ALWAYS_REPLY
25SIGSCI_DEBUG_RPC_TEST_HARNESS
26SIGSCI_DEBUG_LOG_BLOCKED_REQUESTS
27SIGSCI_DEBUG_LOG_RULE_UPDATES
28SIGSCI_DEBUG_LOG_WEB_INPUTS
29SIGSCI_DEBUG_LOG_WEB_OUTPUTS
30SIGSCI_DEBUG_LOG_UPLOADS
31SIGSCI_DEBUG_LOG_PROXY_REQUESTS
32SIGSCI_DEBUG_LOG_CONNECTION_ERRORS
33SIGSCI_DEBUG_LOG_RPC_DATA
34SIGSCI_DEBUG_STANDALONE
35SIGSCI_DEBUG_LOG_ALL_THE_THINGS
36SIGSCI_DEBUG_DISABLE_PROCESSING
37SIGSCI_LEGAL
38SIGSCI_VERSION
39SIGSCI_SITE_KEYS
Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.