Troubleshooting

Apache module fails to load

(The following information has been confirmed for RHEL/CentOS deployments using the default yum module installation.)

The default install location for the SigSci Apache module is /etc/httpd/modules but some systems may have Apache loading it’s config from a non-standard directory. When this happens the yum installer will not install mod_signalsciences.so to /etc/httpd/modules but instead to the following path:

/usr/lib64/httpd/modules/mod_signalsciences.so

If Apache fails to restart after the module installation because it cannot locate mod_signalsciences.so change the LoadModules line in httpd.conf to reflect the correct location on the target system.

How do I configure the agent to use a proxy for egress traffic?

The agent can be configured to use a local proxy for egress traffic to the Signal Sciences cloud infrastructure by setting the HTTPS_PROXY environment variable. Add the following line to /etc/default/sigsci-agent, replacing IP-OR-HOST-NAME with the IP address or hostname to proxy traffic to:

export HTTPS_PROXY=IP-OR-HOSTNAME

Restart the agent and verify the configuration.

How can I view requests that have been blocked or allowed by rules?

When configuring rules with a block or allow action, you can use the Request logging menu to select whether a sample of matched requests are logged or not logged.

When a request is logged, the individual request data and timeseries data for that request will be available throughout the web interface (e.g., on the Requests page). When a request is not logged, only timeseries data for that request will be available in the web interface.

Changing hostname for web servers

By default, the agent asks the OS for the hostname configuration. The agent can be configured to instead use a custom hostname in one of two ways:

Command line

Add the -server-hostname="HOSTNAME" flag when starting the sigsci-agent process via command line:

sigsci-agent -server-hostname="HOSTNAME"

Config file

Add the following line to your agent configuration file (by default at /etc/sigsci/agent.conf):

server-hostname = "HOSTNAME"

Agent or module is not detected

When the module and agent have been successfully installed you will be able to see them reporting within the Agents page of the console. In many cases, customers first realize there may be a problem with their configuration when they have started the agent and everything appears to be running normally but the agent or module are not listed correctly.

Agent is not detected

Although the agent appears to be running, it’s possible for the agent to not be listed in the Agents page of the console. This is typically due to either the agent being misconfigured or a connection issue between the agent and our cloud-hosted backend. Run through the following troubleshooting steps:

Check if the agent is running:
```
ps -aef | grep sigsci-agent
```
Try restarting the agent with:
```
sudo restart sigsci-agent
```
If the agent is running, ensure communication between the agent and the cloud-hosted backend isn’t blocked by your firewall. The Signal Sciences agent communicates with the following endpoints outbound via port 443/TCP:
- c.signalsciences.net
- sigsci-agent-wafconf.s3.amazonaws.com
- sigsci-agent-wafconf-us-west-2.s3.amazonaws.com
Additional information about firewall restrictions can be found in Architecture

Review any log files for error messages:

ls -l /var/log/sigsci-agent

tail -n 20 /var/log/sigsci-agent

If the agent is not starting and nothing is written to the log files, check what messages are displayed if you run the agent manually:
```
stop sigsci-agent
/usr/sbin/sigsci-agent
```
Run the debug tool and send the output, along with a detailed description of the issue and all log files, to our Support team.
```
/usr/sbin/sigsci-agent-diag
```

Module is not detected

Alternatively, although the console may show that the agent is reporting, the module may be listed as “undetected”. There are a few possible causes to this scenario and the following steps are intended to help troubleshoot this condition:

It is necessary to send a request through the system in order for the module to report to the agent. Generating a manual 404 to the server in question by requesting a page that doesn’t exist is the easiest way to start seeing traffic validated on the console. Allow up to 30 seconds from the time of the request for the module to report and the console to display the anomaly.
Confirm the steps for module installation specific to your web server, and any optional configuration changes, have been made correctly.
Restart the web server after module installation.
If the module is still not reporting and no data is showing in the console, check for issues related to domain socket permissions. By default, the agent and module are configured to use /var/run/sigsci.sock as the local domain socket under Linux operating systems and will require sufficient privileges to run properly:
- If using Red Hat/CentOS, check for SELinux:
```
sestatus
```
  If SELinux is enabled refer to the SELinux support guide.
- If using Ubuntu check for AppArmor and adjust security profiles if necessary:
```
sudo apparmor_status
```
If the module is still not reporting, reach out to our Support team with a detailed description of the issue and the following logs:
- NGINX or Apache error.log, IIS error logs (default %SystemDrive%\inetpub\logs\LogFiles)
- If NGINX is your web server, capture the output of:
```
/opt/sigsci/bin/check-nginx
```
- Collect the configuration files /etc/sigsci/agent.conf and if running NGINX /etc/nginx/nginx.conf or if running Apache your httpd.conf normally located in /etc/httpd/conf/httpd.conf.

Agent not receiving request data when integrated with Ambassador

The Ambassador configuration may not have AuthService defined, which is required for the Signal Sciences agent to receive request data. AuthService is enabled by default; if the agent is not receiving requests, run kubectl get authservice to check on the status of this service.

What is a 499 status code?

You may occasionally see the Signal Sciences agent return a status code of 499. A 499 status code indicates the client closed the connection mid-request.

Why are my F5 load balancer health checks failing when going through the Signal Sciences reverse proxy?

F5 load balancer health checks use HTTP/0.9 by default. However, the SigSci reverse proxy does not support HTTP/0.9 because Go—which the Signal Sciences agent is written in—does not support it. This results in the F5 health checks failing with 400 “Bad Request” response codes.

To resolve this, force the F5 health checks to use HTTP/1.0 or HTTP/1.1 instead. Specify the HTTP version in the Send String, which will force the monitor to send an HTTP/1.0 or 1.1 request instead.

Below is an example of an HTTP/0.9 GET request:

GET /index.html

By specifying HTTP/1.0, it will instead become an HTTP/1.0 GET request:

GET /index.html HTTP/1.0

For additional information about altering the F5 health check requests, see F5’s official documentation.

What flags are available for configuring the agent?

The following options were derived from running the command sigsci-agent -help and can be used as command line flags, set in /etc/sigsci/agent.conf or set as ENV vars.

Refer to our Configuration Options to view all flags.

Generated environment variables:

SIGSCI_RPC_ADDRESS
SIGSCI_RPC_VERSION
SIGSCI_ACCESSKEYID
SIGSCI_SECRETACCESSKEY
SIGSCI_MAX_CONNECTIONS
SIGSCI_MAX_BACKLOG
SIGSCI_MAX_PROCS
SIGSCI_MAX_RECORDS
SIGSCI_SAMPLE_PERCENT
SIGSCI_UPLOAD_URL
SIGSCI_UPLOAD_INTERVAL
SIGSCI_UPLOAD_SEND_EMPTY
SIGSCI_DOWNLOAD_URL
SIGSCI_DOWNLOAD_INTERVAL
SIGSCI_SERVER_HOSTNAME
SIGSCI_CLIENT_IP_HEADER
SIGSCI_REVERSE_PROXY
SIGSCI_REVERSE_PROXY_LISTENER
SIGSCI_REVERSE_PROXY_UPSTREAM
SIGSCI_DEBUG_LISTENER
SIGSCI_DEBUG_RPC_SERIAL
SIGSCI_DEBUG_GC_PERCENT
SIGSCI_DEBUG_DELAY
SIGSCI_DEBUG_ALWAYS_REPLY
SIGSCI_DEBUG_RPC_TEST_HARNESS
SIGSCI_DEBUG_LOG_BLOCKED_REQUESTS
SIGSCI_DEBUG_LOG_RULE_UPDATES
SIGSCI_DEBUG_LOG_WEB_INPUTS
SIGSCI_DEBUG_LOG_WEB_OUTPUTS
SIGSCI_DEBUG_LOG_UPLOADS
SIGSCI_DEBUG_LOG_PROXY_REQUESTS
SIGSCI_DEBUG_LOG_CONNECTION_ERRORS
SIGSCI_DEBUG_LOG_RPC_DATA
SIGSCI_DEBUG_STANDALONE
SIGSCI_DEBUG_LOG_ALL_THE_THINGS
SIGSCI_DEBUG_DISABLE_PROCESSING
SIGSCI_LEGAL
SIGSCI_VERSION
SIGSCI_SITE_KEYS