Troubleshooting
Last updated 2023-06-14
Apache module fails to load
(The following information has been confirmed for RHEL/CentOS deployments using the default yum module installation.)
The default install location for the SigSci Apache module is /etc/httpd/modules
but some systems may have Apache loading it's config from a non-standard directory. When this happens the yum
installer will not install mod_signalsciences.so
to /etc/httpd/modules
but instead to the following path:
/usr/lib64/httpd/modules/mod_signalsciences.so
If Apache fails to restart after the module installation because it cannot locate mod_signalsciences.so
change the LoadModules line in httpd.conf
to reflect the correct location on the target system.
How do I configure the agent to use a proxy for egress traffic?
The agent can be configured to use a local proxy for egress traffic to the Signal Sciences cloud infrastructure by setting the HTTPS_PROXY
environment variable. Add the following line to /etc/default/sigsci-agent
, replacing IP-OR-HOST-NAME
with the IP address or hostname to proxy traffic to:
export HTTPS_PROXY=IP-OR-HOSTNAME
Restart the agent and verify the configuration.
How can I view requests that have been blocked or allowed by rules?
When configuring rules with a block or allow action, you can use the Request logging menu to select whether a sample of matched requests are logged or not logged.
When a request is logged, the individual request data and time series data for that request will be available throughout the web interface (e.g., on the Requests page). When a request is not logged, only time series data for that request will be available in the web interface.
Changing hostname for web servers
By default, the agent asks the OS for the hostname configuration. The agent can be configured to instead use a custom hostname in one of two ways:
Command line
Add the -server-hostname="HOSTNAME"
flag when starting the sigsci-agent
process via command line:
sigsci-agent -server-hostname="HOSTNAME"
Config file
Add the following line to your agent configuration file (by default at /etc/sigsci/agent.conf
):
server-hostname = "HOSTNAME"
Agent or module is not detected
When the module and agent have been successfully installed you will be able to see them reporting within the Agents page of the console. In many cases, customers first realize there may be a problem with their configuration when they have started the agent and everything appears to be running normally but the agent or module are not listed correctly.
Agent is not detected
Although the agent appears to be running, it's possible for the agent to not be listed in the Agents page of the console. This is typically due to either the agent being misconfigured or a connection issue between the agent and our cloud-hosted backend. Run through the following troubleshooting steps:
Check if the agent is running:
$ ps -aef | grep sigsci-agentTry restarting the agent with:
$ sudo restart sigsci-agentIf the agent is running, ensure communication between the agent and the cloud-hosted backend isn't blocked by your firewall. The Signal Sciences agent communicates with the following endpoints outbound via port 443/TCP:
c.signalsciences.net
wafconf.signalsciences.net
sigsci-agent-wafconf.s3.amazonaws.com
sigsci-agent-wafconf-us-west-2.s3.amazonaws.com
Additional information about firewall restrictions can be found in Architecture.
Review any log files for error messages:
$ ls -l /var/log/sigsci-agent$ tail -n 20 /var/log/sigsci-agentIf the agent is not starting and nothing is written to the log files, check what messages are displayed if you run the agent manually:
$ stop sigsci-agent$ /usr/sbin/sigsci-agentRun the debug tool and send the output, along with a detailed description of the issue and all log files, to our Support team.
$ /usr/sbin/sigsci-agent-diag
Module is not detected
Alternatively, although the console may show that the agent is reporting, the module may be listed as "undetected". There are a few possible causes to this scenario and the following steps are intended to help troubleshoot this condition:
It is necessary to send a request through the system in order for the module to report to the agent. Generating a manual 404 to the server in question by requesting a page that doesn't exist is the easiest way to start seeing traffic validated on the console. Allow up to 30 seconds from the time of the request for the module to report and the console to display the anomaly.
Confirm the steps for module installation specific to your web server, and any optional configuration changes, have been made correctly.
Restart the web server after module installation.
If the module is still not reporting and no data is showing in the console, check for issues related to domain socket permissions. By default, the agent and module are configured to use
/var/run/sigsci.sock
as the local domain socket under Linux operating systems and will require sufficient privileges to run properly:If using Red Hat/CentOS, check for SELinux:
$ sestatusIf SELinux is enabled refer to the SELinux support guide.
If using Ubuntu check for AppArmor and adjust security profiles if necessary:
$ sudo apparmor_status
If the module is still not reporting, reach out to our Support team with a detailed description of the issue and the following logs:
NGINX or Apache
error.log
, IIS error logs (default%SystemDrive%\inetpub\logs\LogFiles
)If NGINX is your web server, capture the output of:
$ /opt/sigsci/bin/check-nginxCollect the configuration files
/etc/sigsci/agent.conf
and if running NGINX/etc/nginx/nginx.conf
or if running Apache yourhttpd.conf
normally located in/etc/httpd/conf/httpd.conf
.
Agent not receiving request data when integrated with Ambassador
The Ambassador configuration may not have AuthService defined, which is required for the Signal Sciences agent to receive request data. AuthService is enabled by default; if the agent is not receiving requests, run kubectl get authservice
to check on the status of this service.
Why are my F5 load balancer health checks failing when going through the Signal Sciences reverse proxy?
F5 load balancer health checks use HTTP/0.9 by default. However, the SigSci reverse proxy does not support HTTP/0.9 because Go---which the Signal Sciences agent is written in---does not support it. This results in the F5 health checks failing with 400 Bad Request
response codes.
To resolve this, force the F5 health checks to use HTTP/1.0 or HTTP/1.1 instead. Specify the HTTP version in the send string, which will force the monitor to send an HTTP/1.0 or 1.1 request instead.
Below is an example of an HTTP/0.9 GET request:
GET /index.html
By specifying HTTP/1.0
, it will instead become an HTTP/1.0 GET request:
GET /index.html HTTP/1.0
For additional information about altering the F5 health check requests, see F5's official documentation.
What flags are available for configuring the agent?
The following options were derived from running the command sigsci-agent -help
and can be used as command line flags, set in /etc/sigsci/agent.conf
or set as ENV vars.
Refer to our Configuration Options to view all flags.
Generated environment variables:
1SIGSCI_RPC_ADDRESS2SIGSCI_RPC_VERSION3SIGSCI_ACCESSKEYID4SIGSCI_SECRETACCESSKEY5SIGSCI_MAX_CONNECTIONS6SIGSCI_MAX_BACKLOG7SIGSCI_MAX_PROCS8SIGSCI_MAX_RECORDS9SIGSCI_SAMPLE_PERCENT10SIGSCI_UPLOAD_URL11SIGSCI_UPLOAD_INTERVAL12SIGSCI_UPLOAD_SEND_EMPTY13SIGSCI_DOWNLOAD_URL14SIGSCI_DOWNLOAD_INTERVAL15SIGSCI_SERVER_HOSTNAME16SIGSCI_CLIENT_IP_HEADER17SIGSCI_REVERSE_PROXY18SIGSCI_REVERSE_PROXY_LISTENER19SIGSCI_REVERSE_PROXY_UPSTREAM20SIGSCI_DEBUG_LISTENER21SIGSCI_DEBUG_RPC_SERIAL22SIGSCI_DEBUG_GC_PERCENT23SIGSCI_DEBUG_DELAY24SIGSCI_DEBUG_ALWAYS_REPLY25SIGSCI_DEBUG_RPC_TEST_HARNESS26SIGSCI_DEBUG_LOG_BLOCKED_REQUESTS27SIGSCI_DEBUG_LOG_RULE_UPDATES28SIGSCI_DEBUG_LOG_WEB_INPUTS29SIGSCI_DEBUG_LOG_WEB_OUTPUTS30SIGSCI_DEBUG_LOG_UPLOADS31SIGSCI_DEBUG_LOG_PROXY_REQUESTS32SIGSCI_DEBUG_LOG_CONNECTION_ERRORS33SIGSCI_DEBUG_LOG_RPC_DATA34SIGSCI_DEBUG_STANDALONE35SIGSCI_DEBUG_LOG_ALL_THE_THINGS36SIGSCI_DEBUG_DISABLE_PROCESSING37SIGSCI_LEGAL38SIGSCI_VERSION39SIGSCI_SITE_KEYS
Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.