search close

Kubernetes Agent + Ingress Controller + Module

access_time Updated Jun 29, 2022

Introduction

In this example, the Signal Sciences agent is installed as a Docker sidecar, communicating with a Signal Sciences native module for Nginx installed on an ingress-nginx Kubernetes ingress controller.

Integrating the Signal Sciences agent into an ingress controller

In addition to installing Signal Sciences per application, it is also possible to install Signal Sciences into a Kubernetes ingress controller that will receive all external traffic to your applications. Doing this is similar to installing into an application with a Signal Sciences module:

  • Install and configure the Signal Sciences Module into the ingress controller.
  • Add the sigsci-agent container to the ingress pod and mount a sigsci-agent volume.
  • Add an emptyDir{} volume as a place for the sigsci-agent to write temporary data.

Kubernetes Nginx ingress controller

The Kubernetes Nginx Ingress Controller is an Nginx based implementation for the ingress API. Signal Sciences supports a native module for Nginx. This enables you to easily wrap the existing ingress-nginx controller to install the Signal Sciences module.

Wrap the base nginx-ingress-controller to install the Signal Sciences module

Wrapping the nginx-ingress-controller is done by using the base controller and installing the Signal Sciences native Nginx module. An example can be found here and here

A prebuilt container can be pulled from Docker Hub with: docker pull signalsciences/sigsci-nginx-ingress-controller:0.47.0

Installation

There are two methods for installing:

Install via Helm using overrides

The following steps cover installing sigsci-nginx-ingress-controller + sigsci-agent via the official ingress-nginx charts with an override file.

  1. Add the ingress-nginx repo:

    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

  2. Locate the Agent Keys for your Signal Sciences site:

    1. Log in to the Signal Sciences console.

    2. Select a site if you have more than one site.

    3. Click Agents in the navigation bar. The agents page appears.

    4. Click View agent keys. The agent keys window appears.

      The 'View agent keys' button.
    5. Copy the Agent Access Key and Agent Secret Key.

      The agent keys window.
  3. In the sigsci-values.yaml file, add the Agent Keys as SIGSCI_ACCESSKEYID and SIGSCI_SECRETACCESSKEY.

  4. Install with the release name my-ingress in the default namespace:

    helm install -f values-sigsci.yaml my-ingress ingress-nginx/ingress-nginx

    You can specify a namespace with -n flag:

    helm install -n NAMESPACE -f values-sigsci.yaml my-ingress ingress-nginx/ingress-nginx

  5. After a few minutes, the agent will be listed in your Signal Sciences console.

  6. Create an Ingress resource. This step will vary depending on setup and supports a lot of configurations. Official documentation can be found regarding Basic usage - host based routing.

    Here is an example Ingress file:

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/rewrite-target: /
      name: hello-kubernetes-ingress
      #namespace: SET THIS IF NOT IN DEFAULT NAMESPACE
    spec:
      rules:
      - host: example.com
        http:
          paths:
          - pathType: Prefix
            path: /testpath
            backend:
              service:
                name: NAME OF SERVICE
                port:
                  number: 80
        

Helm upgrade with override file

  1. In the sigsci-values.yaml file, update the sigsci-nginx-ingress-controller to the latest version to update the ingress-nginx charts:

    controller:
        # Replaces the default nginx-controller image with a custom image that contains the Signal Sciences Nginx Module
        image:
          repository: signalsciences/sigsci-nginx-ingress-controller
          tag: "0.47.0"
          pullPolicy: IfNotPresent
  2. Run helm upgrade with the override file. This example is running helm upgrade against the my-ingress release created in the previous section:

    helm upgrade -f sigsci-values.yaml my-ingress ingress-nginx/ingress-nginx
    or
    helm upgrade -f sigsci-nginxinc-values.yaml my-ingress ingress-nginx/ingress-nginx

    If ingress is not in default namespace, use -n to specify namespace:

    helm upgrade -n NAMESPACE -f sigsci-values.yaml my-ingress ingress-nginx/ingress-nginx
    or
    helm upgrade -n NAMESPACE -f sigsci-nginxinc-values.yaml my-ingress ingress-nginx/ingress-nginx

Uninstall release

  1. Uninstall release my-ingress.

    helm uninstall my-ingress
  2. If it’s not in the default namespace, use -n to specify the namespace:

    helm uninstall -n NAMESPACE my-ingress

Install with custom file

Integrating the Signal Sciences Agent

The Signal Sciences Agent can be installed as a sidecar into each pod or as a service for some specialized needs.

The recommended way of installing the Signal Sciences Agent in Kubernetes is by integrating the sigsci-agent into a pod as a sidecar. This means adding the sigsci-agent as an additional container to the Kubernetes pod. As a sidecar, the agent will scale with the app/service in the pod instead of having to do this separately. However, in some situations, it may make more sense to install the sigsci-agent container as a service and scale it separately from the application.

The sigsci-agent container can be configured in various ways depending on the installation type and module being used.

Getting and Updating the Signal Sciences Agent Container Image

An official signalsciences/sigsci-agent container image is available from the Signal Sciences account on Docker Hub.

Alternatively, if you want to build your own image or need to customize the image, then follow the sigsci-agent build instructions.

These instructions reference the latest version of the agent with imagePullPolicy: Always, which will pull the latest agent version even if one already exist locally. This is so the documentation does not fall out of date and anyone using this will not have an agent that stays stagnant. However, this may not be what if you need to keep installations consistent or on a specific version of the agent. In these cases, you should specify an agent version. Images on Docker Hub are tagged with their versions and a list of versions is available on Docker Hub.

Whether you choose to use the latest image or a specific version, there are a few items to consider to keep the agent up-to-date:

Using the latest Signal Sciences Container Image

If you do choose to use the latest image, then you will want to consider how you will keep the agent up to date.

  • If you have used the imagePullPolicy: Always option, then the latest image will be pulled on each startup and your agent will continue to get updates.

  • Alternatively, you may instead choose to manually update the local cache by periodically forcing a pull instead of always pulling on startup:

    docker pull signalsciences/sigsci-agent:latest
    

    Then, use latest with imagePullPolicy: Never set in the configuration so that pulls are never done on startup (only manually as above):

    - name: sigsci-agent
       image: signalsciences/sigsci-agent:latest
       imagePullPolicy: Never
       ...
    

Using a Versioned Signal Sciences Container Image

To use a specific version of the agent, replace latest with the agent version. You may also want to change imagePullPolicy: IfNotPresent in this case as the image should not change.

- name: sigsci-agent
   image: signalsciences/sigsci-agent:4.1.0
   imagePullPolicy: IfNotPresent
   ...

This will pull the specified agent version and cache it locally. If you use this method, then it is recommended that you parameterize the agent image, using Helm or similar, so that it is easier to update the agent images later on.

Using a Custom Tag for the Signal Sciences Container Image

It is also possible to apply a custom tag to a local agent image. To do this, pull the agent image (by version or use latest), apply a custom tag, then use that custom tag in the configuration. You will need to specify imagePullPolicy: Never so local images are only updated manually. After doing so, you will need to periodically update the local image to keep the agent up-to-date.

For example:

docker pull signalsciences/sigsci-agent:latest
docker tag signalsciences/sigsci-agent:latest signalsciences/sigsci-agent:testing

Then use this image tag in the configuration:

- name: sigsci-agent
   image: signalsciences/sigsci-agent:testing
   imagePullPolicy: Never
...

Configuring the Signal Sciences Agent Container

Agent configuration is normally done via the environment. Most configuration options are available as environment variables. Environment variables names have the configuration option name all capitalized, prefixed with SIGSCI_ and any dashes (-) changed to underscores (_). For example, the max-procs option would become the SIGSCI_MAX_PROCS environment variable. For more details on what options are available, see the Agent Configuration documentation.

The sigsci-agent container has a few required options that need to be configured:

  • Agent credentials (Agent Access Key and Agent Secret Key).

  • A volume to write temporary files.

Agent Credentials

The sigsci-agent credentials are configured with two environment variables. These variables must be set or the agent will not start.

  • SIGSCI_ACCESSKEYID: The Agent Access Key identifies which site in the Signal Sciences console that the agent is configured for.
  • SIGSCI_SECRETACCESSKEY: The Agent Secret Key is the shared secret key to authenticate and authorize the agent.

The credentials can be found by following these steps:

  1. Log in to the Signal Sciences console.

  2. Select a site if you have more than one site.

  3. Click Agents in the navigation bar. The agents page appears.

  4. Click View agent keys. The agent keys window appears.

    The 'View agent keys' button.
  5. Copy the Agent Access Key and Agent Secret Key.

    The agent keys window.

Because of the sensitive nature of these values, we recommend you use the built in secrets functionality of Kubernetes. With this configuration, the agent will pull the values from the secrets data instead of reading hardcoded values into the deployment configuration. This also makes any desired agent credential rotation easier to manage by having to change them in only one place.

Use the valueForm option instead of the value option to utilize the secrets functionality. For example:

env:
 - name: SIGSCI_ACCESSKEYID
   valueFrom:
     secretKeyRef:
       # Update "my-site-name-here" to the correct site name or similar identifier
       name: sigsci.my-site-name-here
       key: accesskeyid
 - name: SIGSCI_SECRETACCESSKEY
   valueFrom:
     secretKeyRef:
       # Update "my-site-name-here" to the correct site name or similar identifier
       name: sigsci.my-site-name-here
       key: secretaccesskey

The secrets functionality keeps secrets in various stores in Kubernetes. This guide uses the generic secret store in its examples, however any equivalent store can be used. Agent secrets can be added to the generic secret store using YAML similar to the following example:

apiVersion: v1
kind: Secret
metadata:
  name: sigsci.my-site-name-here
stringData:
  accesskeyid: 12345678-abcd-1234-abcd-1234567890ab
  secretaccesskey: abcdefg_hijklmn_opqrstuvwxy_z0123456789ABCD

This can also be created from the command line with kubectl such as with the following example:

kubectl create secret generic sigsci.my-site-name-here \
  --from-literal=accesskeyid=12345678-abcd-1234-abcd-1234567890ab \
  --from-literal=secretaccesskey=abcdefg_hijklmn_opqrstuvwxy_z0123456789ABCD

Additional information about Kubernetes secrets functionality can be found here.

Agent Temporary Volume

For added security, we recommended the sigsci-agent container be executed with the root filesystem mounted as read only. However, the agent still needs to write some temporary files such as the socket file for RPC communication and some periodically updated files such as GeoIP data.

To accomplish this with a read only root filesystem, there needs to be a writeable volume mounted. This writeable volume can also be shared to expose the RPC socket file to other containers in the same pod.

The recommended way of creating a writeable volume is to use the builtin emptyDir volume type. This is typically configured in the volumes section of a deployment, as shown in the following example:

volumes:
 - name: sigsci-tmp
   emptyDir: {}

Containers will then mount this volume at /sigsci/tmp:

volumeMounts:
 - name: sigsci-tmp
   mountPath: /sigsci/tmp

The default in the official agent container image is to have the temporary volume mounted at /sigsci/tmp. If this needs to be moved for the agent container, then the following agent configuration options should also be changed from their defaults to match the new mount location:

  • rpc-address defaults to /sigsci/tmp/sigsci.sock
  • shared-cache-dir defaults to /sigsci/tmp/cache

The Nginx ingress controller is installed with the [mandatory.yaml](/install-guides/kubernetes/mandatory.yaml) file. This file contains a modified template of the Generic Ingress Controller Deployment as described [here](https://kubernetes.github.io/ingress-nginx/deploy/#prerequisite-generic-deployment-command). The main additions are:
  1. Change the ingress container to load the custom Signal Sciences Module/ingress container and add Volume mounts for socket file communication between the Module/ingress container and Agent sidecar container:

    ...
        containers:
          - name: nginx-ingress-controller
            image: signalsciences/sigsci-nginx-ingress-controller:0.47.0
            ...
            volumeMounts:
              - name: sigsci-tmp
                mountPath: /sigsci/tmp
    ...
  2. Load the Signal Sciences Module in the Nginx configuration file (nginx.conf) via ConfigMap:

    kind: ConfigMap
    apiVersion: v1
    data:
      main-snippet: load_module /usr/lib/nginx/modules/ngx_http_sigsci_nxo_module-1.17.7.so;
      http-snippet: sigsci_agent_host unix:/sigsci/tmp/sigsci.sock;
    metadata:
      name: nginx-configuration
      namespace: ingress-nginx
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
  3. Add a container for the Signal Sciences Agent:

    ...
        containers:
        ...
          # Signal Sciences Agent running in default RPC mode
          - name: sigsci-agent
            image: signalsciences/sigsci-agent:latest
            imagePullPolicy: IfNotPresent
            env:
            - name: SIGSCI_ACCESSKEYID
              valueFrom:
                secretKeyRef:
                  # This secret needs added (see docs on sigsci secrets)
                  name: sigsci.my-site-name-here
                  key: accesskeyid
            - name: SIGSCI_SECRETACCESSKEY
              valueFrom:
                secretKeyRef:
                  # This secret needs added (see docs on sigsci secrets)
                  name: sigsci.my-site-name-here
                  key: secretaccesskey
            securityContext:
              # The sigsci-agent container should run with its root filesystem read only
              readOnlyRootFilesystem: true
            volumeMounts:
            # Default volume mount location for sigsci-agent writeable data (do not change mount path)
            - name: sigsci-tmp
              mountPath: /sigsci/tmp
    ...
  4. Define the volume used above:

    ...
          volumes:
          # Define a volume where sigsci-agent will write temp data and share the socket file,
          # which is required with the root filesystem is mounted read only
          - name: sigsci-tmp
            emptyDir: {}
    ...

Setup

The mandatory.yaml file creates the resources in the ingress-nginx namespace. If using Kubernetes Secrets to store the agent access keys, you will need to create the namespace and access keys before running the mandatory.yaml file.

  1. Set the name for the secrets for the agent keys in mandatory.yaml.

    ...
        env:
        - name: SIGSCI_ACCESSKEYID
          valueFrom:
            secretKeyRef:
              # This secret needs added (see docs on sigsci secrets)
              name: sigsci.my-site-name-here
              key: accesskeyid
        - name: SIGSCI_SECRETACCESSKEY
          valueFrom:
            secretKeyRef:
              # This secret needs added (see docs on sigsci secrets)
              name: sigsci.my-site-name-here
              key: secretaccesskey
    ...

  2. Pull or build the “Nginx ingress + Signal Sciences Module” container. Set any preferred registry and repository name, and set the image to match in mandatory.yaml:

    docker pull signalsciences/sigsci-nginx-ingress-controller:0.47.0

  3. Deploy using modified Generic Deployment:

    kubectl apply -f mandatory.yaml

  4. Create the service to expose the Ingress Controller. The steps necessary are dependent on your cloud provider. Official instructions can be found at https://kubernetes.github.io/ingress-nginx/deploy/#provider-specific-steps.

    Below is an example service.yaml file:

    kind: Service
    apiVersion: v1
    metadata:
      name: ingress-nginx
      namespace: ingress-nginx
    spec:
      externalTrafficPolicy: Cluster
      selector:
        app.kubernetes.io/name: ingress-nginx
      type: LoadBalancer
      ports:
        - name: http
          port: 80
          targetPort: http
        - name: https
          port: 443
          targetPort: https

  5. Create the Ingress Resource. Below is an example Ingress Resource:

    apiVersion: extensions/v1
    kind: Ingress
    metadata:
      name: test-ingress
      namespace: ingress-nginx
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: /
    spec:
      rules:
      - http:
          paths:
          - path: /testpath
            backend:
              serviceName: nginx
              servicePort: 80