search close

Kubernetes Istio

access_time Updated Jun 20, 2021

Introduction

In this example, the Signal Sciences runs in a Docker sidecar and integrates directly with an Istio service mesh deployed on the application. In this configuration, Signal Sciences can be configured to inspect east/west (service-to-service) web requests along with the traditional north/south (client to server) requests.

Integrating the Signal Sciences Agent

The Signal Sciences Agent can be installed as a sidecar into each pod or as a service for some specialized needs. The recommended way of installing the Signal Sciences Agent in Kubernetes is by integrating the sigsci-agent into a pod as a sidecar. This just means adding the sigsci-agent as an additional container to the Kubernetes pod. As a sidecar, the agent will scale with the app/service in the pod instead of having to do this separately. However, in some situations, it may make more sense to install the sigsci-agent container as a service and scale it separately from the application. The sigsci-agent container can be configured in various ways depending on the installation type and module being used.

Getting and Updating the Signal Sciences Agent Container Image

The official signalsciences/sigsci-agent container image available from the Signal Sciences account on Docker Hub is the recommended place to get the image. If you want to build your own image or need to customize the image, then follow the sigsci-agent build instructions.

The documentation references the latest version of the agent with imagePullPolicy: Always which will pull the latest agent version even if one already exist locally. This is so the documentation does not fall out of date and anyone using this will not have an agent that stays stagnant, however this may not be what if you need to keep installations consistent or on a specific version of the agent. In this case you should specify a version. Images on Docker Hub are tagged with their versions and a list of versions is available on Docker Hub.

Whether you choose to use the latest image or a specific version, there are a few items to consider to keep the agent up-to-date:

Using the latest Signal Sciences Container Image

If you do choose to use the latest image, then you want to consider how you will keep the agent up-to-date. If you have used the imagePullPolicy: Always option, then the latest image will be pulled on each startup and your agent will continue to get updates. To keep some consistency, you may instead choose to manually update the local cache by periodically forcing a pull instead of always pulling on startup.

docker pull signalsciences/sigsci-agent:latest

Then, use latest with imagePullPolicy: Never set in the configuration so that pulls are never done on startup (only manually as above):

- name: sigsci-agent
   image: signalsciences/sigsci-agent:latest
   imagePullPolicy: Never
   ...

Using a Versioned Signal Sciences Container Image

To use a specific version of the agent, then just replace latest with the agent version. You may also want to change imagePullPolicy: IfNotPresent in this case as the image should not change.

- name: sigsci-agent
   image: signalsciences/sigsci-agent:4.1.0
   imagePullPolicy: IfNotPresent
   ...

This will pull the specified agent version and cache it locally. If you use this method, then it is recommended that you parameterize the agent image, using Helm or similar, so that it is easier to update the agent images later on.

Using a Custom Tag for the Signal Sciences Container Image

It is also possible to apply a custom tag to a local agent image. To do this, pull the agent image (by version or use the latest), apply a custom tag, then use that custom tag in the configuration. You will want to specify imagePullPolicy: Never so that local images are only updated manually. You will need to periodically update the local image to keep the agent up-to-date.

For example:

docker pull signalsciences/sigsci-agent:latest
docker tag signalsciences/sigsci-agent:latest signalsciences/sigsci-agent:testing

Then use this image tag in the configuration:

- name: sigsci-agent
   image: signalsciences/sigsci-agent:testing
   imagePullPolicy: Never
...

Configuring the Signal Sciences Agent Container

Agent configuration is normally done via the environment. Most configuration options are available as environment variables. Environment variables names have the configuration option name all capitalized, prefixed with SIGSCI_ and any dashes (-) changed to underscores (_) (e.g., the max-procs option would become the SIGSCI_MAX_PROCS environment variable). For more details on what options are available, see the Agent Configuration documentation.

The sigsci-agent container has a few required options that need to be configured:

  • Agent credentials (ID and secret key)
  • A volume to write temporary files

Agent Credentials

The sigsci-agent credentials are configured with two environment variables. These variables must be set or the agent will not start.

  • SIGSCI_ACCESSKEYID: Identifies the site that the agent is configured against
  • SIGSCI_SECRETACCESSKEY: The shared secret key to authenticate and authorize the agent

The credentials can be found by following these steps:

  1. Log into the Signal Sciences console
  2. Go to Agents Configurations “Configurations Menu”
  3. On the Agents page click “View Agent Keys”
  4. Copy down the Access Key and Secret Key as this will be used later agent-keys “Agent Keys”

Because of the sensitive nature of these values, it is recommended to use the builtin secrets functionality of Kubernetes. With this configuration, the agent will pull the values from the secrets data instead of reading hardcoded the values into the deployment configuration. This also makes any desired agent credential rotation easier to manage by having to change them in only one place.

Using secrets via environment variables is done using the valueFrom option instead of the value option such as follows:

env:
 - name: SIGSCI_ACCESSKEYID
   valueFrom:
     secretKeyRef:
       # Update "my-site-name-here" to the correct site name or similar identifier
       name: sigsci.my-site-name-here
       key: accesskeyid
 - name: SIGSCI_SECRETACCESSKEY
   valueFrom:
     secretKeyRef:
       # Update "my-site-name-here" to the correct site name or similar identifier
       name: sigsci.my-site-name-here
       key: secretaccesskey

The secrets functionality keeps secrets in various stores in Kubernetes. This documentation uses the generic secret store in its examples, however any equivalent store can be used. Agent secrets can be added to the generic secret store with something like the following YAML:

apiVersion: v1
kind: Secret
metadata:
  name: sigsci.my-site-name-here
stringData:
  accesskeyid: 12345678-abcd-1234-abcd-1234567890ab
  secretaccesskey: abcdefg_hijklmn_opqrstuvwxy_z0123456789ABCD

This can also be created from the command line with kubectl such as with the following:

kubectl create secret generic sigsci.my-site-name-here \
  --from-literal=accesskeyid=12345678-abcd-1234-abcd-1234567890ab \
  --from-literal=secretaccesskey=abcdefg_hijklmn_opqrstuvwxy_z0123456789ABCD

See the documentation on secrets for more details.

Agent Temporary Volume

For added security, it is recommended that the sigsci-agent container be executed with the root filesystem mounted read only. The agent, however, still needs to write some temporary files such as the socket file for RPC communication and some periodically updated files such as GeoIP data. To accomplish this with a read only root filesystem, there needs to be a writeable volume mounted. This writeable volume can also be shared to expose the RPC socket file to other containers in the same pod. The recommended way of creating a writeable volume is to use the builtin emptyDir volume type. Typically this is just configured in the volumes section of a deployment.

volumes:
 - name: sigsci-tmp
   emptyDir: {}

Containers would then typically mount this volume at /sigsci/tmp:

volumeMounts:
 - name: sigsci-tmp
   mountPath: /sigsci/tmp

The default in the official agent container image is to have the temporary volume mounted at /sigsci/tmp. If this needs to be moved for the agent container, then the following agent configuration options should also be changed from their defaults to match the new mount location:

  • rpc-address defaults to /sigsci/tmp/sigsci.sock
  • shared-cache-dir defaults to /sigsci/tmp/cache

Integrating the Signal Sciences Agent into Istio Service Mesh

Istio uses envoy proxy under its hood. Because of this, Istio can utilize the sigsci-agent in gRPC mode in the same you as with a generic envoy install. Installing and configuring the sigsci-agent are similar to a generic envoy install except the envoy proxy is automatically deployed as a sidecar. Envoy is then configured using Istio’s EnvoyFilter. Full Istio integration is only possible in Istio v1.3 or later due to the required extensions to EnvoyFilter.

To add Signal Sciences support to an Istio based application deployment:

  • Add the sigsci-agent container to the pod, configured in envoy gRPC listener mode
  • Add an emptyDir{} volume as a place for the sigsci-agent to write temporary data
  • Add an Istio EnvoyFilter for the app to allow the required envoy configuration to be injected into the generated istio-proxy config

Add the Signal Sciences Agent as an Envoy gRPC Service

...
      containers:
      # Example helloworld app running on port 8000 without sigsci configured
      - name: helloworld
        image: signalsciences/example-helloworld:latest
        imagePullPolicy: IfNotPresent
        args:
        # Address for the app to listen on
        - localhost:8080
        ports:
        - containerPort: 8080
      # Signal Sciences Agent running in envoy gRPC mode (SIGSCI_ENVOY_GRPC_ADDRESS configured)
      - name: sigsci-agent
        image: signalsciences/sigsci-agent:latest
        imagePullPolicy: IfNotPresent
        # Configure the agent to use envoy gRPC on port 9999
        env:
        - name: SIGSCI_ACCESSKEYID
          valueFrom:
            secretKeyRef:
              # This secret needs added (see docs on sigsci secrets)
              name: sigsci.my-site-name-here
              key: accesskeyid
        - name: SIGSCI_SECRETACCESSKEY
          valueFrom:
            secretKeyRef:
              # This secret needs added (see docs on sigsci secrets)
              name: sigsci.my-site-name-here
              key: secretaccesskey
        # Configure the envoy to expect response data (if using a gRPC access log config for envoy)
        - name: SIGSCI_ENVOY_EXPECT_RESPONSE_DATA
          value: "1"
        # Configure the envoy gRPC listener address on any unused port
        - name: SIGSCI_ENVOY_GRPC_ADDRESS
          value: localhost:9999
        ports:
        - containerPort: 9999
        securityContext:
          # The sigsci-agent container should run with its root filesystem read only
          readOnlyRootFilesystem: true

Adding the Signal Sciences Agent Temp Volume Definition to the Deployment

Add the agent temp volume needs to be defined for use by the other containers in the pod. This just uses the builtin emptyDir: {} volume type.

...
      volumes:
      # Define a volume where sigsci-agent will write temp data and share the socket file,
      # which is required with the root filesystem is mounted read only
      - name: sigsci-tmp
        emptyDir: {}

Adding the Istio EnvoyFilter Object to Inject the Required Envoy Config into the Istio Proxy

Istio has a feature rich way of customizing the envoy configuration for the istio-proxy. This is done via the EnvoyFilter object.

You will need to modify the EnvoyFilter metadata.name field and the spec.workloadSelector.labels.app field to be set to the application name below. Additional envoy configuration options are outlined in the envoy install guide. These sections are highlighted with comments in the example YAML.

Example example-helloworld_sigsci-envoyfilter.yaml:

# The following adds the required envoy configuration into the istio-proxy configuration
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  # This needs adjusted to be the app name protected by sigsci
  name: helloworld
spec:
  workloadSelector:
    labels:
      # This needs adjusted to be the app name protected by sigsci
      app: helloworld
 
  # Patch the envoy configuration, adding in the required sigsci config
  configPatches:
 
  # Adds the ext_authz HTTP filter for the sigsci-agent ext_authz API
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        name: virtualInbound
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: INSERT_BEFORE
      value:
        # Configure the envoy.ext_authz here:
        name: envoy.ext_authz
        config:
          grpc_service:
            # NOTE: *SHOULD* use envoy_grpc as ext_authz can use dynamic clusters and has connection pooling
            envoy_grpc:
              cluster_name: sigsci-agent-grpc
            timeout: 0.2s
          failure_mode_allow: true
          with_request_body:
            max_request_bytes: 8192
            allow_partial_message: true
 
  # Adds the access_log entry for the sigsci-agent http_grpc_access_log API
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        name: virtualInbound
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
    patch:
      operation: MERGE
      value:
        name: "envoy.http_connection_manager"
        typed_config:
          "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
          access_log:
          # Configure the envoy.http_grpc_access_log here:
          - name: "envoy.http_grpc_access_log"
            config: 
              common_config:
                log_name: "sigsci-agent-grpc"
                grpc_service:
                  # NOTE: *MUST* use google_grpc as envoy_grpc cannot handle a dynamic cluster for ALS (yet)
                  google_grpc:
                    # The address *MUST* be 127.0.0.1 so that communication is intra-pod
                    # Configure the sigsci-agent port number here:
                    target_uri: 127.0.0.1:9999
                    stat_prefix: "sigsci-agent"
                  timeout: 0.2s
              additional_request_headers_to_log:
              # These are required:
              - "x-sigsci-request-id"
              - "x-sigsci-waf-response"
              # These are additional you want recorded:
              - "accept"
              - "content-type"
              - "content-length"
              additional_response_headers_to_log:
              # These are additional you want recorded:
              - "date"
              - "server"
              - "content-type"
              - "content-length"
 
  # Adds a dynamic cluster for the sigsci-agent via CDS for sigsci-agent ext_authz API
  - applyTo: CLUSTER
    patch:
      operation: ADD
      value:
        name: sigsci-agent-grpc
        type: STRICT_DNS
        connect_timeout: 0.5s
        http2_protocol_options: {}
        load_assignment:
          cluster_name: sigsci-agent-grpc
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    # The address *MUST* be 127.0.0.1 so that communication is intra-pod
                    address: 127.0.0.1
                    # Configure the agent port here:
                    port_value: 9999

The application can then be deployed as you normally would with Istio. Something like:

$ istioctl kube-inject -f example-helloworld-sigsci.yaml | kubectl apply -f -
service/helloworld created
deployment.apps/helloworld created
$ kubectl apply -f example-helloworld-sigsci_envoyfilter.yaml
envoyfilter.networking.istio.io/helloworld created
$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
helloworld-7954bb57bc-pfr22   3/3     Running   2          33s
$ kubectl get pod helloworld-7954bb57bc-pfr22 -o jsonpath='{.spec.containers[*].name}'
helloworld sigsci-agent istio-proxy
$ kubectl logs helloworld-7954bb57bc-pfr22 sigsci-agent | head
2019/10/01 21:04:57.540047 Signal Sciences Agent 4.0.0 starting as user sigsci with PID 1, Max open files=1048576, Max data size=unlimited, Max address space=unlimited, Max stack size=8388608
2019/10/01 21:04:57.541987 =====================================================
2019/10/01 21:04:57.542028 Agent:    helloworld-7954bb57bc-pfr22
2019/10/01 21:04:57.542034 System:   alpine 3.9.4 (linux 4.9.184-linuxkit)
2019/10/01 21:04:57.542173 Memory:   1.672G / 3.854G RAM available
2019/10/01 21:04:57.542187 CPU:      6 MaxProcs / 12 CPU cores available
2019/10/01 21:04:57.542257 =====================================================
2019/10/01 21:04:57.630755 Envoy gRPC server on 127.0.0.1:9999 starting

You will notice that there are three containers running in the pod (app=helloworld, sigsci-agent, and the istio-proxy).