Agent scaling and running as a service

If the sigsci-agent is installed as a sidecar into a pod, the agent will scale however you have chosen to scale the application in the pod. This is the recommended method of installing the agent as it does not require a different means of scaling your application. However, for some installations the agent may need to be scaled at a different rate than the application. In these cases you can install the agent as a service to be used by the application pods. However, there are limitations when installing the agent as a service.

Limitations

  • The sigsci-agent can only be configured for a single site. This means that any agent service would only be able to send to a single site. All of the agents in the service will have the same configuration.
  • The sigsci-agent keeps some request states when processing the responses. This means that the agent that processed the request data needs to be the same agent that processes the response data. Therefore, load balancing agents require affinity, which makes the service more complex to scale.
  • Using the sigsci-agent as a service means configuring the communication channel as TCP instead of a Unix domain socket and this is slightly less efficient.

Installing the Signal Sciences agent as a service

The sigsci-agent can be installed as a service, but care must be taken when configuring the service due the above limitations. The service will be tied to a single site. If you will have multiple sites, then you should name the service based on the site name. To scale the service, it must be configured so that the same agent will process both the request and response data for a transaction. To do this, you need to configure the service to use affinity based on the pod that is sending data to the agent. This is done by setting the affinity to use the Client IP.

Below is an example service tied to a site named my-site-name using Client IP affinity:

1apiVersion: v1
2kind: Service
3metadata:
4 name: sigsci-agent-my-site-name
5 labels:
6 app: sigsci-agent-my-site-name
7spec:
8 ports:
9 # Port names and numbers are arbitrary
10 # 737 is the default RPC port
11 # 8000 may be more appropriate for gRPC used with Envoy
12 - name: rpc
13 port: 737
14 targetPort: 737
15 selector:
16 app: sigsci-agent-my-site-name
17 sessionAffinity: ClientIP
18 sessionAffinityConfig:
19 clientIP:
20 timeoutSeconds: 60

The service must then be backed by a deployment with any number of replicas. The sigsci-agent container must be configured as in a typical sidecar install, but must use TCP instead of a shared Unix domain socket. This is done by setting the SIGSCI_RPC_ADDRESS configuration option. Note that if using this with Envoy, you must use SIGSCI_ENVOY_GRPC_ADDRESS instead.

Example deployment corresponding with the service above:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: sigsci-agent-my-site-name
5 labels:
6 app: sigsci-agent-my-site-name
7spec:
8 replicas: 2
9 selector:
10 matchLabels:
11 app: sigsci-agent-my-site-name
12 template:
13 metadata:
14 labels:
15 app: sigsci-agent-my-site-name
16 spec:
17 containers:
18 - name: sigsci-agent
19 image: signalsciences/sigsci-agent:latest
20 imagePullPolicy: IfNotPresent
21 env:
22 - name: SIGSCI_ACCESSKEYID
23 valueFrom:
24 secretKeyRef:
25 name: sigsci.my-site-name
26 key: accesskeyid
27 - name: SIGSCI_SECRETACCESSKEY
28 valueFrom:
29 secretKeyRef:
30 name: sigsci.my-site-name
31 key: secretaccesskey
32 # Use RPC via TCP instead of default Unix Domain Socket
33 - name: SIGSCI_RPC_ADDRESS
34 value: "0.0.0.0:737"
35 # Use all available resources.limits.cpu cores
36 - name: SIGSCI_MAX_PROCS
37 value: "100%"
38 securityContext:
39 readOnlyRootFilesystem: true
40 volumeMounts:
41 - name: sigsci-tmp
42 mountPath: /sigsci/tmp
43 # Set CPU resource limits (required for autoscaling)
44 resources:
45 limits:
46 cpu: 4
47 requests:
48 cpu: 1
49 volumes:
50 - name: sigsci-tmp
51 emptyDir: {}

The above example will deploy two sigsci-agent pods for the sigsci-agent-my-site-name service to use for the my-site-name site. Each agent will see up to 4 CPU cores, requiring resources for at least one core.

Each application pod must then have its module configured to send to a sigsci-agent at the service name and port defined by the service. In this example the module would be configured to sent to host sigsci-agent-my-site-name and port 737. These values are defined by the service as well as the SIGSCI_RPC_ADDRESS configuration option (or SIGSCI_ENVOY_GRPC_ADDRESS if Envoy is being used).

As for scaling, each pod that connects to this service will be assigned a sigsci-agent running in the service and affinity will be locked to this agent. If the agent is then updated or otherwise removed from the service (such as due to an autoscaling down event) the agent will be reassigned to the client application pod. Because of how agents are assigned to pods with affinity, the maximum number of active agents will not be more than the number of pods connecting to the service. This should be considered when determining the number of replicas and autoscaling parameters.

The deployment can be autoscaled. As an example, it is possible to autoscale with a Horizontal Pod Autoscaler via kubectl autoscale. In the example below, the deployment will use a minimum of 2 agents and be scaled up to 6 agents whenever the overall CPU usage reaches 60%. Note again, however, that all of these agents will only be handling a single site.

$ kubectl autoscale deployment sigsci-agent-my-site-name --cpu-percent=60 --min=2 --max=6

The status of the Horizontal Pod Autoscaler can be viewed via the kubectl get hpa command:

$ kubectl get hpa

That command will produce the following output:

NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
sigsci-agent-my-site-name Deployment/sigsci-agent-my-site-name 42%/60% 2 6 2 53m42s

There are some limitations to this type of scaling. When scaling (by manually setting the replica number or autoscaling), the sigsci-agent pod count will change for the service. When an agent is added, new connections to the service may get assigned affinity to new agent pods, but note that application pods that already have their affinity set to a specific agent pod will not be rebalanced unless the service setting for the affinity timeout (sessionAffinityConfig.clientIP.timeoutSeconds) is hit. Because of this, this scaling works best when the application pods are also scaled so that new application pods will get balanced to new agent pods. Similarly, when an agent pod is removed from the service due to scaling down, the application pods that were assigned to this agent will be reassigned to another agent and affinity set. When scaling back up, these will not get rebalanced. If this occurs often, then you may consider reducing the affinity timeout (sessionAffinityConfig.clientIP.timeoutSeconds) to allow for rebalancing if there is some idle time.

Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.