search close

Envoy Proxy gRPC Authorization Mode

access_time Updated Jun 20, 2021

Overview

Support is available for the Envoy Proxy via builtin Envoy gRPC APIs implemented in the sigsci-agent running as a gRPC server. Envoy v1.11.0 or later is recommended, however, Envoy v1.8.0 or later is supported with limited functionality as noted in the documentation below.

Currently Envoy (as of v1.11) does not support a bidirectional gRPC API for inspecting traffic. There are instead two separate gRPC APIs available to inspect traffic. The External Authorization HTTP filter (envoy.ext_authz) gRPC API allows the request to be held while waiting inbound request inspection, which allows for a request to be blocked if required. An additional gRPC AccessLog Service gRPC API can then be used to inspect the outbound request data. Using these two APIs together with the sigsci-agent running as a gRPC server allows for inspection in both directions using only Envoy builtin APIs. This allows web application inspection without installing a module for every upstream application. In this case the sigsci-agent is acting as the module.

Request Allowed (normal) Processing

Request Allowed Processing

This is the flow for normal requests that the sigsci-agent allows through Envoy.

  1. Client request received by envoy and routed to the Envoy External Authorization (ext_authz) HTTP filter where request metadata is extracted for processing via the sigsci-agent.
  2. Request metadata is sent to the sigsci-agent via gRPC ext_authz API
  3. The sigsci-agent sends back an ‘allow request’ response allowing the request through the ext_authz HTTP filter to continue normal Envoy request processing.
  4. Request makes it through any additional HTTP filters to the Handler, which processes the request and generates the response.
  5. Request/Response metadata is extracted via the Envoy gRPC AccessLog Service (als) asynchronously for processing via the sigsci-agent.
  6. In parallel, additional metadata, such as response headers and the HTTP status code, is sent to the sigsci-agent via gRPC als API for further processing while the response data is sent back to the originating client.

Request Blocked Processing

Request Blocked Processing

This is the flow if the sigsci-agent blocks a request from being processed through Envoy.

  1. Client request received by envoy and routed to the Envoy External Authorization (ext_authz) HTTP filter where request metadata is extracted for processing via the sigsci-agent.
  2. Request metadata is sent to the sigsci-agent via gRPC ext_authz API
  3. The sigsci-agent sends back a ‘block request’ response, disallowing the request to continue being processed by the HTTP filter chain.
  4. This triggers the ext_authz filter to generate a HTTP 406 response, blocking the request from any further processing.

Signal Sciences Agent Configuration

The sigsci-agent is normally installed as a sidecar via kubernetes with a slightly different configuration than a normal install.

The sigsci-agent must be configured to run with an Envoy gRPC listener instead of the normal RPC listener. To do this, configure the Envoy gRPC listener via the envoy-grpc-address agent configuration option, which will then start instead of the default RPC listener.

Setting the configuration value in the sigsci-agent config file:

envoy-grpc-address = "0.0.0.0:8000"

Or setting the configuration value in the sigsci-agent environment:

SIGSCI_ENVOY_GRPC_ADDRESS=0.0.0.0:8000

Optionally, the sigsci-agent can be configured with TLS enabled. To do this, set the certificate and key files in the sigsci-agent configuration.

envoy-grpc-cert = "/path/to/cert.pem"
envoy-grpc-key = "/path/to/key.pem"

OR

SIGSCI_ENVOY_GRPC_CERT=/path/to/cert.pem
SIGSCI_ENVOY_GRPC_KEY=/path/to/key.pem

Additionally, it is recommended to enable response data processing. To do this, the sigsci-agent must be configured to expect response data from Envoy by setting the envoy-expect-response-data agent configuration option available in the sigsci-agent version 3.18.0 or later. By default response data is ignored in the sigsci-agent as this is an optional envoy configuration option in order to better support older versions of Envoy. If you are running Envoy v1.10 or higher, then you should enable this option.

Setting the configuration value in the sigsci-agent config file:

envoy-expect-response-data = 1

Or setting the configuration value in the sigsci-agent environment:

SIGSCI_ENVOY_EXPECT_RESPONSE_DATA=1

As of sigsci-agent version 3.24.0 and later, some aspects of inspection in the sigsci-agent can be configured, but generally should be left as the default. See inspection-* agent configuration for more details.

Envoy Configuration

Envoy must to be configured with an External Authorization HTTP filter (envoy.ext_authz) before the main handler filter to process request data and (optionally, though recommended) a gRPC AccessLog Service to process response data. To do this, multiple configuration items must to be added to the Envoy configuration: a cluster to handle the gRPC calls via the sigsci-agent, the envoy.ext_authz HTTP filter before the main handler, and the envoy.http_grpc_access_log service added to the access_log section of the HTTP listener filter if response data is to be enabled.

Adding the Signal Sciences Agent Cluster

A cluster must be added which is configured with the Envoy gRPC address used in the sigsci-agent configuration. Currently load balancing will not work correctly if response data is enabled as there is not a way to enable consistent hashing for gRPC services in envoy (yet), so it is recommended not to configure load balancing at this time unless only the envoy.ext_authz API is being used without response data inspection.

  clusters:
  - name: sigsci-agent-grpc
    connect_timeout: 0.2s
    type: strict_dns
    #lb_policy: LEAST_REQUEST
    http2_protocol_options: {}
    #tls_context: {}
    ### You can also use 'hosts' below, but this is deprecated
    load_assignment:
      cluster_name: sigsci-agent-grpc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: sigsci-agent
                port_value: 8000

The address is a resolvable hostname or IP for the sigsci-agent and the port_value must match that configured in the sigsci-agent configuration for the envoy-grpc-address option.

NOTE: The connect_timeout is the timeout to connect to the sigsci-agent (but not to process the data) and can be adjusted if required. The tls_context option must be defined if TLS is to be used. TLS can be configured in the sigsci-agent config via envoy-grpc-cert and envoy-grpc-key. If TLS is configured in the sigsci-agent, then just the empty tls_context must be configured (e.g., tls_context: {}) to let envoy know to connect via TLS. If certificate validation is desired, then validation_context must be configured in the tls_context to specify a trusted_ca filename to use for validation. As gRPC services are HTTP/2 based, the http2_protocol_options: {} option is required so that traffic is sent to the sigsci-agent cluster as HTTP/2.

Adding the Envoy External Authorization HTTP Filter

The listener must have an External Authorization HTTP filter (envoy.ext_authz) added before the main handler which points at the sigsci-agent cluster.

http_filters:
- name: envoy.ext_authz
  config:
    grpc_service:
      envoy_grpc:
        cluster_name: sigsci-agent-grpc
      timeout: 0.2s
    failure_mode_allow: true
    ### Include the request body data (Envoy v1.10+ only, see limitations and
    ### workarounds for older versions)
    with_request_body:
      # Maximum request body bytes buffered and sent to the sigsci-agent
      max_request_bytes: 8192
      # NOTE: If allow_partial_message is set false, then any request over
      # the above max bytes will fail with an HTTP "413 Payload Too Large"
      # so it is recommended to set this to true.
      allow_partial_message: true
- name: envoy.router
  config: {}

NOTE: failure_mode_allow: true is so that this will fail open, which is recommended. And timeout allows failing with the defined failure mode (true for fail open, false for fail closed) after a given time duration. Once this is done, all HTTP requests will be first sent to the envoy.ext_authz filter handled by the sigsci-agent cluster. The sigsci-agent will then process requests and deny auth with a 406 HTTP status code if the request is to be blocked or allow the request through to the next HTTP filter if it is allowed. Any additional HTTP request headers are also added to the request as they are in other modules.

Adding the Envoy gRPC AccessLog Service

NOTE: This is a recommended, but optional step. If it is configured in envoy, then the agent MUST also be configured to expect response data by setting the envoy-expect-response-data agent configuration option as noted in the Signal Sciences Agent Configuration section. The envoy External Authorization (envoy.ext_authz) HTTP Filter can only process request data. As the sigsci-agent needs the response data for full functionality, a gRPC AccessLog Service must be set up to send the response data to the sigsci-agent. To do this an access_log section must be added to the envoy configuration under the listener filter (typically under the envoy.http_connection_manager filter) if it does not already exist. If it does exist, then it must be appended to.

Refer to the access_log configuration option of the HTTP Connection Manager for more details. An envoy.http_grpc_access_log entry must be added here (in addition to any other existing access log entries).

Recommended Configuration (see Current Limitations for further customizations to minimize limitations):

access_log:
- name: envoy.http_grpc_access_log
  config:
    common_config:
      log_name: "sigsci-agent-grpc"
      grpc_service:
        envoy_grpc:
          cluster_name: sigsci-agent-grpc
        timeout: 0.2s
    additional_request_headers_to_log:
    # These sigsci-agent headers are required for correct processing:
    - "x-sigsci-request-id"
    - "x-sigsci-waf-response"
    # Optionally, additional headers can be added that should be recorded:
    - "accept"
    - "content-type"
    - "content-length"
    additional_response_headers_to_log:
    - "date"
    - "server"
    - "content-type"
    - "content-length"

Current Limitations

Here are the current limitations when using the sigsci-agent with Envoy Proxy. As support for Envoy Proxy improves in the future, these limitations will be addressed and should be reduced.

No request bodies are processed by default

Prior to Envoy v1.10.0, the Envoy External Authorization did not send the request body. In all versions of Envoy, the request body is not included in the ext_authz call by default and it will not be inspected by the sigsci-agent unless configured.

For Envoy v1.10.0 or higher, support to include the request body is built in to the envoy.ext_authz configuration and it is now possible to configure the with_request_body in this section of the Envoy configuration as noted above.

For Envoy v1.11.0 or higher, support was extended to be able to detect partial bodies more accurately.

For HTTP/2 (and gRPC) support envoy must be running a version later than v1.12.1. In envoy v1.10.0 - v1.12.1 envoy is not properly sending the request body using with_request_body. However, as of sigsci-agent v4.3.0, it is possible to work around this envoy limitation using Lua until an envoy upgrade is possible.

The following is an example Lua filter that can be used to pass on gRPC based bodies to the sigsci-agent for inspection (sigsci-agent v4.3.0+): To do this, the Lua HTTP filter (envoy.lua) HTTP filter can be configured before the envoy.ext_authz filter to add an internal x-sigsci-encoded-body header with this data. A small snippet of lua code must be added to extract the body and add it to the request as follows:

          http_filters:
          - name: envoy.lua
            config:
              inline_code: |
                -- Add a special header to pass the encoded body
                function envoy_on_request(req)
                  local len = 0
                  local reqbody
                  -- Determine the body length
                  local cl = req:headers():get("content-length")
                  if cl ~= nil then
                    len = tonumber(cl)
                  end
                  -- gRPC does not have a content-length header to limit the body before buffering
                  if len == 0 and req:headers():get("content-type") == "application/grpc" then
                    -- Triggers buffering
                    len = req:body():length()
                  end
                  -- Limit body length sent to the agent (adjust as needed)
                  if len > 0 and len <= 8192 then
                    -- Triggers buffering
                    reqbody = req:body():getBytes(0, len)
                    -- Encode the body for use in a header value
                    local enc, t = string.gsub(reqbody, "[^%w]", function(chr)
                      return string.format("%%%02X",string.byte(chr))
                    end)
                    req:headers():add("x-sigsci-encoded-body", enc)
                  end
                end
          - name: envoy.ext_authz
            config:
              grpc_service:
                envoy_grpc:
                  cluster_name: sigsci-agent-grpc
                timeout: 0.2s
              failure_mode_allow: true
#              with_request_body:
#                max_request_bytes: 8192
#                allow_partial_message: true
          - name: envoy.router
            config: {}

For older agents (before v4.3.0) an older workaround is available for passing the body if HTTP/2 support is not required. To do this, the Lua HTTP filter (envoy.lua) HTTP filter can be configured before the envoy.ext_authz filter to add an internal :body header with this data. A small snippet of lua code must be added to extract the body and add it to the request as follows:

NOTE: The following Lua workaround may cause 503 responses to be returned if HTTP/2 is enabled. If you must enable HTTP/2 support, then you should use envoy 1.10 or newer so that the body can be extracted using the native method via the with_request_body option instead of using this workaround. If you cannot use envoy v1.10.0 or greater, then it is recommended that you upgrade to sigsci-agent v4.3.0 or later and use the previous Lua workaround which will work in more cases.

Example of including the request body data if it is <= 8KB (adjust the limit if required):

          http_filters:
          - name: envoy.lua
            config:
              inline_code: |
                -- Add an internal :body header to pass the body if <= 8KB
                function envoy_on_request(req)
                  len = 0
                  cl = req:headers():get("content-length")
                  if cl ~= nil then
                    len = tonumber(cl)
                  end
                  if len > 0 and len <= 8192 then
                    reqbody = req:body():getBytes(0, len)
                    req:headers():add(":body", reqbody)
                  end
                end
          - name: envoy.ext_authz
            config:
              grpc_service:
                envoy_grpc:
                  cluster_name: sigsci-agent-grpc
              failure_mode_allow: true
          - name: envoy.router
            config: {}

No TLS handshake metadata is extracted

There is not currently a means for the sigsci-agent to see the TLS handshake metadata (e.g., cipher and protocol version) used in the originating request as this is not (yet) available in Envoy. Any TLS handshake metadata based signals will not be seen in the product for this site.

The following system signals are currently NOT supported due to this limitation:

  • WEAKTLS

Only minimal request headers are recorded by default if there were only response-based signals

If the request was inspected by the envoy.ext_authz filter and no signals were issued, then the response will be processed by the envoy.http_grpc_access_log service. If a signal is found in the response data, then only minimal request headers will be recorded with the signal due to the API not being sent all request headers by default. However, if additional request headers are desired to be recorded, then these should be added via the additional_request_headers_to_log option of the access_log configuration in envoy.

Currently these headers will automatically be added:

  • Host
  • User-Agent
  • Referer
  • X-Forwarded-For

Two sigsci-agent specific headers must be added. Additionally any additional request headers can be added explicitly via additional_request_headers_to_log:

additional_request_headers_to_log:
# These sigsci-agent headers are required for correct processing:
- "x-sigsci-request-id"
- "x-sigsci-waf-response"
# Optionally, additional headers can be added that should be recorded:
- "accept"
- "content-type"
- "content-length"
- "x-real-ip"

No response headers are processed by default

Similar to above with minimal request headers not being processed by the envoy.http_grpc_access_log service, there are no response headers sent to this API by default. Any headers that are desired to be recorded must be explicitly listed in the additional_response_headers_to_log option of the access_log configuration in envoy as there is not currently any means to wildcard this. The following are recommended.

additional_response_headers_to_log:
- "date"
- "server"
- "content-type"
- "content-length"

Next Steps

Explore other installation options: