search close

Data Storage and Sampling

access_time Updated Mar 24, 2023

When our agent sends requests to our collectors, we store two types of data: timeseries data and individual request data.

Timeseries data

Timeseries data counts the number of signals (e.g., XSS, SQLi, 404s) observed per minute, while individual request data includes individual records of anonymized requests. Timeseries data powers graphs visible throughout the product, as well as metrics such as tallies of request types.

An example graph showing the number of injection attacks received over the last 24 hours, broken up by attack type.

Individual request data

While all timeseries data is stored and available in the product, a representative sample of individual request data is stored. Individual request data provides detailed information about specific requests, such as the originating IP address and request parameters:

A screenshot of the requests page with example requests.

What data does Signal Sciences store?

We store all timeseries data sent to our collectors (powering graphs and metrics throughout the product).

We store individual request data based on the type of signals that requests are tagged with or the way that custom rules are configured. Storage categories include:

  • All: all requests matching this storage category are stored and available for reference throughout the console.
  • Sampled: a random sample of requests matching this storage category will be stored and available for reference throughout the console.
  • Timeseries only: requests matching this storage category aren’t stored. Timeseries data for all signals tagged to the request will be stored and visible.
  • Not stored: requests matching this category aren’t stored.

Note: Timeseries-only data storage category is only available on agents 3.12 and above. Matching requests processed on earlier agents will be processed according to the Sampled data storage category.

Request signal type Description Storage category
Individual requests containing attack signals Any requests containing 1 or more attack signals (e.g., SQLi, XSS) All
Individual requests containing CVE signals Any requests containing 1 or more CVE signals applied by virtual patching rules All
Individual requests containing only anomaly signals Requests that contain only anomaly signals (e.g., 404, Tor traffic) but no attack or CVE signals Sampled
Individual requests containing custom signals Requests containing custom signals but no attack or CVE signals. See Custom Signals for more information about creating and using signals. Sampled
Individual requests containing only API or ATO templated rules signals, known as informational signals Requests which are tagged with only a specific set of API or ATO templated rules signals, and no custom, anomaly, attack, or CVE signals Timeseries only
Individual requests that aren’t tagged with a signal Requests containing no signals Not stored

Note: Any requests containing at least one attack or CVE signal will be stored, including requests that also have anomaly, informational, or custom signals.