Extracting your data

Signal Sciences stores requests that contain attacks and anomalies, with some qualifications. If you would like to extract this data in bulk for ingestion into your own systems, we offer a request feed API endpoint which makes available a feed of recent data, suitable to be called by (for example) an hourly cron.

This functionality is typically used by SOC teams to automatically import data into SIEMs such as Splunk, ELK, and other commercial systems.

Data extraction vs searching

We have a separate API endpoint for searching request data. Its use case is for finding requests that meet certain criteria, as opposed to bulk data extraction:

Time span restrictions

The following restrictions are in effect when using this endpoint:

• The until parameter has a maximum of five minutes in the past. This is to allow our data pipeline sufficient time to process incoming requests - see below.
• The from parameter has a minimum value of 24 hours and five minutes in the past.
• Both the from and until parameters must fall on full minute boundaries.
• Both the from and until parameters require Unix timestamps with second level detail (e.g., 1445437680).

Delayed data

A five-minute delay is enforced to build in time to collect and aggregate data across all of your running agents, and then ingest, analyze, and augment the data in our systems. Our five-minute delay is a tradeoff between data that is both timely and complete.

Pagination

This endpoint returns data 1,000 requests at a time. If the time span specified contains more than 1,000 requests, a next url will be provided to retrieve the next batch. Each next url is valid for one minute from the time it's generated.

Sort order

As a result of our data warehousing implementation, the data you get back from this endpoint will be complete for the time span specified, but is not guaranteed to be sorted. Once all data for the given time span has been accumulated, it can be sorted using the timestamp field, if necessary.

Rate limiting

Limits for concurrent connections to this endpoint:

• Two per site
• Five per corp

Example usage

A common way to use this endpoint is to set up a cron that runs at 5 minutes past each hour and fetches the previous full hour's worth of data. In the example below, we calculate the previous full hour's start and end timestamps and use them to call the API.

Python

1import sys, requests, os, calendar, json
2from datetime import datetime, timedelta
3
4if 'SIGSCI_EMAIL' not in os.environ or 'SIGSCI_TOKEN' not in os.environ or 'SIGSCI_CORP' not in os.environ:
5    print ("ERROR: You need to define SIGSCI_EMAIL, SIGSCI_TOKEN and SIGSCI_CORP environment variables")
6    print ("Please fix and run again. Exiting....")
7    sys.exit(1)
8
9# Initial setup
10api_host = 'https://dashboard.signalsciences.net'
11email = os.environ.get('SIGSCI_EMAIL')
12token = os.environ.get('SIGSCI_TOKEN')
13corp_name = os.environ.get('SIGSCI_CORP')
14# List of comma-delimited sites that you want to extract data from
15site_names = [ 'site123', 'site345' ]
16
17# Calculate UTC timestamps for the previous full hour
18# For example, if now is 9:05 AM UTC, the timestamps will be 8:00 AM and 9:00 AM
19until_time = datetime.utcnow().replace(minute=0, second=0, microsecond=0)
20from_time = until_time - timedelta(hours=1)
21until_time = calendar.timegm(until_time.utctimetuple())
22from_time = calendar.timegm(from_time.utctimetuple())
23
24# Set up Headers will use
25headers = {
26    'Content-type': 'application/json',
27    'Content-Encoding': 'gzip',
28    'x-api-user' : email,
29    'x-api-token': token
30}
31
32for site_name in site_names:
33
34    url = api_host + ('/api/v0/corps/%s/sites/%s/feed/requests?from=%s&until=%s' % (corp_name, site_name, from_time, until_time))
35    first = True
36
37    print ("{ \"site_name\": \"%s\", \"data\": [" % (site_name))
38
39    # Loop across all the data and output the data in one big JSON object
40    while True:
41        response_raw = requests.get(url, headers=headers)
42        if response_raw.status_code != 200:
43            sys.stderr.write("There was an error fetching requests for site_name=%s.\nURL=%s failed" % (site_name, url))
44            break
45
46        response = json.loads(response_raw.text)
47
48        for request in response['data']:
49            data = json.dumps(request)
50            if first:
51                first = False
52            else:
53                data = ',\n' + data
54            sys.stdout.write(data)
55
56        next_url = response['next']['uri']
57        if next_url == '':
58            break
59        url = api_host + next_url
60
61    print ("\n] }")