Extracting your data

  Last updated 2024-08-22

IMPORTANT

This guide only applies to Next-Gen WAF customers with access to the Next-Gen WAF control panel.

Next-Gen WAF stores requests that contain attacks and anomalies, with some qualifications. If you would like to extract this data in bulk for ingestion into your own systems, we offer a request feed API endpoint which makes available a feed of recent data, suitable to be called by (for example) an hourly cron.

This functionality is typically used by security operation center (SOC) teams to automatically import data into security information and event management (SIEM) solutions such as Datadog, ELK, and other commercial systems.

Data extraction vs searching

We have a separate API endpoint for searching request data. Its use case is for finding requests that meet certain criteria, as opposed to bulk data extraction:

SearchingData Extraction
Search using full query syntaxReturns all requests, optionally filtered by signals
Limited to 1,000 requestsReturns all requests
Window: up to 7 days at a timeWindow: past 24 hours
Retention: 30 days24 hours

Time span restrictions

The following restrictions are in effect when using this endpoint:

  • The until parameter has a maximum of five minutes in the past. This is to allow our data pipeline sufficient time to process incoming requests - see below.
  • The from parameter has a minimum value of 24 hours and five minutes in the past.
  • Both the from and until parameters must fall on full minute boundaries.
  • Both the from and until parameters require Unix timestamps with second level detail (e.g., 1445437680).

Delayed data

A five-minute delay is enforced to build in time to collect and aggregate data across all of your running agents, and then ingest, analyze, and augment the data in our systems. Our five-minute delay is a tradeoff between data that is both timely and complete.

Pagination

This endpoint returns data either 1,000 requests at a time or by the size specified in the limit query parameter. If the time span specified contains more than 1,000 requests (default) or more than defined by the limit parameter, a next URL will be provided to retrieve the next batch. Each next URL is valid for one minute from the time it's generated.

Retrieved data can vary in size, sometimes greatly. To avoid exceeding URL size limitations, send the next parameter and its value as POST parameters in a POST request using a Content-Type of application/x-www-form-urlencoded.

Sort order

As a result of our data warehousing implementation, the data you get back from this endpoint will be complete for the time span specified, but is not guaranteed to be sorted. Once all data for the given time span has been accumulated, it can be sorted using the timestamp field, if necessary.

Rate limiting

Limits for concurrent connections to this endpoint:

  • Two per site (also known as a workspace)
  • Five per corp (also known as an account)

Example usage

A common way to use this endpoint is to set up a cron that runs at 5 minutes past each hour and fetches the previous full hour's worth of data. In the example below, we calculate the previous full hour's start and end timestamps and use them to call the API.

Python

1import requests
2import os
3import json
4import calendar
5from datetime import datetime, timedelta, timezone
6

7# Set up environment variables
8NGWAF_EMAIL = os.getenv('NGWAF_USER_EMAIL')
9NGWAF_TOKEN = os.getenv('NGWAF_TOKEN')
10NGWAF_CORP = os.getenv('CORP_NAME')
11NGWAF_SITE = os.getenv('SITE_NAME')
12

13if not NGWAF_EMAIL or not NGWAF_TOKEN or not NGWAF_CORP or not NGWAF_SITE:
14    raise EnvironmentError("Please set NGWAF_EMAIL, NGWAF_TOKEN, NGWAF_CORP, and NGWAF_SITE environment variables.")
15

16# Base URL for the API
17base_url = 'https://dashboard.signalsciences.net/api/v0'
18

19# Set up headers with authentication
20headers = {
21    'x-api-user': NGWAF_EMAIL,
22    'x-api-token': NGWAF_TOKEN
23}
24

25# Calculate UTC timestamps for the previous full hour
26until_time = datetime.now(timezone.utc).replace(minute=0, second=0, microsecond=0)
27from_time = until_time - timedelta(hours=1)
28until_time = calendar.timegm(until_time.utctimetuple())
29from_time = calendar.timegm(from_time.utctimetuple())
30

31# Set up the initial URL for the GET request
32get_url = f'{base_url}/corps/{NGWAF_CORP}/sites/{NGWAF_SITE}/feed/requests?from={from_time}&until={until_time}'
33

34# Debugging: print the URL and timestamps
35print(f"Fetching data from: {get_url}")
36print(f"from_time: {from_time}, until_time: {until_time}")
37

38def fetch_paginated_data(url):
39    data_list = []
40    while url:
41        # Make the initial GET request
42        response_raw = requests.get(url, headers=headers)
43        if response_raw.status_code != 200:
44            raise RuntimeError(f"Failed to fetch data from {url}. Status Code: {response_raw.status_code}")
45        
46        response = response_raw.json()
47        data_list.extend(response.get('data', []))
48        
49        next_uri = response.get('next', {}).get('uri', '')
50        if not next_uri:
51            break
52

53        # Extract the next parameter from the URI
54        next_value = next_uri.split('next=')[-1]
55

56        # Prepare the POST request for pagination
57        post_url = f'{base_url}/corps/{NGWAF_CORP}/sites/{NGWAF_SITE}/feed/requests'
58        post_data = {'next': next_value}
59        headers['Content-Type'] = 'application/x-www-form-urlencoded'  # Add the necessary header
60        post_response_raw = requests.post(post_url, headers=headers, data=post_data)
61        if post_response_raw.status_code != 200:
62            raise RuntimeError(f"Failed to fetch paginated data from {post_url}. Status Code: {post_response_raw.status_code}")
63        
64        post_response = post_response_raw.json()
65        data_list.extend(post_response.get('data', []))
66        
67        next_uri = post_response.get('next', {}).get('uri', '')
68        if not next_uri:
69            break
70

71    return data_list
72

73# Fetch data
74data = fetch_paginated_data(get_url)
75

76# Output the data or save to a file, etc.
77print(json.dumps(data, indent=4))
Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Fastly
© Fastly 2024