Using DuckDB with Fastly Object Storage

Fastly Object Storage works with the S3-compatible API to store and access large files from Fastly. This same API can be used with DuckDB, the fast, portable open-source, in-process analytical database.

Prerequisites

To remotely load files from Fastly Object Storage into DuckDBextension, you must install the httpfs DuckDBextension by running the INSTALL SQL command. This only needs to be run once. DuckDB may also try to auto-install the extension.

You will also need to create an Object Storage access key used to authenticate against the S3-compatible API.

Using DuckDB with Fastly Object Storage

To use DuckDB with Fastly Object Storage, first create an S3 secret in DuckDB, setting the CREATE SECRET parameters as follows:

• KEY_ID and SECRET: enter the appropriate values generated from the Object Storage access key you created.
• URL_STYLE: enter path.
• REGION: enter the Fastly data storage region you want to use (e.g. eu-central).
• ENDPOINT: enter <region>.object.fastlystorage.app where <region> matches the region value above.

Example:

1
2
3
4
5
6
7
8

CREATE SECRET my_secret (
    TYPE S3,
    KEY_ID 'AKIAIOSFODNN7EXAMPLE',
    SECRET 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY',
    URL_STYLE 'path',
    REGION 'us-east',
    ENDPOINT 'us-east.object.fastlystorage.app' -- see note below
);

After setting up the Fastly Object Storage credentials, you can query the data using DuckDB's built-in methods, such as read_csv or read_parquet:

SELECT * FROM 's3://⟨fastly_bucket_name⟩/(file).csv';
SELECT * FROM read_parquet('s3://⟨fastly_bucket_name⟩/⟨file⟩.parquet');