Storj DCS Object Storage

Storj DCS can be used as an origin for public and private Storj buckets via the Storj DCS S3 Gateway. Built on the Storj Network, Storj DCS is a decentralized object storage service that is S3 compatible and end-to-end encrypted by default.

Prerequisites

Before adding Storj DCS as an origin for Fastly services, you will need to create a Storj DCS account, project and access credentials, and a bucket that will serve as your origin.

Using Storj DCS as an origin

To use Storj DCS as an origin and make your Storj bucket available through Fastly via the Storj DCS S3 Gateway, follow the steps below.

Creating a new service

Follow the instructions for creating a new service.

  1. When you create the new domain and the new host:
    • In the Domain Name field on the Create a domain page, enter the hostname you want to use as the URL (e.g., cdn.example.com).
    • In the Hosts field on the Origins page, enter the IP address or hostname of your Storj DCS Gateway Endpoint using the format <BUCKET>.gateway.<REGION>.storjshare.io including your bucket (e.g., origin.gateway.us1.storjshare.io).
  2. When you edit the host details on the Edit this host page:
    • In the Name field, enter any descriptive name for your service if you haven't already done so.
    • In the Address field, ensure you've entered the IP address or hostname of your Storj DCS Gateway Endpoint. You entered this information during host creation.
  3. When you edit the Transport Layer Security (TLS) area information for your host:
    • If you've set up TLS for your Storj DCS S3 Gateway, leave the Enable TLS? default set to Yes to secure the connection between Fastly and your origin.
    • Under the SNI hostname field, select the checkbox to Match the SNI hostname to the Certificate hostname. The address you entered during host creation appears.
    • In the Certificate hostname field, enter the IP address or hostname of your Storj DCS S3 Gateway.

Testing your results

By default, we create a DNS mapping called yourdomain.global.prod.fastly.net. In the example above, it would be cdn.example.com.global.prod.fastly.net. Create a DNS alias for the domain name you specified (e.g., CNAME cdn.example.com to global-nossl.fastly.net).

Fastly will cache any content without an explicit Cache-Control header for 1 hour. You can verify whether you are sending any cache headers using curl. For example:

$ curl -I https://cdn.example.com
Accept-Ranges: bytes
Content-Length: 250
Content-Type: application/xml
Server: MinIO/DEVELOPMENT.GOGET
Vary: Origin
Date: Wed, 07 Oct 2020 02:31:27 GMT

In this example, no Cache-Control headers are set so the default TTL will be applied.

Enhanced cache control

If you need more control over how different types of assets are cached (e.g., JavaScript files, images), check out our documentation on cache freshness.

Using a Storj DCS bucket for origin hosting

To use a Storj DCS S3 Gateway as an origin with Fastly, you must implement version 4 of Amazon’s header-based authentication. You can do this using custom VCL. Start by obtaining the following information from AWS:

ItemDescription
Bucket nameThe name of your private bucket. When you download items from your bucket, this is the string listed in the URL path or hostname of each object.
Access keyThe access key string associated with a Storj DCS Access Grant that has at least read permissions on the bucket.
Secret keyThe secret access key paired with the access key above.

Once you have this information, you can configure your Fastly service to authenticate against your private bucket using header authentication by calculating the appropriate header value in VCL.

IMPORTANT

Consider leaving the Override host field for the origin blank in your service settings. This setting will override the host header from the snippets shown here and may invalidate the signature that authenticates the information being sent.

Start by creating a regular VCL snippet. Give it a meaningful name, such as Storj DCS Origin. When you create the snippet, select within subroutine to specify its placement and choose miss as the subroutine type. Then, populate the VCL field with the following code (be sure to change specific values as noted to ones relevant to your own bucket):

1declare local var.accessKey STRING;
2declare local var.secretKey STRING;
3declare local var.storjBucket STRING;
4declare local var.storjGateway STRING;
5declare local var.region STRING;
6declare local var.canonicalHeaders STRING;
7declare local var.signedHeaders STRING;
8declare local var.canonicalRequest STRING;
9declare local var.canonicalQuery STRING;
10declare local var.stringToSign STRING;
11declare local var.dateStamp STRING;
12declare local var.signature STRING;
13declare local var.scope STRING;
14
15
16set var.accessKey = "YOUR_ACCESS_KEY"; # Change this value to your own data
17set var.secretKey = "YOUR_SECRET_KEY"; # Change this value to your own data
18set var.storjBucket = "YOUR_BUCKET_NAME"; # Change this value to your own data
19set var.storjGateway = "STORJ-DCS_GATEWAY"; # Change this value to your own data
20set var.region = "decentralized";
21
22
23if (req.method == "GET" && !req.backend.is_shield) {
24
25 set bereq.http.x-amz-content-sha256 = digest.hash_sha256("");
26 set bereq.http.x-amz-date = strftime({"%Y%m%dT%H%M%SZ"}, now);
27 set bereq.http.host = var.storjBucket "." var.storjGateway;
28 set bereq.url = querystring.remove(bereq.url);
29 set bereq.url = regsuball(urlencode(urldecode(bereq.url.path)), {"%2F"}, "/");
30 set var.dateStamp = strftime({"%Y%m%d"}, now);
31 set var.canonicalHeaders = ""
32 "host:" bereq.http.host LF
33 "x-amz-content-sha256:" bereq.http.x-amz-content-sha256 LF
34 "x-amz-date:" bereq.http.x-amz-date LF
35 ;
36 set var.canonicalQuery = "";
37 set var.signedHeaders = "host;x-amz-content-sha256;x-amz-date";
38 set var.canonicalRequest = ""
39 "GET" LF
40 bereq.url.path LF
41 var.canonicalQuery LF
42 var.canonicalHeaders LF
43 var.signedHeaders LF
44 digest.hash_sha256("")
45 ;
46
47 set var.scope = var.dateStamp "/" var.region "/s3/aws4_request";
48
49
50 set var.stringToSign = ""
51 "AWS4-HMAC-SHA256" LF
52 bereq.http.x-amz-date LF
53 var.scope LF
54 regsub(digest.hash_sha256(var.canonicalRequest),"^0x", "")
55 ;
56
57 set var.signature = digest.awsv4_hmac(
58 var.secretKey,
59 var.dateStamp,
60 var.region,
61 "s3",
62 var.stringToSign
63 );
64
65
66 set bereq.http.Authorization = "AWS4-HMAC-SHA256 "
67 "Credential=" var.accessKey "/" var.scope ", "
68 "SignedHeaders=" var.signedHeaders ", "
69 "Signature=" + regsub(var.signature,"^0x", "")
70 ;
71
72 unset bereq.http.Accept;
73 unset bereq.http.Accept-Language;
74 unset bereq.http.User-Agent;
75 unset bereq.http.Fastly-Client-IP;
76}
Was this guide helpful?

Do not use this form to send sensitive information. If you need assistance, contact support. This form is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.