Storj DCS Object Storage

Storj DCS can be used as an origin for public and private Storj buckets via the Storj DCS S3 Gateway. Built on the Storj Network, Storj DCS is a decentralized object storage service that is S3 compatible and end-to-end encrypted by default.

Prerequisites

Before adding Storj DCS as an origin for Fastly services, you will need to create a Storj DCS account, project and access credentials, and a bucket that will serve as your origin.

Using Storj DCS as an origin

To use Storj DCS as an origin and make your Storj bucket available through Fastly via the Storj DCS S3 Gateway, follow the steps below.

Creating a new service

Follow the instructions for creating a new service.

1. When you create the new domain and the new host:
   • In the Domain Name field on the Create a domain page, enter the hostname you want to use as the URL (e.g., cdn.example.com).
   • In the Hosts field on the Origins page, enter the IP address or hostname of your Storj DCS Gateway Endpoint using the format <BUCKET>.gateway.<REGION>.storjshare.io including your bucket (e.g., origin.gateway.us1.storjshare.io).
2. When you edit the host details on the Edit this host page:
   • In the Name field, enter any descriptive name for your service if you haven't already done so.
   • In the Address field, ensure you've entered the IP address or hostname of your Storj DCS Gateway Endpoint. You entered this information during host creation.
3. When you edit the Transport Layer Security (TLS) area information for your host:
   • If you've set up TLS for your Storj DCS S3 Gateway, leave the Enable TLS? default set to Yes to secure the connection between Fastly and your origin.
   • Under the SNI hostname field, select the checkbox to Match the SNI hostname to the Certificate hostname. The address you entered during host creation appears.
   • In the Certificate hostname field, enter the IP address or hostname of your Storj DCS S3 Gateway.

Testing your results

By default, we create a DNS mapping called yourdomain.global.prod.fastly.net. In the example above, it would be cdn.example.com.global.prod.fastly.net. Create a DNS alias for the domain name you specified (e.g., CNAME cdn.example.com to global-nossl.fastly.net).

Fastly will cache any content without an explicit Cache-Control header for 1 hour. You can verify whether you are sending any cache headers using curl. For example:

$ curl -I https://cdn.example.com

Accept-Ranges: bytes
Content-Length: 250
Content-Type: application/xml
Server: MinIO/DEVELOPMENT.GOGET
Vary: Origin
Date: Wed, 07 Oct 2020 02:31:27 GMT

In this example, no Cache-Control headers are set so the default TTL will be applied.

Enhanced cache control

If you need more control over how different types of assets are cached (e.g., JavaScript files, images), check out our documentation on cache freshness.

Using a Storj DCS bucket for origin hosting

To use a Storj DCS S3 Gateway as an origin with Fastly, you must implement version 4 of Amazon’s header-based authentication. You can do this using custom VCL. Start by obtaining the following information from AWS:

Once you have this information, you can configure your Fastly service to authenticate against your private bucket using header authentication by calculating the appropriate header value in VCL.

Start by creating a regular VCL snippet. Give it a meaningful name, such as Storj DCS Origin. When you create the snippet, select within subroutine to specify its placement and choose miss as the subroutine type. Then, populate the VCL field with the following code (be sure to change specific values as noted to ones relevant to your own bucket):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

declare local var.accessKey STRING;
declare local var.secretKey STRING;
declare local var.storjBucket STRING;
declare local var.storjGateway STRING;
declare local var.region STRING;
declare local var.canonicalHeaders STRING;
declare local var.signedHeaders STRING;
declare local var.canonicalRequest STRING;
declare local var.canonicalQuery STRING;
declare local var.stringToSign STRING;
declare local var.dateStamp STRING;
declare local var.signature STRING;
declare local var.scope STRING;


set var.accessKey = "YOUR_ACCESS_KEY";   # Change this value to your own data
set var.secretKey = "YOUR_SECRET_KEY";   # Change this value to your own data
set var.storjBucket = "YOUR_BUCKET_NAME";   # Change this value to your own data
set var.storjGateway = "STORJ-DCS_GATEWAY";   # Change this value to your own data
set var.region = "decentralized";


if (req.method == "GET" && !req.backend.is_shield) {

  set bereq.http.x-amz-content-sha256 = digest.hash_sha256("");
  set bereq.http.x-amz-date = strftime({"%Y%m%dT%H%M%SZ"}, now);
  set bereq.http.host = var.storjBucket "." var.storjGateway;
  set bereq.url = querystring.remove(bereq.url);
  set bereq.url = regsuball(urlencode(urldecode(bereq.url.path)), {"%2F"}, "/");
  set var.dateStamp = strftime({"%Y%m%d"}, now);
  set var.canonicalHeaders = ""
    "host:" bereq.http.host LF
    "x-amz-content-sha256:" bereq.http.x-amz-content-sha256 LF
    "x-amz-date:" bereq.http.x-amz-date LF
  ;
  set var.canonicalQuery = "";
  set var.signedHeaders = "host;x-amz-content-sha256;x-amz-date";
  set var.canonicalRequest = ""
    "GET" LF
    bereq.url.path LF
    var.canonicalQuery LF
    var.canonicalHeaders LF
    var.signedHeaders LF
    digest.hash_sha256("")
  ;

  set var.scope = var.dateStamp "/" var.region "/s3/aws4_request";


  set var.stringToSign = ""
    "AWS4-HMAC-SHA256" LF
    bereq.http.x-amz-date LF
    var.scope LF
    regsub(digest.hash_sha256(var.canonicalRequest),"^0x", "")
  ;

  set var.signature = digest.awsv4_hmac(
    var.secretKey,
    var.dateStamp,
    var.region,
    "s3",
    var.stringToSign
  );


 set bereq.http.Authorization = "AWS4-HMAC-SHA256 "
    "Credential=" var.accessKey "/" var.scope ", "
    "SignedHeaders=" var.signedHeaders ", "
    "Signature=" + regsub(var.signature,"^0x", "")
  ;

  unset bereq.http.Accept;
  unset bereq.http.Accept-Language;
  unset bereq.http.User-Agent;
  unset bereq.http.Fastly-Client-IP;
}