Tardigrade Decentralized Object Cloud Storage

      Last updated November 05, 2020

    Tardigrade Cloud Storage can be used as an origin for public and private Tardigrade buckets via the Tardigrade S3 Gateway. Built on the Storj Network, Tardigrade is a decentralized object storage service that is S3 compatible and end-to-end encrypted by default.

    Prerequisites

    Before adding Tardigrade as an origin for Fastly services, you will need to:

    1. Create a Tardigrade account.
    2. Create a project in your Tardigrade account.
    3. Create an API key for the project.
    4. Set up Tardigrade Uplink CLI and create an access grant for the project.
    5. Create a Tardigrade bucket that will serve as your origin.
    6. Set up a Tardigrade S3 gateway that Fastly can use as an origin server.

    Using Tardigrade Cloud Storage as an origin

    To use Tardigrade Cloud Storage as an origin and make your Tardigrade bucket available through Fastly via the Tardigrade S3 Gateway, follow the steps below.

    Creating a new service

    Follow the instructions for creating a new service.

    1. When you create the new domain and the new host:
      • In the Domain Name field on the Create a domain page, enter the hostname you want to use as the URL (e.g., cdn.example.com).
      • In the Hosts field on the Origins page, enter the IP address or hostname of your Tardigrade S3 Gateway.
    2. When you edit the host details on the Edit this host page:
      • In the Name field, enter any descriptive name for your service if you haven't already done so.
      • In the Address field, ensure you've entered the IP address or hostname of your Tardigrade S3 Gateway. You entered this information during host creation.
    3. When you edit the Transport Layer Security (TLS) area information for your host:
      • If you've set up TLS for your Tardigrade S3 Gateway, leave the Enable TLS? default set to Yes to secure the connection between Fastly and your origin.
      • Under the SNI hostname field, select the checkbox to Match the SNI hostname to the Certificate hostname. The address you entered during host creation appears.
      • In the Certificate hostname field, enter the IP address or hostname of your Tardigrade S3 Gateway.

    Testing your results

    By default, we create a DNS mapping called yourdomain.global.prod.fastly.net. In the example above, it would be cdn.example.com.global.prod.fastly.net. Create a DNS alias for the domain name you specified (e.g., CNAME cdn.example.com to global-nossl.fastly.net).

    Fastly will cache any content without an explicit Cache-Control header for 1 hour. You can verify whether you are sending any cache headers using cURL. For example:

    1
    2
    3
    4
    5
    6
    7
    8
    
    $ curl -I https://cdn.example.com
    
    Accept-Ranges: bytes
    Content-Length: 250
    Content-Type: application/xml
    Server: MinIO/DEVELOPMENT.GOGET
    Vary: Origin
    Date: Wed, 07 Oct 2020 02:31:27 GMT
    

    In this example, no cache control headers are set so the default TTL will be applied.

    Enhanced cache control

    If you need more control over how different types of assets are cached (e.g., Javascript files, images), check out the Amazon S3 configuration information in our Cache Control tutorial.

    Using a Tardigrade S3 Gateway private bucket

    To use a Tardigrade S3 Gateway as an origin with Fastly, you must implement version 4 of Amazon’s header-based authentication. You can do this using custom VCL. Start by obtaining the following information from AWS:

    Item Description
    Bucket name The name of your private bucket. When you download items from your bucket, this is the string listed in the URL path or hostname of each object.
    Access key The access key string for an IAM account that has at least read permission on the bucket.
    Secret key The secret access key paired with the access key above.

    Once you have this information, you can configure your Fastly service to authenticate against your private bucket using header authentication by calculating the appropriate header value in VCL.

    Start by creating a regular VCL snippet. Give it a meaningful name, such as Storj protected origin. When you create the snippet, select within subroutine to specify its placement and choose miss as the subroutine type. Then, populate the VCL field with the following code (be sure to change specific values as noted to ones relevant to your own bucket):

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    
    declare local var.accessKey STRING;
    declare local var.secretKey STRING;
    declare local var.storjBucket STRING;
    declare local var.storjGateway STRING;
    declare local var.region STRING;
    declare local var.canonicalHeaders STRING;
    declare local var.signedHeaders STRING;
    declare local var.canonicalRequest STRING;
    declare local var.canonicalQuery STRING;
    declare local var.stringToSign STRING;
    declare local var.dateStamp STRING;
    declare local var.signature STRING;
    declare local var.scope STRING;
    
    
    set var.accessKey = "YOUR_ACCESS_KEY";   # Change this value to your own data
    set var.secretKey = "YOUR_SECRET_KEY";   # Change this value to your own data
    set var.storjBucket = "YOUR_BUCKET_NAME";   # Change this value to your own data
    set var.storjGateway = "YOUR_GATEWAY_HOST";   # Change this value to your own data
    set var.region = "decentralized";
    
    
    if (req.method == "GET" && !req.backend.is_shield) {
    
      set bereq.http.x-amz-content-sha256 = digest.hash_sha256("");
      set bereq.http.x-amz-date = strftime({"%Y%m%dT%H%M%SZ"}, now);
      set bereq.http.host = var.storjBucket var.storjGateway;
      set bereq.url = querystring.remove(bereq.url);
      set bereq.url = regsuball(urlencode(urldecode(bereq.url.path)), {"%2F"}, "/");
      set var.dateStamp = strftime({"%Y%m%d"}, now);
      set var.canonicalHeaders = ""
    	"host:" bereq.http.host LF
    	"x-amz-content-sha256:" bereq.http.x-amz-content-sha256 LF
    	"x-amz-date:" bereq.http.x-amz-date LF
      ;
      set var.canonicalQuery = "";
      set var.signedHeaders = "host;x-amz-content-sha256;x-amz-date";
      set var.canonicalRequest = ""
    	"GET" LF
    	bereq.url.path LF
    	var.canonicalQuery LF
    	var.canonicalHeaders LF
    	var.signedHeaders LF
    	digest.hash_sha256("")
      ;
    
      set var.scope = var.dateStamp "/" var.region "/s3/aws4_request";
    
    
      set var.stringToSign = ""
    	"AWS4-HMAC-SHA256" LF
    	bereq.http.x-amz-date LF
    	var.scope LF
    	regsub(digest.hash_sha256(var.canonicalRequest),"^0x", "")
      ;
    
      set var.signature = digest.awsv4_hmac(
    	var.secretKey,
    	var.dateStamp,
    	var.region,
    	"s3",
    	var.stringToSign
      );
    
    
     set bereq.http.Authorization = "AWS4-HMAC-SHA256 "
    	"Credential=" var.accessKey "/" var.scope ", "
    	"SignedHeaders=" var.signedHeaders ", "
    	"Signature=" + regsub(var.signature,"^0x", "")
      ;
    
      unset bereq.http.Accept;
      unset bereq.http.Accept-Language;
      unset bereq.http.User-Agent;
      unset bereq.http.Fastly-Client-IP;
    }
    
    Back to Top

    Additional resources: