Skip to content

How to Host Your Data on AWS S3

Hosting your data on AWS S3 provides high performance and reliability for CartoScope. Users could open their own S3 bucket and upload their data to it.

While users could set up their own S3 bucket and upload their data to it manually, CartLoader provides a more automated way to upload your data to AWS S3.

This is guide shows how to use CartLoader to upload your data to your AWS S3 bucket in a more automated way. Then it could be loaded and used in CartoScope.

1. Set up AWS S3 Bucket

Follow the AWS S3 Official Documentation:

2. Set Up AWS CLI

3. Configure Bucket Policy

Your S3 bucket needs to be publicly accessible. Go to "Permissions" tab of your bucket and click "Edit" button to edit the bucket policy as follows

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
}

Note that you need to replace your-bucket-name with your actual bucket name.

4. Configure Cross-Origin Resource Sharing (CORS)

Cross-Origin Resource Sharing (CORS) is a security feature that allows web applications to access resources from different origins. This needs to be configured for CartoScope to access your data. Go to "Permissions" tab of your bucket and click "Edit" button to edit the CORS configuration as follows

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [
            "Content-Range",
            "Content-Length",
            "ETag"
        ],
        "MaxAgeSeconds": 3000
    }
]

5. Upload Data

Below is an example showing how to use CartLoader to efficiently upload one dataset to your AWS S3 bucket.

1
2
3
4
5
6
7
AWS_DIR="s3://example/aws/prefix"   # S3 prefix that will contain the dataset
DATA_ID="EXAMPLE_ID"                # dataset identifier

cartloader upload_aws \
    --in-dir /path/to/cartload2/output \
    --s3-dir "${AWS_DIR}/${DATA_ID}" \
    --n-jobs 10

6. Get the Dataset URL

Your dataset's entry point is the YAML file that contains metadata about your dataset, which typically has the name catalog.yaml. The URL format is:

1
2
# replace with your actual bucket name, folder name, and catalog filename
s3://<your-bucket-name>/<folder_name>/<catalog_filename>

Tips

If your metadata file is catalog.yaml, you can also use the URL of the S3 folder for your dataset as entry point. For example:

1
s3://<your-bucket-name>/<folder_name>

Next

To load your data in CartoScope, use this URL in CartoScope's Load External Data dialog.