How to Host Your Data on AWS S3¶

Hosting your data on AWS S3 provides high performance and reliability for CartoScope. Users could open their own S3 bucket and upload their data to it.

While users could set up their own S3 bucket and upload their data to it manually, CartLoader provides a more automated way to upload your data to AWS S3.

This is guide shows how to use CartLoader to upload your data to your AWS S3 bucket in a more automated way. Then it could be loaded and used in CartoScope.

1. Set up AWS S3 Bucket¶

Follow the AWS S3 Official Documentation:

1) Setting up Amazon S3.
2) Creating a bucket.

2. Set Up AWS CLI¶

1) Install AWS CLI.
2) Configure AWS CLI.

3. Configure CORS for CartoScope?¶

placeholder

4. Upload Data¶

Below is an example showing how to use CartLoader to efficiently upload one dataset to your AWS S3 bucket.

AWS_DIR="s3://example/aws/prefix"   # S3 prefix that will contain the dataset
DATA_ID="EXAMPLE_ID"                # dataset identifier

cartloader upload_aws \
    --in-dir /path/to/cartload2/output \
    --s3-dir "${AWS_DIR}/${DATA_ID}" \
    --n-jobs 10

5. Get the Dataset URL¶

Your dataset's entry point is the YAML file that contains metadata about your dataset, which typically has the name catalog.yaml. The URL format is:

# replace with your actual bucket name, folder name, and catalog filename
s3://<your-bucket-name>/<folder_name>/<catalog_filename> 

Tips

If your metadata file is catalog.yaml, you can also use the URL of the S3 folder for your dataset as entry point. For example:

1	`s3://<your-bucket-name>/<folder_name>`

Next¶

To load your data in CartoScope, use this URL in CartoScope's Load External Data dialog.