How to Host Your Data on AWS S3¶

Hosting your data on AWS S3 provides high performance and reliability for CartoScope. Users could open their own S3 bucket and upload their data to it.

While users could set up their own S3 bucket and upload their data to it manually, CartLoader provides a more automated way to upload your data to AWS S3.

1. Set up AWS S3 Bucket¶

Follow the AWS S3 Official Documentation:

1) Setting up Amazon S3.
2) Creating a bucket.

2. Set Up AWS CLI¶

1) Install AWS CLI.
2) Configure AWS CLI.

3. Configure Bucket Policy¶

Your S3 bucket needs to be publicly accessible. Go to "Permissions" tab of your bucket and click "Edit" button to edit the bucket policy as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-bucket-name/*"
        }
    ]
}

Note that you need to replace your-bucket-name with your actual bucket name.

Cross-Origin Resource Sharing (CORS) is a security feature that allows web applications to access resources from different origins. This needs to be configured for CartoScope to access your data. Go to "Permissions" tab of your bucket and click "Edit" button to edit the CORS configuration as follows:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "HEAD"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [
            "Content-Range",
            "Content-Length",
            "ETag"
        ],
        "MaxAgeSeconds": 3000
    }
]

5. Upload Data¶

Now upload your data to AWS S3 bucket using aws-cli or cartloader.

5.1 Upload Data for A Single Dataset¶

Upload Dataset using cartloader command

Below is an example showing how to use CartLoader to efficiently upload a single dataset to your AWS S3 bucket in parallel.

AWS_DIR="s3://example/aws/prefix"   # S3 prefix that will contain the dataset
DATA_ID="EXAMPLE_ID"                # dataset identifier

cartloader upload_aws \
    --in-dir /path/to/cartload2/output \
    --s3-dir "${AWS_DIR}/${DATA_ID}" \
    --n-jobs 10

Upload Dataset using aws-cli command

Alternatively, you can use the aws s3 sync or aws s3 cp CLI command to recursively upload your processed directory to S3.

aws s3 sync automatically preserves directory hierarchy and only uploads new or modified files

AWS_DIR="s3://example/aws/prefix"   # S3 prefix that will contain the dataset
DATA_ID="EXAMPLE_ID"                # dataset identifier

aws s3 sync /path/to/cartload2/output "${AWS_DIR}/${DATA_ID}"

aws s3 cp --recursive can be used to upload a directory

AWS_DIR="s3://example/aws/prefix"   # S3 prefix that will contain the dataset
DATA_ID="EXAMPLE_ID"                # dataset identifier

aws s3 cp --recursive /path/to/cartload2/output "${AWS_DIR}/${DATA_ID}"

5.2 Upload Data for a Multi-Sample Session¶

Multi-sample cohort analysis outputs typically consist of a master multi-catalog.yaml file, shared analysis results, and multiple nested sample dataset subfolders (e.g. sample_1/, sample_2/).

Info

cartloader currently does not support multi-sample analysis uploads, the recommended way to upload an entire multi-sample analysis output directory is using aws-cli commands to upload files recursively to S3, preserving the relative folder structure.

Upload Multi-Sample Directory using aws-cli command

aws s3 sync automatically preserves directory hierarchy and only uploads new or modified files

AWS_DIR="s3://example/aws/prefix"   # S3 prefix that will contain the dataset
Session_ID="EXAMPLE_ID"             # session identifier

aws s3 sync /path/to/multi_sample_output "${AWS_DIR}/${Session_ID}"

aws s3 cp --recursive can be used to upload a directory

AWS_DIR="s3://example/aws/prefix"   # S3 prefix that will contain the dataset
Session_ID="EXAMPLE_ID"             # session identifier

aws s3 cp --recursive /path/to/multi_sample_output "${AWS_DIR}/${Session_ID}"

6. Get the Entry Point URL¶

6.A Single Dataset Entry Point (`catalog.yaml`)¶

Your dataset's entry point is the YAML file containing metadata about your dataset, which typically has the name catalog.yaml:

# Replace with your actual bucket name, folder name, and catalog filename
s3://<your-bucket-name>/<folder_name>/<catalog_filename> 
# Example: s3://cartostore/data/sample_1/catalog.yaml

Use Folder URL

If your metadata file is named catalog.yaml, you can also use the S3 folder URL as the entry point:

s3://<your-bucket-name>/<folder_name>/
# Example: s3://cartostore/data/sample_1/

6.B Multi-Sample Cohort Session Entry Point (`multi-catalog.yaml`)¶

To launch a multi-sample cohort session containing multiple sample slides, use the multi-catalog.yaml master catalog file:

# Multi-Sample Catalog URL
s3://<your-bucket-name>/<session_name>/multi-catalog.yaml
# Example: s3://cartostore/cohorts/brain_study_2026/multi-catalog.yaml

Use Folder URL

If your metadata file is named multi-catalog.yaml, you can also use the S3 folder URL as the entry point:

s3://<your-bucket-name>/<session_name>/
# Example: s3://cartostore/cohorts/brain_study_2026/

Related Pages

Concepts

Manual Pages

Load Your Own Data

How-to Guides