Generic Platform Tutorial

Use the generic profile (--platform generic) for any platform without a dedicated profile, or whenever your data is a plain transcript table: one row per molecule (or per spatial bin and gene) with X, Y, gene, and optionally a count and a cell ID.

To show the workflow on real data, this tutorial treats the Xenium starter table, a small region of the adult mouse brain, as if it came from an unknown platform. It has the typical quirks of a table from another tool: comma-separated, its own column names, no count column, and a cell-ID column. (For actual Xenium data, use the Xenium profile instead.)

It first runs the whole analysis with a single run_together command, then walks through the same analysis step by step.

---

Run with run_together

Input Data

Download the example data, hosted on Zenodo (10.5281/zenodo.17953582), into a work directory:

work_dir=/path/to/work/directory        # replace with your work directory
mkdir -p ${work_dir} && cd ${work_dir}
wget https://zenodo.org/records/17953582/files/xenium_starter.raw.tar.gz
tar --strip-components=1 -zxvf xenium_starter.raw.tar.gz   # extracts input.tsv.gz

input.tsv.gz is comma-separated despite its name. Its first lines:

"transcript_id","cell_id","overlaps_nucleus","feature_name","x_location","y_location","z_location","qv"
281827164036151,133793,1,"Sox10",2350.0232,4153.6846,16.592316,40.0

Describe the Table

Tell run_together how to read the table:

What run_together needs	In this file	Option
X and Y coordinates	x_location, y_location	--colname-transcript-x, --colname-transcript-y
Gene name	feature_name	--colname-transcript-feature
Count	none: each row is one molecule	--colname-transcript-count (omitted here)
Coordinate unit	micrometers	--units-per-um 1 (input units per µm)
Cell ID (optional)	cell_id	--colname-transcript-cell cell_id and --colname-cell cell_id
Delimiter	comma	ingest.extra_flags in a config file (below); tab is the default

The delimiter has no command-line option, so set it in a small config file:

cat > generic_starter.json <<'EOF'
{ "ingest": { "extra_flags": ["--csv-delim ,"] } }
EOF

Run

Run the pipeline, either with a local installation or with the Docker image:

LocallyDocker

cartloader run_together \
  --platform generic \
  --raw-transcript ./input.tsv.gz \
  --config generic_starter.json \
  --colname-transcript-x x_location \
  --colname-transcript-y y_location \
  --colname-transcript-feature feature_name \
  --colname-transcript-cell cell_id \
  --colname-cell cell_id \
  --units-per-um 1 \
  --out-dir ./out/generic_hippo \
  --id generic_hippo \
  --width 12 \
  --n-factor 6,12 \
  --threads 4 -j 4

docker run --rm -v ${work_dir}:/data hyunminkang/cartloader:latest \
  run_together \
  --platform generic \
  --raw-transcript /data/input.tsv.gz \
  --config /data/generic_starter.json \
  --colname-transcript-x x_location \
  --colname-transcript-y y_location \
  --colname-transcript-feature feature_name \
  --colname-transcript-cell cell_id \
  --colname-cell cell_id \
  --units-per-um 1 \
  --out-dir /data/out/generic_hippo \
  --id generic_hippo \
  --width 12 \
  --n-factor 6,12 \
  --threads 4 -j 4

What happens:

• SGE conversion reads the table with the column mapping above and writes the unified format. The generic profile drops negative-control and unassigned probes by default (^(Unassigned|Neg|BLANK|Blank|Intergenic|Deprecated|System|NCS-|NCP-)); set --ingest-exclude-feature-regex to change that.
• FICTURE trains one model per --width × --n-factor combination and decodes every pixel.
• Cell analysis runs because --colname-cell is given: transcripts are grouped by cell_id into cells, which are clustered with the largest model. Transcripts outside any cell (cell_id of -1 here) are ignored. Without the two cell options, the run stays pixel-level.
• Packaging writes the deployable result to out/generic_hippo/cartl/generic_hippo/: a catalog.yaml and the PMTiles it lists.

Add --dry-run to print the commands without running them.

Adapting It to Your Data

• A tab-separated table needs no config file: tab is the default delimiter.
• A count column (e.g. UMIs per spatial bin): name it with --colname-transcript-count.
• Other units: set --units-per-um to the number of coordinate units per micrometer, e.g. 1000 for nanometers (see How do I define scaling?).
• No cell IDs: omit --colname-transcript-cell and --colname-cell.
• Cell centroids or polygons from your segmentation can be added with --in-cell-xy and --in-cell-boundary; see the generic platform page for their formats, for several samples at once, and for a full config.

How run_together differs from the step-by-step path

• FICTURE runs through run_ficture2_multi (a joint model over a single sample) with single-molecule pixel decoding, instead of run_ficture2.
• Feature filters are the run_together defaults: control features (Unassigned, Neg, BLANK/Blank, Intergenic, Deprecated, System, NCS-, NCP-) are dropped at ingest; predicted (Gm<digit>), mitochondrial (mt-/MT-), and ribosomal (Rps/Rpl) genes stay in the data but are left out of the factor models. Change them with --ingest-exclude-feature-regex and --exclude-feature-regex.
• Point layers are packaged with pmpoint (--use-pmpoint, 500 gene bins) instead of tippecanoe.
• Outputs go under --out-dir: tsv/<id>/ (converted SGE), fic/ (FICTURE), and cartl/<id>/ (PMTiles and catalog.yaml; deploy this), instead of sge/, ficture2/, and cartload2/.
• Publishing is built in: --s3-upload uploads cartl/<id>/ to S3 (see Publishing). For Zenodo, run upload_zenodo on cartl/<id>/ as in the step-by-step path.

---

Step by Step (Individual Commands)

The same pixel-level analysis, one command at a time. (The cell analysis that run_together adds is run by run_ficture2_multi_cells; it is easiest to get through run_together.)

Set Up the Environment

# ====
# Replace the placeholders with values for your system.
# ====
work_dir=/path/to/work/directory        # directory holding the input data; outputs are written here too
cd $work_dir

n_jobs=10                               # number of parallel jobs (also used as threads per job below)

# Activate the Python environment in which CartLoader is installed
conda activate ENV_NAME                 # replace ENV_NAME (or: source /path/to/venv/bin/activate)

No tool paths needed

Once submodules/build.sh has run (see Installation), every CartLoader command finds spatula, punkst, tippecanoe, pmtiles, and pmpoint in the submodules/ directory of the repository, and the default color map at assets/default_color_map.tsv. Pass --spatula, --ficture2 (the punkst repository directory, which contains bin/punkst), --tippecanoe, --pmtiles, or --cmap-file only to use a different build or file. upload_aws runs the aws CLI from your PATH (--aws to override).

Input Data

Use the data downloaded above.

Define ID and Parameters

# Unique identifier for your dataset
DATA_ID="generic_hippo"                  # change this to reflect your dataset name

# LDA parameters
train_width=12                           # define LDA training hexagon width (comma-separated if multiple widths are applied)
n_factor=6,12                            # define number of factors in LDA training (comma-separated if multiple n-factor values are provided)

SGE Format Conversion

Convert the table to the unified SGE format with sge_convert in generic mode. The --csv-* options play the role of run_together's --colname-transcript-* options:

cartloader sge_convert \
  --makefn sge_convert.mk \
  --platform generic \
  --in-csv ./input.tsv.gz \
  --csv-delim , \
  --csv-colname-x x_location \
  --csv-colname-y y_location \
  --csv-colname-feature-name feature_name \
  --units-per-um 1 \
  --out-dir ./sge \
  --exclude-feature-regex '^(Unassigned|Neg|BLANK|Blank|Intergenic|Deprecated|System|NCS-|NCP-)' \
  --sge-visual \
  --n-jobs ${n_jobs}

Without --csv-colname-count, each row counts as one molecule.

FICTURE Analysis

Compute spatial factors using punkst (FICTURE2). See more details on the Reference page.

cartloader run_ficture2 \
  --makefn run_ficture2.mk \
  --main \
  --in-transcript ./sge/transcripts.unsorted.tsv.gz \
  --in-feature ./sge/feature.clean.tsv.gz \
  --in-minmax ./sge/coordinate_minmax.tsv \
  --exclude-feature-regex '^(mt-.*$|Gm\d+$)' \
  --out-dir ./ficture2 \
  --width ${train_width} \
  --n-factor ${n_factor} \
  --n-jobs ${n_jobs} \
  --threads ${n_jobs}

Parameter	Required	Type	Description
--main	required 1	flag	Run all five main steps
--in-transcript	required	string	Path to the transcript-indexed SGE file from sge_convert
--out-dir	required	string	Output directory
--width	required	int or comma-separated list	Hexagon width(s) in µm for LDA training
--n-factor	required	int or comma-separated list	Number(s) of LDA factors; one model per width × factor combination
--makefn		string	File name for the generated Makefile (default: run_ficture2.mk)
--in-feature		string	Path to the per-gene count file from sge_convert (default: the feature list made during tiling)
--in-minmax		string	Path to the coordinate min/max file from sge_convert (default: the range computed during tiling)
--exclude-feature-regex		regex	Feature names to leave out of the factor analysis. punkst matches the whole name, so end each alternative with .* or $ as needed (the example drops mitochondrial mt-* and predicted Gm<digits> genes)
--cmap-file		string	Color map TSV (default: assets/default_color_map.tsv in the CartLoader repository)
--n-jobs		int	Number of parallel jobs (default: 1)
--threads		int	Number of threads per job (default: 4)

_{¹: CartLoader requires at least one action: --tile (tiling), --segment (hexagon segmentation), --init-lda (LDA training), --decode (pixel-level decoding), --umap (UMAP of the factors), or --main for all five.}

CartLoader Asset Packaging

Package the SGE and the FICTURE results into PMTiles and a catalog.yaml for CartoScope. See more details in its Reference page.

cartloader run_cartload2 \
  --makefn run_cartload2.mk \
  --fic-dir ./ficture2 \
  --out-dir ./cartload2 \
  --id ${DATA_ID} \
  --n-jobs ${n_jobs} \
  --threads ${n_jobs}

Packaging without factor analysis

To host only the SGE (no factors), use cartloader run_together ... --no-ficture, which tiles the transcripts and packages them without training any model.

Parameter	Required	Type	Description
--fic-dir	required	string	FICTURE output directory from run_ficture2; must contain ficture.params.json (the SGE paths are read from it)
--out-dir	required	string	Output directory for the PMTiles, assets, and catalog.yaml
--id	required	string	Dataset ID used in the catalog and output metadata (no whitespace)
--makefn		string	File name for the generated Makefile (default: run_cartload2.mk)
--n-jobs		int	Number of parallel jobs (default: 1)
--threads		int	Number of threads per job (default: 4)
--spatula, --tippecanoe, --pmtiles		string	Tool paths (default: the copies built under submodules/)

Upload to Data Repository

Choose a data repository to host/share your output

CartLoader supports two upload options (AWS and Zenodo) for storing PMTiles of SGE and spatial factors in a data repository.

Choose the one that best suits your needs.

AWS UploadsZenodo Uploads

Upload the generated CartLoader outputs to your designated AWS S3 directory. The aws CLI must be configured with credentials that can write there (see AWS Upload Failures).

# AWS S3 target location
S3_DIR=s3://your-bucket/path/to/dir     # recommended: end with DATA_ID, e.g. s3://your-bucket/test-data/${DATA_ID}

cartloader upload_aws \
  --in-dir ./cartload2 \
  --s3-dir "${S3_DIR}" \
  --n-jobs ${n_jobs}

Parameter	Required	Type	Description
--in-dir	required	string	Directory with the CartLoader packaging output (containing catalog.yaml)
--s3-dir	required	string	Target S3 directory
--profile		string	AWS CLI profile to use
--aws		string	Path to the AWS CLI (default: aws)
--n-jobs		int	Number of parallel jobs (default: 2)

Upload the generated CartLoader outputs to a new Zenodo deposition or an existing one (see Zenodo Setup for the access token).

zenodo_token=/path/to/zenodo/token/file    # file holding your Zenodo access token

cartloader upload_zenodo \
  --in-dir ./cartload2 \
  --upload-method catalog \
  --zenodo-token ${zenodo_token} \
  --title "Your Title" \
  --creators "Lastname, Firstname" \
  --description "This is an example description"

Parameter	Required	Type	Description
--in-dir	required	string	Directory with the CartLoader packaging output
--zenodo-token	required	string	Path to the file holding your Zenodo access token
--upload-method		string	Which files to upload: all (every file in --in-dir; default), catalog (the files listed in catalog.yaml), or files (the names given with --files)
--catalog-yaml		string	Catalog to read with --upload-method catalog (default: <in-dir>/catalog.yaml)
--zenodo-deposition-id		string	Existing deposition to upload into (default: create a new one)
--title	required 1	string	Title of the deposition
--creators	required 1	list of str	Creators in "Lastname, Firstname" format, each quoted
--description		string	Description of the deposition

_{¹: Required when creating a new deposition, or when the existing one lacks it.}

---

Output Data

The run_together run writes everything under out/generic_hippo/ (see Where are outputs written?); the deployable result is out/generic_hippo/cartl/generic_hippo/. For the outputs of each individual command, see the reference pages for sge_convert, run_ficture2, and run_cartload2.