Cluster Bucket Management

For users on the Teams plan, we deploy Spell in your cloud and provide the same cluster management tools backing our own internal infrastructure. You can directly mount data from your S3/GS buckets into your runs. This document explains how to accomplish that.

Save Bucket

When you set up Spell within your cluster, the setup script creates a new bucket that is used exclusively for outputs from your runs and Jupyter workspaces. Spell will never write/modify any other buckets. The name of this bucket is displayed under Bucket Name on the cluster dashboard. Contents of this bucket will be displayed under runs in Spell resources.

Mounting bucket resources directly

If a bucket is public you can mount resources from that bucket into your runs directly. See Mounting Resources on how to do that.

Adding a Bucket

To add a bucket to the Spell FS, run spell cluster add-bucket:

$ spell cluster add-bucket
---------------------------------------------
All of this will be done within your project 'spell-external' - continue? [Y/n]: y
This command will
    - List your buckets to generate an options menu of buckets that can be added to Spell
    - Add list and read permissions for that bucket to the service account associated with the cluster
    - Ensure that the interoperable S3 access credentials associated with the cluster are able to read this bucket.
...
Please choose a bucket: spell-my-cluster
Bucket spell-my-cluster]has been added to cluster 1!

Now, when you run spell ls s3/ (AWS cluster) or spell ls gs/ (GCP cluster), you will see the new bucket:

$ spell ls gs/
-        -              spell-my-cluster

Mounting from a bucket

You can mount bucket resources just as any other SpellFS resource into your run:

spell run --mount gs/spell-my-cluster/file:file 'cat file'