Cluster Bucket Management
For users on the Teams plan, we deploy Spell in your cloud and provide the same cluster management tools backing our own internal infrastructure. You can directly mount data from your S3/GS buckets into your runs. This document explains how to accomplish that.
When you set up Spell within your cluster, the setup script creates a new bucket that is used exclusively for outputs from your runs and Jupyter workspaces. Spell will never write/modify any other buckets. The name of this bucket is displayed under
Bucket Name on the cluster dashboard. Contents of this bucket will be displayed under
runs in Spell resources.
Mounting bucket resources directly
If a bucket is public you can mount resources from that bucket into your runs directly. See Mounting Resources on how to do that.
Adding a Bucket
To add a bucket to the Spell FS, run
spell cluster add-bucket:
$ spell cluster add-bucket --------------------------------------------- All of this will be done within your project 'spell-external' - continue? [Y/n]: y This command will - List your buckets to generate an options menu of buckets that can be added to Spell - Add list and read permissions for that bucket to the service account associated with the cluster - Ensure that the interoperable S3 access credentials associated with the cluster are able to read this bucket. ... Please choose a bucket: spell-my-cluster Bucket spell-my-cluster]has been added to cluster 1!
Now, when you run
spell ls s3/ (AWS cluster) or
spell ls gs/ (GCP cluster), you will see the new bucket:
$ spell ls gs/ - - spell-my-cluster
Mounting from a bucket
You can mount bucket resources just as any other SpellFS resource into your run:
spell run --mount gs/spell-my-cluster/file:file 'cat file'