Teams Resource Management

For users on the Spell for Teams plan, we deploy Spell in your cloud and provide the same cluster management tools backing our own internal infrastructure. You can directly mount data from your S3/GS buckets into your runs. This document explains how to accomplish that.

How resources are managed

When you set up Spell within your cluster, the setup script creates a new bucket that is used exclusively for outputs from your runs and Jupyter workspaces. Spell will never write/modify any other buckets. The name of this bucket is displayed under Bucket Name on the cluster dashboard. The contents of this bucket will be displayed under runs in Spell resources.

How runs disks are scaled

Spell will automatically resize a run when your disk when it is nearing capacity without interrupting the run. Using any more than 80% of your disk will trigger a resize doubling that disk's capacity. Larger disk size increases cost, but Spell recycles machines that are no longer in use to ensure you are using machines efficiently and keeping costs down.

Effectively, this means that runs will always start with the the disk size you specified during machine type creation and resize to larger disks as needed.

AWS disks can resize slowly and only once every six hours. These are limitations of AWS.

GCP disks can resize quickly and multiple times in succession. If you need more disk, the disk will keep resizing.

Adding a private bucket to a cluster

If a bucket is public you can mount resources from that bucket into your runs directly. See the section "Mounting Resources" of the Resources guide for more information.

If a bucket is private, before using it you must first add it to SpellFS by running spell cluster add-bucket:

$ spell cluster add-bucket
---------------------------------------------
All of this will be done within your project 'spell-external' - continue? [Y/n]: y
This command will
    - List your buckets to generate an options menu of buckets that can be added to Spell
    - Add list and read permissions for that bucket to the service account associated with the cluster
    - Ensure that the interoperable S3 access credentials associated with the cluster are able to read this bucket.
...
Please choose a bucket: spell-my-cluster
Bucket spell-my-cluster]has been added to cluster 1!

Now, when you run spell ls s3/ (AWS cluster) or spell ls gs/ (GCP cluster), you will see the new bucket:

$ spell ls gs/
-        -              spell-my-cluster

After doing this, you can mount private bucket the same way you would mount any other resource, using the mount flag:

spell run --mount gs/spell-my-cluster/file:file 'cat file'