My First Run

Hello World

Spell lets you easily execute commands on a remote server.

With Spell, you don’t have to worry about installing dependencies or managing your environment. All that is done via our fully customizable environments.

All your code is tracked making experiments easy to reproduce.

Spell and Git

To get started, clone your favorite GitHub repository. In this example, we’ll use Richard Zhang’s colorize repository.

Next, cd into your repository.

Spell uses Git to keep track of your runs. It’s not strictly necessary to run commands within a Git repo. However, it’s helpful for keeping your runs organized.

spell $ git clone https://github.com/richzhang/colorization.git
Cloning into 'colorization'...
remote: Enumerating objects: 4812, done.
remote: Total 4812 (delta 0), reused 0 (delta 0), pack-reused 4812
Receiving objects: 100% (4812/4812), 176.13 MiB | 6.71 MiB/s, done.
Resolving deltas: 100% (384/384), done.
spell $ cd colorization/

The Run Command

You can run commands on Spell by executing spell run followed by your command.

If you don’t specify a machine type with the -t or --machine-type flag, then the command will run on Spell’s default CPU.

This is useful for running commands that don’t need a powerful GPU, such as the .sh file in the colorize example.

Any outputs from your run will be saved on Spell and you can easily use those files in subsequent runs.

spell $ spell run ./models/fetch_release_models.sh

Counting objects: 451, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (283/283), done.
Writing objects: 100% (451/451), 25.28 MiB | 2.60 MiB/s, done.
Total 451 (delta 164), reused 451 (delta 164)
remote: Resolving deltas: 100% (164/164), done.
To git.spell.run:test2/f49d90s6g6s9sdugy7asoifhjqpwqi66371srg.git
* [new branch] HEAD -> br_sr91235yghsoqo028ty589e0220eqjrfnxidq729rft
💫 Casting spell #2…
✨ Stop viewing logs with ^C
✨ Machine_Requested… done
✨ Building… done
✨ Run is running
2450K .......... .......... .......... .......... .......... 1% 118M 8s

Upload Files

Upload files once to Spell and use the same files in multiple runs.

Spell’s upload feature caches your uploaded files which makes it faster to run multiple runs using the same data.

Data can be text files, images, pre-trained models, and more.

Here we’re uploading a black and white photo of a flower that we’ll use in our colorization example.

spell $ ls
bw-flower.jpeg
spell $ spell upload bw-flower.jpeg
Enter a name for the upload.
'/Users/spell/Desktop/demo/bw-flower.jpeg'
will be accessible at 'uploads/NAME/bw-flower.jpeg' [spell]: bw-flower
Total upload size: 14K
Uploading to uploads/bw-flower [###########################################] 100%

Mounting Files

You can add files to a run using the --mount or -m flag.

You can mount files from your uploads directory, or the output of a previous run. We’re doing both here. You can also mount data from public S3/GS buckets.

We mount the black and white flower photo from uploads, as well as the output of an earlier run, run 2.

spell/colorization $ spell run -m uploads/bw-flower:img -t K80 --framework caffe --apt python-tk --pip scikit-image --pip numpy --pip matplotlib --pip scipy -m runs/2/models:/mnt/models "python ./colorization/colorize.py -img_in img/bw-flower.jpeg -img_out ./out.png --caffemodel /mnt/models/colorization_release_v2.caffemodel"
Everything up-to-date
💫 Casting spell #3…
✨ Stop viewing logs with ^C
✨ Machine_Requested… done
✨ Building… done
✨ Run is running

Machine Types

Spell then sends your command to be executed on our remote servers.

You can tell Spell what machine you want to run your command on.

In this example, we’ve specified running on a K80 GPU using the -t flag. This is the same as the --machine-type flag.

Spell supports all the latest hardware options from NIVIDIA and Google, with no markup on what our hardware providers charge.

spell/colorization $ spell run -m uploads/bw-flower:img -t K80 --framework caffe --apt python-tk --pip scikit-image --pip numpy --pip matplotlib --pip scipy -m runs/2/models:/mnt/models "python ./colorization/colorize.py -img_in img/bw-flower.jpeg -img_out ./out.png --caffemodel /mnt/models/colorization_release_v2.caffemodel"
Everything up-to-date
💫 Casting spell #3…
✨ Stop viewing logs with ^C
✨ Machine_Requested… done
✨ Building… done
✨ Run is running

Managing Dependencies

Spell manages your dependencies for you. Just indicate the pip and apt dependencies you need, or include a pip-req file.

You can also specify your preferred framework, including a specific version.

Here is the command we’re running:

$ spell run --machine-type K80 \
  --pip matplotlib --pip numpy --pip scikit-image --pip scipy --apt python-tk \
  --mount runs/2/models:/mnt/models \
  --mount uploads/bw-flower:/spell/colorization/img \
  --framework caffe \
  'python ./colorization/colorize.py \
  -img_in img/bw-flower.jpeg -img_out ./out.png \
  --caffemodel /mnt/models/colorization_release_v2.caffemodel'

Read more about customizing your environment in our docs.

spell/colorization $ spell run -m uploads/bw-flower:img -t K80 --framework caffe --apt python-tk --pip scikit-image --pip numpy --pip matplotlib --pip scipy -m runs/2/models:/mnt/models "python ./colorization/colorize.py -img_in img/bw-flower.jpeg -img_out ./out.png --caffemodel /mnt/models/colorization_release_v2.caffemodel"
Everything up-to-date
💫 Casting spell #3…
✨ Stop viewing logs with ^C
✨ Machine_Requested… done
✨ Building… done
✨ Run is running

Downloading Files

To download the output of a run to your local directory, use the spell cp command.

Here, we download and open the output of our colorization run.

spell/colorization $ spell ls runs/3
230530 Nov 02 13:59 out.png
spell/colorization $ spell cp runs/3/out.png
✔ Copied 1 files
spell/colorization $ open out.png