Command Files

Many CLI commands allow you to load and save arguments and options to and from a YAML or JSON file. These commands will include both --from-file and --save-command options when you run --help on them. These files can be used to save commonly-used parameters and prevent accidental mistypings.

Loading from a file

You can load options and arguments from a file using the --from-file option. Using the example from the style transfer demo, we can represent the command:

$ spell run --mount runs/1/data:datasets \
            --machine-type K80 \
            --apt ffmpeg \
            --pip moviepy \
            --verbose \
  "python style.py \
            --checkpoint-dir ckpt \
            --style images/style/my-style-image.jpg \
            --style-weight 1.5e2 \
            --train-path datasets/train2014 \
            --vgg-path datasets/imagenet-vgg-verydeep-19.mat"

Using the command file:

# run-style.yaml
command: style.py
args:
  - --checkpoint-dir ckpt
  - --style images/style/my-style-image.jpg
  - --train-path datasets/train2014
  - --vgg-path datasets/imagenet-vgg-verydeep-19.mat
mount: 
  - runs/1/data:datasets
machine-type: K80
apt:
  - ffmpeg
pip:
  - moviepy
verbose: true

And run it as:

spell run --from-file run-style.yaml

Saving a command to a file

To save a command to use for later use, you can add the --save-command option. In our previous example, we can save the command to a file using the --save-command option, shown here:

$ spell run --mount runs/1/data:datasets \
            --machine-type K80 \
            --apt ffmpeg \
            --pip moviepy \
            --save-command save-style.yaml \
  "python style.py \
            --checkpoint-dir ckpt \
            --style images/style/my-style-image.jpg \
            --style-weight 1.5e2 \
            --train-path datasets/train2014 \
            --vgg-path datasets/imagenet-vgg-verydeep-19.mat"

This will generate the following command file:

# save-style.yaml
apt:
  - ffmpeg
args: []
background: false
command: echo
commit-ref: HEAD
conda-env: null
conda-file: null
description: null
distributed: null
docker-image: null
env:
- MYVAR=Hello everyone
force: false
framework: null
github-ref: null
github-url: null
idempotent: false
label: []
local-caching: false
machine-type: K80
mount: 
  - runs/1/data:datasets
  - 
pip:
  - moviepy
pip-req: null
provider: null
stop-condition: []
tensorboard-dir: null
verbose: false

Overriding command files

The options in command files can be overridden in the command line. For example, if you have a command file such as:

# run.yaml
env:
  - MYVAR=World

Then the command:

spell run --from-file run.yaml --env MYVAR=Spell echo $MYVAR

will run echo Spell not echo World.

Note

You cannot override positional arguments. All arguments must be specified either in the command file or in the CLI

Command file syntax and semantics

Positional arguments

Positional arguments are represented using the name of the argument as represented in its help documentation. For example,

$ spell run --help
Usage: spell run [OPTIONS] COMMAND [ARGS]...
...

has positonal arguments represented as command and args in the command file.

Options

All options take the form --option in the command line. In a command file, the leading -'s are stripped. So an option --foo bar in the would be foo: bar in the command file.

Lists and strings

Any option or positional argument which can be represented as a list can be equivalently represented as a space-separated string. For example the command:

$ spell run --mount runs/1/data:datasets \
            --mount runs/2/data:datasets \
            --machine-type K80 \
            --apt ffmpeg \
            --pip moviepy \
            --pip requests \
            --save-command save-style.yaml \
  "python style.py \
            --checkpoint-dir ckpt \
            --style images/style/my-style-image.jpg \
            --style-weight 1.5e2 \
            --train-path datasets/train2014 \
            --vgg-path datasets/imagenet-vgg-verydeep-19.mat"

Could be represented in the command file as:

# run-style.yaml
command: style.py
args: --checkpoint-dir ckpt --style images/style/my-style-image.jpg --train-path datasets/train2014 --vgg-path datasets/imagenet-vgg-verydeep-19.mat
mount: runs/1/data:datasets runs/2/data:datasets
machine-type: K80
apt: ffmpeg
pip: moviepy requests

Boolean flags

Flags such as the --verbose flag in the the run command can be enabled in the command file as verbose: true and disabled using verbose: false. This is interpreted as passing the --verbose option to the CLI when verbose: true and not passing it when verbose: false.