Core Concepts

Below are a few core concepts that'll help you get started with Spell.

Machine Types

Spell provides a variety of machine types to run your code. By default, we use a regular CPU (Central Processing Unit) machine with 2 cores and 8 GB of memory. This is similar to what you'd see in a typical home computer. Neural networks can run on a CPU, but are much faster if run on a GPU.

A GPU is a specialized piece of hardware that takes advantage of running many calculations simultaneously and is great for doing matrix math, an essential part of machine learning algorithms.

For a full list of hardware and associated pricing, see our pricing table.

To specify a GPU or CPU type, just use the --machine-type or -t flag (examples below).

# default CPU
$ spell run "python example.py"

# Nvidia's Tesla K80
$ spell run -t K80 "python example.py"

# Nvidia's Tesla V100 multi-GPU option
$ spell run -t V100x4 "python multi-example.py"

Versioning

Spell will keep track of all changes that are made when you run your code. In order to do this, we require your code to be tracked in a Git version control system. Each time you run, your Git repository must be in a clean state, meaning no changes are allowed to be staged or un-staged.

Spell is focused on maintaining reproducibility, so every run corresponds to a commit hash. We also do intelligent caching of your runs, so if no code changes were made, we won't rebuild your entire environment, because we've cached it from a previous run. This means that if you run the same code multiple times, subsequent builds of runs will run faster.

The Spell Model

This next section will explain some of the foundational concepts behind Spell and provide a mental model for how our system works.

Runs

Runs are the foundation of Spell. Creating a run is how you execute your code on Spell's computing infrastructure. For a detailed overview of runs, go to our What Is a Run doc.

Workflows

Complex machine learning applications often require multi-stage pipelines (e.g., data loading, transforming, training, testing, iterating) and each stage might be appropriate for a different Spell run. Workflows are designed to help you automate this process. A workflow is a special type of run that can create other runs. All runs created within a workflow are linked to that workflow. Thus, a workflow represents a logical grouping of runs, typically from a single script or application. This allows you to keep track of which runs were created from a specific application pipeline.

Workflows are particularly convenient for long running pipelines or scripts, since you don't have to worry about keeping your computer up and running for the duration of the script if you execute the workflow on Spell's infrastructure.

The spell workflow command is used to create a workflow. It is very similar to the spell run command with a few notable differences:

  • A new workflow is automatically created prior to executing the command argument to spell workflow.
  • Temporary authentication credentials for your user are passed into the run to enable additional requests to the Spell server (e.g., to create additional runs).
  • The --repo option can be specified to add a Git repository at a specific commit to this workflow. Similar to a normal run, the code in the specified repo will be synced to the remote machine prior to executing the workflow. All of the --repo specified respositories will be synced prior to executing the workflow. Part of the --repo specification is a label that can be used in subsequent run requests during the workflow to specify that Git commit for the run.

Workflows are typically used in conjunction with the Spell Python API, since it provides an easy way to programmatically interact with Spell. See Spell Python API for more information.

Workflow example

An example workflow can be found in the spell-examples repo on GitHub. This workflow is a Python script that uses the Spell Python API to create three runs:

  • one to download a large text file
  • one to mount this large text file and train a recurrent neural network from character-level language models to predict the next characters in a sequence
  • one to use the trained model to generate new artificial text, similar to the training file

To run the workflow:

  1. Clone the Tensorflow character-level language RNN model repo:
        $ git clone https://github.com/sherjilozair/char-rnn-tensorflow.git
        
  2. Clone the spell-examples repo:
        $ git clone https://github.com/spellrun/spell-examples.git
        
  3. Run the workflow
        $ cd spell-examples
        
        $ spell workflow --repo char-rnn=../char-rnn-tensorflow/ python workflows/char-rnn-workflow/workflow.py
        
    The workflow will print out the generated text as the last line of the logs.

Resources

For an in-depth explanation of how Spell uses resources (including custom and public datasets), go to What Is a Resource.