NAV Navbar

Quilt

Quilt enables anyone to deploy complex applications to the cloud by encoding expertise about how to run those applications in JavaScript blueprints.

This site describes how to use Quilt. It describes how to install Quilt to run a blueprint, and also how to write your own blueprint for a new application.

We hope you will find this documentation to be a helpful guide to using Quilt. If you run into any issues using Quilt, don’t hesitate to contact us. If you notice issues in the documentation, please submit a Github issue, or feel free to fix it and submit a pull request!

Overview

Run An Application in the Cloud

Walk through the Getting Started tutorial and learn how to quickly deploy an application to your favorite cloud provider.

Write Blueprints for Your Favorite Systems

Check out the Blueprint Writers Guide to learn how to write sharable blueprints so others can easily run the systems you’re an expert in.

Contribute to Quilt

Want to help develop Quilt? Check out the Quilt GitHub and our guide to new contributors.

Reference

Browse the Quilt API reference, and check out the Quilt CLI docs to learn how to easily run and manage deployments.

Getting Started

Installing Quilt

Quilt relies on Node.js. Start by downloading Node.js from the Node.js download page. We have only tested Quilt with Node version v7.10.0 and above.

Next, use Node.js’s package manager, npm, to install Quilt:

$ npm install -g @quilt/install

Note that if installing as root, the --unsafe-perm flag is required:

$ sudo npm install -g @quilt/install --unsafe-perm

If Quilt was successfully installed, the quilt command should output a blurb about how to use the command:

Configuring a Cloud Provider

Quilt needs access to your cloud provider credentials in order to launch machines on your account. Before continuing, make sure you have an account with one of Quilt’s supported cloud providers (Google Cloud, Amazon EC2, or DigitalOcean), and that you have located your credentials for this account.

For Amazon EC2, you can create an account with Amazon Web Services and then find your “Access Keys” from the Security Credentials page in the AWS Management Console. If you don’t have a key already, you should “Create New Access Key”. In the next step, we’ll use these credentials.

If you prefer to use another provider than Amazon EC2, check out Cloud Provider Configuration. Follow the instructions to find your credentials, but come back to this section before running quilt init.

Creating an Infrastructure

To run any applications with Quilt, you need to give Quilt access to your cloud provider account from the last step, and specify an infrastructure that Quilt should launch your application on. The easiest way to do this, is to run

$ quilt init

The quilt init command will ask a number of questions, and then set up the provider and infrastructure based on the given answers. For the sake of this tutorial, make sure to use the name default for the infrastructure. Additionally, when choosing a machine instance size, keep in mind that some providers have a free tier for certain instance types.

If you are unsure about how to answer any of the questions, the default values are appropriate for this tutorial.

For more information about quilt init, see the documentation.

It is also possible to use the Quilt blueprint API to specify Machines directly in blueprints, but that’s a topic for another time.

Getting the Blueprint

This section will walk you through how to run Nginx (an open-source web server) using Quilt. In the example, we’ll use Nginx to serve a simple webpage. Start by downloading the blueprint using git:

$ git clone https://github.com/quilt/nginx.git

The blueprint in main.js imports the app.js Nginx blueprint, and then deploys the Nginx app to the base infrastructure you created with quilt init.

Before running anything, you’ll need to download the JavaScript dependencies of the blueprint. The Nginx blueprint depends on the @quilt/quilt module; more complicated blueprints may have other dependencies as well. Use npm, the Node.js package manager, to install all dependencies in the nginx folder:

$ npm install .

Running the Application

To run a blueprint, you first need to have a Quilt daemon running. If you haven’t already started one, open a new terminal window and start it:

$ quilt daemon

The daemon is a long running process that periodically prints some log messages. Leave this running. In another terminal window, navigate to the nginx directory and run the blueprint:

$ quilt run ./main.js

This command tells the daemon to launch the containers described in main.js on your default base infrastructure. It will return immediately, because the daemon process does the heavy lifting.

It takes a few minutes from you quilt run until the VMs and application are fully up and running. To see how things are progressing, use Quilt’s show command:

$ quilt show
MACHINE         ROLE      PROVIDER    REGION       SIZE     PUBLIC IP    STATUS
e5b1839d2bea    Master    Amazon      us-west-1    t2.micro              disconnected
e2401c348c78    Worker    Amazon      us-west-1    t2.micro              disconnected

Your output will look similar to the output above (note that you may get an error that begins with “unable to query connections: rpc error” when you first run quilt show; this error is benign and can occur while the machines are booting).

The output above means that Quilt has launched two machines, one as a master and one as a worker, in Amazon. Both machines are disconnected, because they’re still being initialized. When a machine is fully booted and configured, it will be marked as connected in the STATUS column.

Accessing the Web App

After a few minutes, when the VMs and containers are up and running, the output of show will look something like this:

$ quilt show
MACHINE         ROLE      PROVIDER    REGION       SIZE        PUBLIC IP         STATUS
e5b1839d2bea    Master    Amazon      us-west-1    t2.micro    54.183.98.15      connected
e2401c348c78    Worker    Amazon      us-west-1    t2.micro    54.241.251.192    connected

CONTAINER       MACHINE         COMMAND        HOSTNAME           STATUS     CREATED               PUBLIC IP
bd681b3d3af7    e2401c348c78    nginx:1.13     web_tier           running    About a minute ago    54.241.251.192:80

The bottom row lists the container that’s running nginx. The nginx deployment is relatively simple and has just one container, but a typical application running in Quilt will have many containers running (one for each part of the application; for example, your website application might require a second container that runs a database).

The PUBLIC IP column shows the address you can use to access the website. Simply copy-paste this IP address into your browser. A site with “Hello, world!” text should appear.

This is all it takes to run an application on Quilt. The remainder of this tutorial will cover some of the things you might want to do after your application is up and running – e.g. debugging or changing the website content, and importantly, how to shut down the deployment.

Making the Application Publicly Accessible

By default, Quilt-managed containers are disconnected from the public internet and isolated from one another. This helps to keep your application secure by preventing all access except for what you explicitly specify. In order to make the Nginx container accessible from the public internet, nginx/app.js explicitly opens port 80 on the Nginx container to the outside world:

webTier.allowFrom(publicInternet, port);

Without this line, the website wouldn’t be accessible from the browser. You can change the value of the port variable to use a port other than 80.

Debugging Applications with Quilt

Once the containers are running, you might need to log in to change something or debug an issue. The quilt ssh command makes this easy. Use the container ID from the quilt show output as the argument to quilt ssh to log in to that container. For instance, to ssh into a container or VM whose ID starts with bd68:

$ quilt ssh bd68

To check the logs of the same container or VM, use quilt logs:

$ quilt logs bd68

If you run logs on the nginx container for instance, you’ll see that nginx logs a GET request for each time you access the website. This is not thrilling information, but the logs will come in handy if you ever encounter any errors.

Note that you don’t need to type the whole ID; as long as Quilt gets a unique prefix of the ID, it will log in to the correct machine.

Changing the Website Content

You may later decide that you’d like to change the contents of the simple website. You could do this by logging into the container, but for the sake of example, let’s do it using Quilt. On your laptop, open the nginx/index.html that was downloaded from github and change the contents of the page (e.g., you can change the “Hello, World!” message). Now, with the daemon still running, re-deploy the webpage with Quilt:

$ quilt run ./main.js

Quilt automatically detects the changes to the deployment, and will update it to implement your changes. Note that we didn’t need to tell Quilt to stop the nginx container and start a new one; we just updated the view of what the deployment should look like (in this case, by changing index.html), and Quilt automatically detects this and updates the cluster accordingly. Quilt will prompt you to accept the changes that you’re making to your deployment; type y.

Run quilt show again and notice that Quilt has stopped the old container and is starting a new one. When the new container is running, navigate to the new IP address and check that the modified page is up.

Stopping the Application

When you’re done experimenting with Quilt, make sure to stop the machines you’ve started! Otherwise, your cloud provider might charge you for the VMs that are still running. Quilt’s stop command will stop all VMs and containers:

$ quilt stop

At this point, you can safely kill the Quilt daemon.

How Quilt Works

This section describes what happens when you run an application using Quilt.

The key idea behind Quilt is a blueprint: a blueprint describes every aspect of running a particular application in the cloud, and is written in JavaScript. Quilt blueprints exist for many common applications. Using Quilt, you can run one of those applications by executing just two commands on your laptop:

Quilt Diagram

The first command,quilt daemon, starts a long-running process that handles launching machines in the cloud (on your choice of cloud provider), configuring those machines, and managing them (e.g., detecting when they’ve failed so need to be re-started). The quilt daemon command starts the daemon, but doesn’t yet launch any machines. To launch an application, call quilt run with a JavaScript blueprint (in this example, the blueprint is called my_app.js). The run command passes the parsed blueprint to the daemon, and the daemon sets up the infrastructure described in the blueprint.

Quilt runs applications using Docker containers. You can think of a container as being like a process: as a coarse rule-of-thumb, anything that you’d launch as its own process should have it’s own container with Quilt. While containers are lightweight (like processes), they each have their own environment (including their own filesystem and their own software installed) and are isolated from other containers running on the same machine (unlike processes). If you’ve never used containers before, it may be helpful to review the Docker getting started guide.

In this example, my_app.js described an application made up of three containers, and it described a cluster with one master machine and two worker machines. The master is responsible for managing the worker machines, and no application containers run on the master. The application containers are run on the workers; in this case, Quilt ran two containers on one worker machine and one container on the other.

Blueprint Writers Guide

The previous section described how to use Quilt to run an application that already had a blueprint. This guide describes how to write the Quilt blueprint for a new application, using the lobste.rs application as an example. lobste.rs is an open source project that implements a reddit-like web page, where users can post content and vote up or down other content.

Decomposing the application into containers

The first question you should ask yourself is “how should this application be decomposed into different containers?” Be sure you’ve read the How Quilt Works section, which gives a brief overview of containers. If you’ve already figured out the containers that are needed for your application (e.g., if you’re already using Docker), you can skip the rest of this section.

Specifying the containers for your application

As an example of how to specify the containers for your application, let’s use the lobste.rs example. lobste.rs requires mysql to run, so we’ll use one container for mysql. We’ll use a second container for the lobste.rs program to run in.

For each container that your application uses, you’ll need a container image. The container image describes the filesystem that will be on the container when it’s started. For mysql, for exampe, the container image includes all of the dependencies that mysql needs to run, so that after starting a new mysql container, you can simply launch mysql (no more installation is needed). Most popular applications already have containers that you can use, and a quick google search yields an existing mysql image that we can use for lobste.rs.

For the container that runs lobste.rs, we’ll need to create a new image by writing our own Dockerfile, which describes how the Docker image should be created. In this case, the Dockerfile is relatively simple:

# This container is based on the Ruby image, which means that it
# automatically inherits the Ruby installation defined in that image.
FROM ruby:2.3.1

# Install NodeJS, which is required by lobste.rs.
RUN apt-get update && apt-get install nodejs -y

# Download and build the lobste.rs code.
RUN git clone git://github.com/jcs/lobsters.git
WORKDIR lobsters
RUN bundle

# Add a file to the container that contains startup code for lobste.rs. This
# command assumes that start-lobsters.sh is in the same directory as this
# Dockerfile.
COPY start-lobsters.sh /lobsters/

# When the container starts, it should run the lobste.rs server using the
# start-lobsters.sh bash file that we copied above.  This is a common
# "gotcha" to people new to containers: unlike VMs, each container is based
# on a process (in this case, rails, which is started at the end of
# start-lobsters.sh) and will be shutdown when that process stops.
ENTRYPOINT ["/bin/sh", "/lobsters/start-lobsters.sh"]

In this case, we wrote an additional bash script, start-lobsters.sh, to help start the application. The important thing about that script is that it does some setup that needed to be done after the container was started, so it couldn’t be done in the Dockerfile. For example, the first piece of setup it does it to initialize the SQL database. Because that requires a connection to mysql, it needs to be done after the container is launched (and configured to access the mysql container, as discussed below). After initializing the database, the start-lobsters.sh script launches the rails server, which is the main process run by the container.

To create a docker image using this file, run docker build in the directory with the Dockerfile (don’t forget the period at the end!):

$ docker build -t kayousterhout/lobsters .

In this case, we called the resulting image kayousterhout/lobsters, because we’ll push it to the Dockerhub for kayousterhout; you’ll want to use your own Dockerhub id to name your images.

This will take a few minutes, and creates a new image with the name kayousterhout/lobsters. If you want to play around with the new container, you can use Docker to launch it locally:

$ docker run -n lobsters-test kayousterhout/lobsters

To use a shell on your new container to poke around (while the rails server is running), use:

$ docker exec -it lobsters-test /bin/bash

This can be helpful for making sure everything was installed and is running as expected (although in this case, lobste.rs won’t work when you start it with Docker, because it’s not yet connected to a mysql container).

Deploying the containers with Quilt

So far we have a mysql container image (we’re using an existing one hosted on Dockerhub) and a lobste.rs container image that we just made. You should similarly have the containers ready for your application. Up until now, we haven’t done anything Quilt-specific: if you were using another container management service like Kubernetes, you would have had to create the container images like we did above. These containers aren’t yet configured to communicate with each other, which is what we’ll set up with Quilt. We’ll also use Quilt to descrbie the machines to launch for the containers to run on.

To run the containers for your application with Quilt, you’ll need to write a Quilt blueprint. Quilt blueprints are written in Javascript, and the Quilt Javascript API is described here. In this guide, we’ll walk through how to write a Quilt blueprint for lobste.rs, but the Quilt API has more functionality than we could describe here. See the API guide for more usage information.

Writing the Quilt blueprint for MySQL

First, let’s write the Quilt blueprint to get the MySQL container up and running. We need to create a container based on the mysql image:

let sql = new Container('sql', 'mysql:5.6.32');

Here, the argument to Container is the hostname for the container, and the name of an image. You can also pass in a Dockerfile to use to create a new image, as described in the Javascript API documentation.

Next, the SQL container requires some environment variables to be set. In particular, we need to specify a root password for SQL. We can set the root password to foo with the setEnv function:

sql.setEnv('MYSQL_ROOT_PASSWORD', 'foo');

Writing the Quilt blueprint for lobste.rs

Next, we can similarly initialize the lobsters container. The lobsters container is a little trickier to initialize because it requires an environment variable (DATABASE_URL) to be set to the URL of the SQL container. Quilt containers are each assigned unique hostnames when they’re initialized, so we can create the lobsters container and initialize the URL as follows:

let lobsters = new Container('kayousterhout/lobsters');
const sqlDatabaseUrl = 'mysql2://root:' + mysqlOpts.rootPassword + '@' + sqlContainer.getHostname() + ':3306/lobsters';
lobsters.setEnv('DATABASE_URL', sqlDatabaseUrl);

Allowing network connections

At this point, we’ve written code to create a mysql container and a lobsters container. With Quilt, by default, all network connections are blocked. To allow lobsters to talk to mysql, we need to explicitly open the mysql port (3306):

sql.allowFrom(lobsters, 3306);

Because lobsters is a web application, the relevant port should also be open to the public internet on the lobsters container. Quilt has a publicInternet variable that can be used to connect containers to any IP address:

lobsters.allowFrom(publicInternet, 3000);

Deploying the application on infrastructure

Finally, we’ll use Quilt to launch some machines, and then start our containers on those machines. First, we’ll define a “base machine.” We’ll deploy a few machines, and creating the base machine is a useful way to create one machine that all of the machines in our deployment will be based off of. In this case, the base machine will be an Amazon instance that allows ssh access from the public key “bar”:

let baseMachine = new Machine({provider: 'Amazon', sshKeys: ['ssh-rsa bar']});

Now, using that base machine, we can deploy a master and a worker machine. All quilt deployments must have one master, which keeps track of state for all of the machines in the cluster, and 0 or more workers. To deploy machines and containers, you must create a deployment object, which maintains state about the deployment.

const deployment = createDeployment();
deployment.deploy(baseMachine.asMaster());
deployment.deploy(baseMachine.asWorker());

We’ve now defined a deployment with a master and worker machine. Let’s finally deploy the two containers on that infrastructure:

deployment.deploy(sql);
deployment.deploy(lobsters);

We’re done! Running the blueprint is now trivial. With a quilt daemon running, run your new blueprint (which, in this case, is called lobsters.js):

$ quilt run ./lobsters.js

Now users of lobsters, for example, can deploy it without needing to worry about the details of how different containers are connected with each other. All they need to do is to quilt run the existing blueprint.

Quilt.js API Documentation

This section documents use of the Quilt JavaScript library, which is used to write blueprints.

Global

Members

(constant) publicInternet

Implements:

Methods

allow(src, dst, port) → {void}

allow is a utility function to allow calling `allowFrom` on groups of containers.

Parameters:
Name Type Description
src Container | publicInternet The containers that can initiate a connection.
dst Array.<Connectable> The objects that traffic can be sent to. Examples of connectable objects are Containers, LoadBalancers, publicInternet, and user-defined objects that implement allowFrom.
port int | Port | PortRange The ports that traffic is allowed on.
Returns:
Type
void

baseInfrastructure(name) → {Deployment}

Returns a base infrastructure. The infrastructure can be deployed to simply by calling .deploy() on the returned object. The base infrastructure itself is automatically deployed, so there is no need to .deploy() it. The base infrastructure could be created with `quilt init`.

Parameters:
Name Type Default Description
name string default The name of the infrastructure, as passed to `quilt init`.
Returns:

A deployment object representing the infrastructure.

Type
Deployment
Example

Retrieve the base infrastructure called NAME, and deploy an nginx container on it.

const inf = baseInfrastructure('NAME');
inf.deploy(new Container('web', 'nginx'));

createDeployment(deploymentOpts) → {Deployment}

Overwrites the deployment object with a new one.

Parameters:
Name Type Description
deploymentOpts Object Options for the new deployment object.
Returns:

A deployment object.

Type
Deployment

getDeployment() → {Deployment}

Returns:

The global deployment object.

Type
Deployment

githubKeys(user) → {string}

Gets the public key associated with a github username.

Parameters:
Name Type Description
user string The GitHub username.
Returns:

The SSH key.

Type
string

Interface: Connectable

Connectable

Interface for classes that can allow inbound traffic.

Methods

allowFrom(src, port) → {void}

Allows traffic from src on port

Parameters:
Name Type Description
src Container The container that can initiate connections.
port int | Port | PortRange The ports to allow traffic on.
Returns:
Type
void

Class: Connection

Connection(from, ports)

new Connection(from, ports)

Creates a Connection.

Parameters:
Name Type Description
from string The host from which connections are allowed.
ports PortRange The port numbers which are allowed.

Class: Container

Container(hostnamePrefix, image, optionalArgsopt)

new Container(hostnamePrefix, image, optionalArgsopt)

Creates a new Container, which represents a container to be deployed. If a Container uses a custom image (e.g., the image is created by reading in a local Dockerfile), Quilt tracks the Dockerfile that was used to create that image. If the Dockerfile is changed and the blueprint is re-run, the image will be re-built and all containers that use the image will be re-started with the new image.

Parameters:
Name Type Attributes Description
hostnamePrefix string The network hostname of the container.
image Image | string An Image that the container should boot, or a string with the name of a Docker image (that exists in Docker Hub) that the container should boot.
optionalArgs Object <optional>
Additional, named, optional arguments.
Properties
Name Type Attributes Description
command string <optional>
The command to use when starting the container.
env Object.<string, string> <optional>
Environment variables to set in the booted container. The key is the name of the environment variable.
filepathToContent Object.<string, string> <optional>
Text files to be installed on the container before it starts. The key is the path on the container where the text file should be installed, and the value is the contents of the text file. If the file content specified by this argument changes and the blueprint is re-run, Quilt will re-start the container using the new files. Files are installed with permissions 0644 and parent directories are automatically created.
Implements:
Example

Create a Container with hostname myApp that uses the nginx image on Docker Hub, and that includes a file located at /etc/myconf with contents foo.

const container = new Container(
  'myApp', 'nginx', {filepathToContent: {'/etc/myconf': 'foo'}});

Methods

allowFrom(src, port) → {void}

Allows traffic from src on port

Parameters:
Name Type Description
src Container The container that can initiate connections.
port int | Port | PortRange The ports to allow traffic on.
Implements:
Returns:
Type
void

getHostname() → {string}

Returns:

The container's hostname.

Type
string

withFiles(fileMap) → {Container}

Creates a new container that replaces the mapping of filepaths to filecontent with the given mapping.

Parameters:
Name Type Description
fileMap Object.<string, string> Text files to be installed on the container before it starts. Uses the same format as the filepathToContent argument to the Container constructor.
Returns:

A new container that is identical to this one, except that filepathToContent is set to the given mappng.

Type
Container
Example

Create a container with hostname haproxy and using an image named haproxyImage that has a file at path /etc/myconf containing the text foo.

const c = new Container('haproxy', haproxyImage).withFiles({
  '/etc/myconf': 'foo'
});

Class: Deployment

Deployment(deploymentOpts)

new Deployment(deploymentOpts)

Creates a new deployment object with the given options.

Parameters:
Name Type Description
deploymentOpts Object An object containing options that can tweak the behavor of the namespace. Options include: `maxPrice` which defines the price that should be bid in spot auctions for preemptible machines, `namespace` which instructs the deployment what namespace it should operate in, and `adminACL` which defines what network traffic should be allowed to access the deployment.

Class: Image

Image(name, dockerfileopt)

new Image(name, dockerfileopt)

Creates a Docker Image. If two images with the same name but different Dockerfiles are referenced, an error will be thrown.

Parameters:
Name Type Attributes Description
name string The name to use for the Docker image, or if no Dockerfile is specified, the repository to get the image from. The repository can be a full URL (e.g., quay.io/coreos/etcd) or the name of an image in Docker Hub (e.g., nginx or nginx:1.13.3).
dockerfile string <optional>
The string contents of the Dockerfile that constructs the Image.
Examples

Create an image that uses the nginx image stored on Docker Hub.

const image = new Image('nginx');

Create an image that uses the etcd image stored at quay.io.

const image = new Image('quay.io/coreos/etcd');

Create an Image named my-image-name that's built on top of the nginx image, and additionally includes the Git repository at github.com/my/web/repo cloned into /web_root.

const image = new Image('my-image-name',
  'FROM nginx\n' +
  'RUN cd /web_root && git clone github.com/my/web_repo');

Create an image named my-inage-name that's built using a Dockerfile saved locally at 'Dockerfile'.

const container = new Image('my-image-name', fs.readFileSync('./Dockerfile'));

Class: LoadBalancer

LoadBalancer(name, containers)

new LoadBalancer(name, containers)

Creates a new LoadBalancer object which represents a collection of containers behind a load balancer.

Parameters:
Name Type Description
name string The name of the load balancer.
containers Array.<Container> The containers behind the load balancer.
Implements:

Methods

allowFrom(srcArg, portRange) → {void}

Allows inbound connections to the load balancer. Note that this does not allow direct connections to the containers behind the load balancer.

Parameters:
Name Type Description
srcArg Container | Array.<Container> The containers that can open connections to this load balancer.
portRange int | Port | PortRange The ports on which containers can open connections.
Implements:
Returns:
Type
void

Class: Machine

Machine(optionalArgsopt)

new Machine(optionalArgsopt)

Creates a new Machine object, which represents a machine to be deployed.

Parameters:
Name Type Attributes Description
optionalArgs Object.<string, string> <optional>
Optional arguments that modify the machine.
Properties
Name Type Attributes Default Description
provider string <optional>
The cloud provider that the machine should be launched in. Accepted values are Amazon, DigitalOcean, Google, and Vagrant. This argument is optional, but the provider attribute of the machine must be set before it is deployed.
role string <optional>
The role the machine will run as (accepted value are Master and Worker). A Machine's role must be set before it can be deployed. This argument is not required, so that users can create a template to use for all machines in the cluster; Machine#asWorker and Machine#asMaster can be called on the template to create a machine with the appropriate role, as in the example.
region string <optional>
The region the machine will run-in (provider-specific; e.g., for Amazon, this could be 'us-west-2').
size string <optional>
The instance type (provider-specific).
cpu Range | int <optional>
The desired number of CPUs.
ram Range | int <optional>
The desired amount of RAM in GiB.
diskSize int <optional>
The desired amount of disk space in GB.
floatingIp string <optional>
A reserved IP to associate with the machine.
sshKeys Array.<string> <optional>
Public keys to allow to login to the machine.
preemptible boolean <optional>
false Whether the machine should be preemptible. Only supported on the Amazon provider.
Examples

Create a template Machine on Amazon, and then use the template to create a master and 2 workers. This will use the default size and region for Amazon.

const baseMachine = new Machine({provider: 'Amazon'});
const master = baseMachine.asMaster();
const workers = baseMachine.asWorker().replicate(2);

Create a worker machine with the 'n1-standard-1' size in GCE's 'us-east1-b' region.

const googleWorker = new Machine({
  provider: 'Google',
  region: 'us-east1-b',
  size: 'n1-standard-1',
  role: 'Worker',
});

Create a DigitalOcean master droplet with the '512mb' size in the 'sfo1' zone.

const googleWorker = new Machine({
  provider: 'DigitalOcean',
  region: 'sfo1',
  size: '512mb',
  role: 'Master',
});

Methods

asMaster() → {Machine}

Returns:

A new machine with role Master.

Type
Machine

asWorker() → {Machine}

Returns:

A new machine with role Worker.

Type
Machine

Class: Port

Port(p)

new Port(p)

Creates a Port object.

Parameters:
Name Type Description
p integer The port number.

Class: Range

Range(min, max)

new Range(min, max)

Creates a Range object.

Parameters:
Name Type Description
min integer The minimum of the range (inclusive).
max integer The maximum of the range (inclusive).

Quilt CLI

Quilt’s CLI, quilt, is a handy command line tool for starting, stopping, and managing deployments. Quilt CLI commands have the following format:

$ quilt [OPTIONS] COMMAND

To see the help text for a specific command, run:

$ quilt COMMAND --help

Options

Name, shorthand Default Description
--log-level, -l info Logging level (debug, info, warn, error, fatal, or panic)
--verbose, -v false Turn on debug logging
-log-file Log output file (will be overwritten)

Commands

Name Description
counters Display internal counters tracked for debugging purposes. Most users will not need this command.
daemon Start the quilt daemon, which listens for quilt API requests.
debug-logs Fetch logs for a set of machines or containers.
init Create an infrastructure that can be accessed in blueprints using baseInfrastructure().
inspect Visualize a blueprint.
logs Fetch the logs of a container or machine minion.
minion Run the quilt minion.
show Display the status of quilt-managed machines and containers.
run Compile a blueprint, and deploy the system it describes.
ssh SSH into or execute a command in a machine or container.
stop Stop a deployment.
version Show the Quilt version information.
setup-tls Create the files necessary for TLS-encrypted communication with Quilt.

Init

The quilt init command is a simple way to create reusable infrastructure. The command prompts the user for information about their desired infrastructure and then creates an infrastructure based on the answers. The infrastructure can be used in blueprints by calling baseInfrastructure(NAME), where NAME is the infrastructure name given to quilt init.

It is possible to create multiple infrastructures with quilt init, but we recommend at least having a small infrastructure called default with your standard configuration. Some example blueprints will assume such a default infrastructure exists.

To edit the infrastructure after creation, either rerun quilt init using the same name, or directly edit the infrastructure blueprint stored in ~/.quilt/infra/<NAME>.js.

Most of the quilt init questions are self-explanatory, but the following might warrant a little explanation:

Cloud Provider Configuration

This section describes the basic configuration of the cloud providers supported by Quilt, and gives some details about how to enable extra features (e.g., floating IP addresses) on each cloud provider.

Quilt needs access to your cloud provider credentials in order to make the API calls needed to boot your deployment. Don’t worry, Quilt will never store your credentials or use them for anything else than deploying your application.

Amazon EC2

Set Up Credentials

  1. If you don’t already have an account with Amazon Web Services, go ahead and create one.

  2. Get your access credentials from the Security Credentials page in the AWS Management Console. Choose “Access Keys” and then “Create New Access Key.”

  3. Run quilt init on the machine that will be running the daemon, and pass it your AWS credentials. The formatted credentials will be placed in ~/.aws/credentials.

Formatting Credentials

While it is recommended to use quilt init to format the provider credentials, it is possible to manually create the credentials file in ~/.aws/credentials on the machine that will be running the daemon:

[default]
aws_access_key_id = <YOUR_ID>
aws_secret_access_key = <YOUR_SECRET_KEY>

The file needs to appear exactly as above (including the [default] at the top), except with <YOUR_ID> and <YOUR_SECRET_KEY> filled in appropriately.

DigitalOcean

Set Up Credentials

  1. If you don’t have a DigitalOcean account, go ahead and create one.

  2. Create a new token here. The token must have both read and write permissions.

  3. Run quilt init on the machine that will be running the Quilt daemon, and pass it your token. The token will be placed in ~/.digitalocean/key.

Floating IPs

Unless there are already droplets running, DigitalOcean doesn’t allow users to create floating IPs under the “Networking” tab on their website. Instead, this link can be used to reserve IPs that Quilt can then assign to droplets.

Google Compute Engine

Set Up Credentials

  1. If you don’t have an account on Google Cloud Platform, go ahead and create one.

  2. Create a Google Cloud Platform Project: All instances are booted under a Cloud Platform project. To setup a project for use with Quilt, go to the console page, then click the project dropdown at the top of page, and hit the plus icon. Pick a name, and create your project.

  3. Enable the Compute API: Select your newly created project from the project selector at the top of the console page, and then select APIs & services -> Library from the navbar on the left. Search for and enable the Google Compute Engine API.

  4. Save the Credentials File: Go to Credentials on the left navbar (under APIs & services), and create credentials for a Service account key. Create a new service account with the Project -> Editor role, and select the JSON output option.

  5. Run quilt init on the machine from which you will be running the Quilt daemon, and give it the path to the downloaded JSON from step 3. The credentials will be placed in ~/.gce/quilt.json.

Developing Quilt

Setup

Install Go

The project is written in Go and supports Go version 1.8 or later. Install Go using your package manager (Go is commonly referred to as “golang” in package managers and elsewhere) or via the Go website.

If you’ve never used Go before, we recommend reading the overview to Go workspaces here. In short, you’ll need to configure the GOPATH environment variable to be the location where you’ll keep all Go code. For example, if you’d like your Go workspace to be $HOME/gowork:

export GOPATH="$HOME/gowork"
export PATH="$GOPATH/bin:$PATH"

Add these commands to your .bashrc so that they’ll be run automatically each time you open a new shell.

Download Quilt

Clone the Quilt repository into your Go workspace using go get:

$ go get github.com/quilt/quilt

This will install Quilt in your Go workspace at $GOPATH/src/github.com/quilt/quilt, and compile Quilt. After running installing Quilt, the quilt command should execute successfully in your shell.

Protobufs

If you change any of the proto files, you’ll need to regenerate the protobuf code. We currently use protoc v3. On a Mac with homebrew, you can install protoc v3 using:

$ brew install protobuf

On other operating systems you can directly download the protoc binary here, and then add it to your $PATH.

To generate the protobufs simply call:

$ make generate

Dependencies

We use govendor for dependency management. If you are using Go 1.5 make sure GO15VENDOREXPERIMENT is set to 1.

To add a new dependency:

  1. Run go get foo/bar
  2. Edit your code to import foo/bar
  3. Run govendor add +external

To update a dependency:

$ govendor update +vendor

Building and Testing

Building and Testing the Go Code

To build Quilt, run go install in the Quilt directory. To do things beyond basic build and install, several additional build tools are required. These can be installed with the make go-get target.

Note that if you’ve previously installed Quilt with npm, there will be another Quilt binary installed on your machine (that was downloaded during the npm installation). If you want to develop Quilt, you probably want to make sure that when you run quilt, the version you’re developing (that was compiled from the Go code) gets run, and not the Quilt release that was downloaded from npm. Check that this is the case:

$ which quilt
/Users/kay/gowork/bin/quilt

If running which quilt results in a path that includes your $GOPATH$, like the one above, you’re all set. If it instead returns someplace else, e.g., /usr/local/bin/quilt, you’ll need to fix your $PATH variable so that $GOPATH/bin comes first.

To run the go tests, use the gocheck Make target in the root directory:

$ make gocheck

If you’d like to run the tests in just one package, e.g., the tests in the engine package, use go test with the package name:

$ go test github.com/quilt/quilt/engine

Building and Testing the JavaScript Code

To run the JavaScript code, you’ll need to use npm to install Quilt’s dependencies:

$ npm install .

If you’re developing the @quilt/quilt package, you must also tell Node.js to use your local development copy of the Quilt JavaScript bindings (when you use Quilt to run blueprints) by running:

$ npm link .

in the directory that contains your local Quilt source files. For each blueprint that uses the Quilt JavaScript bindings, you must also run:

$ npm link @quilt/quilt

in the directory that contains the blueprint JavaScript files.

To run the JavaScript tests for bindings.js, use the jscheck build target:

$ make jscheck

Contributing Code

We highly encourage contributions to Quilt from the Open Source community! Everything from fixing spelling errors to major contributions to the architecture is welcome. If you’d like to contribute but don’t know where to get started, feel free to reach out to us for some guidance.

The project is organized using a hybrid of the Github and Linux Kernel development workflows. Changes are submitted using the Github Pull Request System and, after appropriate review, fast-forwarded into master. See Submitting Patches for details.

Go Coding Style

The coding style is as defined by the gofmt tool: whatever transformations it makes on a piece of code are considered, by definition, the correct style. Unlike official go style, in Quilt lines should be wrapped to 89 characters. To make sure that your code is properly formatted, run:

$ make golint

Running make format will fix many (but not all) formatting errors.

JavaScript Coding Style

Quilt uses the AirBnb JavaScript style guide. To make sure that your JavaScript code is properly formatted, run:

$ make jslint

Git Commits

The fundamental unit of work in the Quilt project is the git commit. Each commit should be a coherent whole that implements one idea completely and correctly. No commits should break the code, even if they “fix it” later. Commit messages should be wrapped to 80 characters and begin with a title of the form <Area>: <Title>. The title should be capitalized, but not end with a period. For example, provider: Move the provider interfaces into the cloud directory is a good title. When possible, the title should fit in 50 characters.

All but the most trivial of commits should have a brief paragraph below the title (separated by an empty line), explaining the context of the commit. Why the patch was written, what problem it solves, why the approach was taken, what the future implications of the patch are, etc.

Commits should have proper author attribution, with the full name of the commit author, capitalized properly, with their email at the time of authorship. Commits authored by more than one person should have a Co-Authored-By: tag at the end of the commit message.

Submitting Patches

Patches are submitted for inclusion in Quilt using a Github Pull Request.

A pull request is a collection of well formed commits that tie together in some theme, usually the larger goal they’re trying to achieve. Completely unrelated patches should be included in separate pull requests.

Pull requests are reviewed by one person: either by a committer, if the code was submitted by a non-committer, or by a non-committer otherwise. You do not need to choose a reviewer yourself; quilt-bot will randomly select a reviewer from the appropriate group. Once the reviewer has approved the pull request, a committer will merge it. If the reviewer requests changes, leave a comment in the PR once you’ve implemented the changes, so that the reviewer knows that the PR is ready for another look.

It should be noted that the code review assignment is just a suggestion. If a another contributor, or member of the public for that matter, happens to do a detailed review and provide a +1 then the assigned reviewer is relieved of their responsibility. If you’re not the assigned reviewer, but would like to do the code review, please comment in the PR to that effect so the assigned reviewer knows they need not review the patch.

We expect patches to go through multiple rounds of code review, each involving multiple changes to the code. After each round of review, the original author is expected to update the pull request with appropriate changes. These changes should be incorporated into the patches in their most logical places. I.E. they should be folded into the original patches or, if appropriate inserted as a new patch in the series. Changes should not be simply tacked on to the end of the series as tweaks to be squashed in later – at all stages the PRs should be ready to merge without reorganizing commits.

Code Structure

Quilt is structured around a central database (db) that stores information about the current state of the system. This information is used both by the global controller (Quilt Global) that runs locally on your machine, and by the minion containers on the remote machines.

Database

Quilt uses the basic db database implemented in db.go. This database supports insertions, deletions, transactions, triggers and querying.

The db holds the tables defined in table.go, and each table is simply a collection of rows. Each row is in turn an instance of one of the types defined in the db directory - e.g. Cluster or Machine. Note that a table holds instances of exactly one type. For instance, in ClusterTable, each row is an instance of Cluster; in ConnectionTable, each row is an instance of Connection, and so on. Because of this structure, a given row can only appear in exactly one table, and the developer therefore performs insertions, deletions and transactions on the db, rather than on specific tables. Because there is only one possible table for any given row, this is safe.

The canonical way to query the database is by calling a SelectFromX function on the db. There is a SelectFromX function for each type X that is stored in the database. For instance, to query for Connections in the ConnectionTable, one should use SelectFromConnection.

Quilt Global

The first thing that happens when Quilt starts is that your blueprint is parsed by Quilt’s JavaScript library, quilt.js. quilt.js then puts the connection and container specifications into a sensible format and forwards them to the engine.

The engine is responsible for keeping the db updated so it always reflects the desired state of the system. It does so by computing a diff of the config and the current state stored in the database. After identifying the differences, engine determines the least disruptive way to update the database to the correct state, and then performs these updates. Notice that the engine only updates the database, not the actual remote system - cloud takes care of that.

The cloud takes care of making the state of your system equal to the state of the database. cloud continuously checks for updates to the database, and whenever the state changes, cloud boots or terminates VMs in you system to reflect the changes in the db.

Now that VMs are running, the minion container will take care of starting the necessary system containers on its host VM. The foreman acts like the middle man between your locally run Quilt Global, and the minion on the VMs. Namely, the foreman configures the minion, notifies it of its (the minion‘s) role, and passes it the policies from Quilt Global.

All of these steps are done continuously so the blueprint, database and remote system always agree on the state of the system.

Quilt Remote

As described above, cloud is responsible for booting VMs. On boot, each VM runs docker and a minion. The VM is furthermore assigned a role - either worker or master - which determines what tasks it will carry out. The master minion is responsible for control related tasks, whereas the worker VMs do “the actual work” - that is, they run containers. When the user specifies a new container the config file, the scheduler will choose a worker VM to boot this container on. The minion on the chosen VM is then notified, and will boot the new container on its host. The minion is similarly responsible for tearing down containers on its host VM.

While it is possible to boot multiple master VMs, there is only one effective master at any given time. The remaining master VMs simply perform as backups in case the leading master fails.

Developing the Minion

If you’re developing code in minion, you’ll need to do some extra setup to test your new code. To make Quilt run your local version of the minion image, and not the default Quilt minion image, follow these steps:

  1. Create a new empty repository on your favorite registry - docker hub for example.
  2. Modify quiltImage in cfg.go to point to your repo.
  3. Modify Version in version.go to be “latest”. This ensures that you will be using the most recent version of the minion image that you are pushing up to your registry.
  4. Create a .mk file (for example: local.mk) to override variables defined in Makefile. Set REPO to your own repository (for example: REPO = sample_repo) inside the .mk file you created.
  5. Create the docker image: make docker-build-quilt
    • Docker for Mac and Windows is in beta. See the docs for install instructions.
  6. Sign in to your image registry using docker login.
  7. Push your image: make docker-push-quilt.

After the above setup, you’re good to go - just remember to build and push your image first, whenever you want to run the minion with your latest changes.

Security

TLS

Quilt uses grpc for communication with the daemon and deployed clusters. Functionality exposed through grpc include deploying new blueprints and querying deployment information. Thus, TLS should be enabled for all non-experimental deployments. It is currently disabled by default.

Quickstart

Generate the necessary TLS files.

$ quilt setup-tls ~/.quilt/tls

Start the daemon with TLS enabled.

$ quilt daemon -tls-dir ~/.quilt/tls

Use the other Quilt commands as normal.

$ quilt run ./example.js
$ quilt show
MACHINE         ROLE      PROVIDER    REGION       SIZE         PUBLIC IP         STATUS
8a0d2198229c    Master    Amazon      us-west-1    m3.medium    54.153.11.92      connected
b92d625c6847    Worker    Amazon      us-west-1    m3.medium    52.53.170.129     connected

CONTAINER       MACHINE         COMMAND                     HOSTNAME  STATUS     CREATED           PUBLIC IP
1daa461f0805    b92d625c6847    alpine tail -f /dev/null    alpine    running    24 seconds ago    52.53.170.129:8000

However, trying to connect to a cluster with different credentials fails. This can be simulated by restarting the daemon and running the same blueprint, but with different TLS credentials. Note that the machines never connect.

$ quilt daemon -tls-dir ~/.quilt/other-credentials
$ quilt run ./example.js
$ quilt show
MACHINE         ROLE      PROVIDER    REGION       SIZE         PUBLIC IP        STATUS
8a0d2198229c    Master    Amazon      us-west-1    m3.medium    54.153.11.92     connecting
b92d625c6847    Worker    Amazon      us-west-1    m3.medium    52.53.170.129    connecting

Trying to connect in Insecure mode also fails. Note that the machines never connect.

$ quilt daemon
$ quilt run ./example.js
$ quilt show
MACHINE         ROLE      PROVIDER    REGION       SIZE         PUBLIC IP        STATUS
8a0d2198229c    Master    Amazon      us-west-1    m3.medium    52.53.170.129    connecting
b92d625c6847    Worker    Amazon      us-west-1    m3.medium    54.153.11.92     connecting

Setup

The certificate hierarchy can be easily created using the setup-tls subcommand. For example,

$ quilt setup-tls ~/.quilt/tls

Will create the file structure described in tls-dir. No additional setup is necessary – the -tls-dir flag can now be set to your chosen TLS directory.

tls-dir

TLS is enabled with the -tls-dir option. The TLS directory should have the following structure when passed to quilt daemon:

$ tree ~/.quilt/tls
├── certificate_authority.crt
├── certificate_authority.key
├── quilt.crt
├── quilt.key

Other files in the directory are ignored by Quilt.

admin-ssh-private-key

The daemon installs keys using SFTP, so the daemon requires SSH access to the machines. By default, the daemon generates an in-memory key to use for distributing keys. A key can be specified from the filesystem using the admin-ssh-private-key flag.

For example,

$ quilt daemon -admin-ssh-private-key ~/.quilt/id_rsa -tls-dir ~/.quilt/tls

Frequently Asked Questions

This section includes answers to common questions about Quilt, and solutions to various issues. If you run into an issue and can’t find the answer here, don’t hesitate to email us at discuss@quilt.io.

I tried to quilt run a blueprint on Amazon and nothing seems to be working.

If you’re running a blueprint on AWS and the containers are not getting properly created, you may have an issue with your VPC (Virtual Private Cloud) settings on Amazon. When this issue occurs, if you run quilt show, the machines will all have status connected, but the containers will never progress to the scheduled state (either the status will be empty, or for Dockerfiles that are built in the cluster, the status will say built). This issue only occurs if you’ve changed your default VPC on Amazon, so if you don’t know what a VPC is or you haven’t used one before on Amazon, this is probably not the issue you’re experiencing.

If you previously changed your default VPC on Amazon, it may be configured in a way that prevents Quilt containers from properly communicating with each other. This happens if the subnet configured for your VPC overlaps with 10.0.0.0/8, which is the subnet that Quilt uses. This problem can manifest in many ways; typically it looks like nothing seems to be working correctly. For example, a recent user who experienced this problem saw the following logs on the etcd container on a Quilt worker:

2017-06-05 21:14:40.823760 W | etcdserver: could not get cluster response from http://10.201.160.199:2380: Get http://10.201.160.199:2380/members: dial tcp 10.201.160.199:2380: getsockopt: no route to host
2017-06-05 21:14:40.823787 W | etcdmain: proxy: could not retrieve cluster information from the given urls

You can check (and fix) your VPC settings in the VPC section of the online AWS console.