-
Notifications
You must be signed in to change notification settings - Fork 4
GettingStarted
Cmd.io is a service that remotely runs commands and scripts. The primary interface to Cmd.io is SSH, meaning you can use Cmd.io commands from any host with zero configuration. A couple of use cases for Cmd.io include:
-
CLI utilities as a service. Example, bring
jqwherever you go, without installing it. -
Avoid installing big CLI programs. Example,
latexis huge and difficult to build but used infrequently. - Scripts you can use from anywhere. Trigger and orchestrate other systems using any language.
- Share and control access to automation. Build tools for your team or Rickroll your friends.
Cmd.io commands can run any x86 64-bit program in an Linux Docker container. However, it's designed for commands and scripts as opposed to long-running daemons. In fact, commands can only run for a limited time before they timeout. So Cmd.io is also not suitable for replacing your shell. Kudos for such a clever idea, though.
Commands are unable to listen on routable ports, which is not common with most commands anyway. In the rare cases you need to, such as debugging with netcat, consider using a tool like ngrok.
Another important constraint of commands is that they are stateless. Besides configured environment variables, nothing persists across command runs. If the command makes a file, it will not be there the next time you run it. Utilities that expect files on the filesystem will need to be wrapped so they can receive the file(s) via STDIN.
We'd love it if you took notes about your experience getting started and then using Cmd.io. You can then share on UserNotes and we can use them as living documents to improve the experience. Also, if something is wrong on this page or any other, feel free to edit this wiki!
This document is currently for the alpha release channel (alpha.cmd.io) ... any accidental references to the cmd.io domain should be read as alpha.cmd.io
Cmd.io uses your GitHub user for authentication. It also relies on the public keys stored with your GitHub account. If you haven't uploaded a public key to GitHub, you can easily add one in Settings under SSH and GPG keys.
Now you can connect to Cmd.io over SSH:
$ ssh progrium@alpha.cmd.io
Usage:
ssh <user>@cmd.io [command]
Available Commands:
:add Install a command
:ls List installed commands
:rm Uninstall a command
Use "[command] --help" for help about a meta command.
If you're using a local user with the same username as your GitHub username, SSH will use it by default. If not, you can also avoid specifying a username with some configuration added to your ~/.ssh/config file:
Host alpha.cmd.io
User progrium
When you start, the only commands available will be the root meta commands. Meta commands are prefixed with : and can either be root commands or meta commands on the commands you install. We'll talk about the former first. Root commands deal with managing your Cmd.io commands. For example, we can install a command with :add. We can see how with --help:
$ ssh alpha.cmd.io :add --help
Install a command
Usage:
ssh <user>@cmd.io :add <name> <source>
Although this help could be more helpful, it does at least tell us the arguments to :add. The first, name, is a name you choose for the command. The second, source, is where to get the command. Right now, the only supported sources are public Docker registries like Docker Hub. Let's use this mysterious Docker image, progrium/welcome:
$ ssh alpha.cmd.io :add demo progrium/welcome
Command installed
$ ssh alpha.cmd.io
Usage:
ssh <user>@cmd.io [command]
Available Commands:
:add Install a command
:ls List installed commands
:rm Uninstall a command
demo
Use "[command] --help" for help about a meta command.
Although it lacks a description, the command demo is now available. Run it:
$ ssh alpha.cmd.io demo
Hello! This is a cmd.io command. All it does is display this message.
However, cmd.io commands can do lots more. They can pretty much do
anything you can do in a Docker container, except for long-running
processes like daemons.
You can install cmd.io commands from a number of sources, including
anything off Docker Hub. Once you have a command installed, you can
configure it and share access to it. Anybody that has access to your
command can run it from anywhere they have an SSH client.
Ignoring the fact this message is lying to you (Docker Hub is currently the only source for commands), you've run your first Cmd.io command!
The container images you install don't need to be specific to Cmd.io. Pretty much any CLI tool in a container can be installed. Here's a netcat container we can use to show that cmd.io port 22 is open:
$ ssh alpha.cmd.io :add nc gophernet/netcat
Command installed
$ ssh alpha.cmd.io nc -z -v cmd.io 22
cmd.io (159.203.159.60:22) open
You'll probably be using your own commands more than off-the-shelf commands. Right now, since Docker Hub is the only source for commands, making and publishing commands is as easy as any Docker container. After you've gone through this once, you might be surprised at how quickly you can make and update Cmd.io commands. It's literally build, push, use.
Up to this point, we haven't needed to use or install Docker. For the time being, you'll need Docker to make Cmd.io commands. We highly recommend Docker for Mac if you're running macOS.
You'll also need a Docker Hub account and be sure to login with Docker (docker login).
The recommended way to build commands from existing open source utilities is to install them via a package manager. To keep your experience snappy, and since Cmd.io may enforce a size limit on images, we highly encourage you to use Alpine Linux for all command containers.
Alpine combines the small size of Busybox (~5MB) with a large package index optimized for small disk footprints. You can search for packages based on name or based on contents. If you can't find a package for a utility, you can try using ubuntu-debootstrap, which is a minimal Ubuntu image with apt-get. However it starts at ~90MB and easily bloats from there.
Here is jq for Alpine v3.4, so we can make a container for it with a simple Dockerfile that uses the apk package tool:
FROM alpine:3.4
RUN apk add --update --no-cache jq
ENTRYPOINT ["/usr/bin/jq"]
The directives used in this example are nearly all that make sense to use for Cmd.io commands, but here is the full Dockerfile reference.
Now we can build this with Docker, assuming we're in the directory with the Dockerfile. Immediately after building, we can push to Docker Hub. Replace progrium with your Docker ID.
$ docker build -t progrium/jq .
...
$ docker push progrium/jq
...
At this point you can now install this command on Cmd.io like before. If you push new versions of the image to Docker Hub, Cmd.io will pull it just before the next run.
Making a container for a script is not that different from making it for an existing utility. You'll want to install the interpreter and any other utilities the script depends on the same way as before. But you'll also be adding your script and making it the entrypoint.
Create a file called netpoll and make sure it's executable with chmod +x netpoll. Inside it, put this Bash script that uses netcat to poll an address and port for roughly 10 seconds or until the port accepts a connection. If it connects it returns. If it times out it returns non-zero. A rather handy little script.
#!/bin/bash
for retry in $(seq 1 ${TIMEOUT:-10}); do
nc -z -w 1 "$1" "$2" && break
done
In the same directory create a Dockerfile like this:
FROM alpine:3.4
RUN apk add --update bash netcat-openbsd
COPY ./netpoll /bin/netpoll
ENTRYPOINT ["/bin/netpoll"]
Build, push, and install with Cmd.io. Let's say I installed it as netpoll. I can run it against cmd.io port 22 and it returns immediately with status 0. Run against cmd.io port 23 and it blocks for at least 10 seconds before giving up and returning status 1.
$ ssh alpha.cmd.io netpoll cmd.io 22; echo $?
0
$ ssh alpha.cmd.io netpoll cmd.io 23; echo $?
1
Some of the real power in Cmd.io comes from what you can do with meta commands. Let's use the :help meta command on our netpoll command from before to see what we can do:
$ ssh alpha.cmd.io netpoll:help
Usage:
ssh <user>@cmd.io netpoll:[command]
Available Commands:
:access Manage command access
:admins Manage command admins
:config Manage command configuration
Use "[command] --help" for help about a meta command.
You can explore :access and :admins on your own. In short, access lets you share a command with others by adding and removing GitHub usernames. They can run it prefixed with your username and a slash. For example, if I shared netpoll with you, you could run it with ssh cmd.io progrium/netpoll. Access also lets you make a command public, letting any user run it, or make it private again.
The difference between access and admins is currently that admins have access to these meta commands just like you. Users with access don't. A useful dynamic this allows is storing credentials in configuration, and users with just access can use your commands that use those credentials, but aren't able to see the credential themselves.
However, if there is a way to display the environment in your command, those credentials will be plainly visible. Use at your own risk.
Let's look at the :config meta command:
$ ssh alpha.cmd.io netpoll:config --help
Manage command configuration
Usage:
demo :config [command]
Available Commands:
set Manage command configuration
unset Manage command configuration
Use "[command] --help" for help about a meta command.
What this help is not telling you is that running :config without --help will list current configuration values. However if you do that now, there will be no output since there is no configuration. Let's set configuration on netpoll.
Configuration is exposed to commands as environment variables, so you can usually just think of :config as managing environment variables. If you look back, our netpoll script actually uses a variable if set called TIMEOUT, which defaults to 10. We can change that value by setting TIMEOUT:
$ ssh alpha.cmd.io netpoll:config set TIMEOUT=30
Config updated.
$ ssh alpha.cmd.io netpoll:config
TIMEOUT=30
Now if you use netpoll against a port that's not accepting connections, it's going to loop 30 times instead of 10. This will apply to any user that has access to this command.
There are a number of patterns and recipes to cover elsewhere in this wiki, but I'm going to share this one here. You can use Cmd.io commands over SSH as Git remote endpoints to implement and react to git push from your repositories. Specifically, you can use one command for Git remotes: git-receive-pack.
You see, all that happens when you git push to an SSH remote like git@github.com:gliderlabs/cmd.git is it translates to the SSH command ssh git@github.com git-receive-pack 'gliderlabs/cmd.git'. This opens a session to github.com and runs the command git-receive-pack with the repo path argument. The actual git-receive-pack command then reads packed Git data over STDIN and applies it to the bare repository on the filesystem that was provided as an argument.
What Heroku and later gitreceive, Dokku, Flynn, etc all do in some form effectively is wrap git-receive-pack and install a pre-receive hook into the repository (perhaps created on the fly) that does some task. Git is designed to display the output of that hook back to the user during the push, so from that hook script you can git archive to get a tar of what was pushed and do whatever you want with it.
Currently, you can do this with Cmd.io by creating a command named git-receive-pack, which will handle all Git pushes to Cmd.io that authenticate with your username. Cmd.io is doing nothing special to make this work, but now you need a command that will properly handle the push.
Here's an example to get you started. It involves three files: Dockerfile, git-receive, and pre-receive. Remember the last two need to have chmod +x run on them.
FROM alpine:3.4
RUN apk add --update --no-cache git sed bash
COPY ./git-receive /bin/git-receive
COPY ./pre-receive /hooks/
ENTRYPOINT ["/bin/git-receive"]
Note that anything your pre-receive script is going to use also needs to be installed in this Dockerfile.
#!/bin/bash
repo="$1"
if [[ "$repo" != /* ]]; then
repo="/$repo"
fi
git init --quiet --bare "$repo"
cp /hooks/pre-receive $repo/hooks
git-shell -c "git-receive-pack '$repo'"
This normalizes the repository argument, creates a bare repo in that location, installs pre-receive as a hook for that repository, and performs the actual git-receive-pack. This will then trigger pre-receive.
#!/bin/bash
main() {
# reads git push header data into variables
read old new ref
# use archive to tarpipe pushed branch files to a working directory
git archive "$new" | (cd /tmp && tar -xpf -)
# go to that directory
cd /tmp
# do something with the files!
# exit non-zero and the push will fail.
}
delete-remote-prefix() {
# this removes "remote: " that git prefixes hook output with client side
sed -u "s/^/"$'\e[1G'"/"
}
main | delete-remote-prefix
This is just a template but from here you could deploy, do builds, run checks, or something more creative.
Once you've made a command from the above and installed it as git-receive-pack, you can add a remote to the repositories you want to push from like this:
$ git remote add cmd ssh://progrium@cmd.io/repo/path
The name of the remote can be anything you like, here it's cmd. You can also drop the username if you were previously. The repo path can be anything as well, perhaps use it to determine what to do in your pre-receive script.
Unfortunately, git-receive-pack is not sharable. Perhaps in the near future there will be more integrated support in Cmd.io for this pattern.
Now have fun! Hopefully you took notes during this process that you can add to UserNotes. We'd love a record of your feedback. Keep taking notes about your experience and perhaps use them to document clever uses or patterns you discover.
See you in Slack!