Skip to content

Edit Storage System Definition

Andrew Magill edited this page Jul 11, 2016 · 26 revisions

#####Previous Step - Generate Agave API Authorization Token


Agave supports about a dozen authentication protocols for accessing systems, including iRODS and Amazon S3. I will provide a couple simple examples that will allow you to register a system using either username/password or SSH keys; refer to this documentation for a more comprehensive set of examples.

Option 1 - Authenticate with Username and Password:

Create a text file on your system called my-system.json (or another name that you prefer) in an easily accessible directory.

my-system.json:

{
  "id": "sftp.storage.example.com",
  "name": "Example SFTP Storage System",
  "type": "STORAGE",
  "description": "My example storage system using SFTP to store data for testing",
  "storage": {
    "host": "storage.example.com",
    "port": 22,
    "protocol": "SFTP",
    "rootDir": "/",
    "homeDir": "/home/apiuser",
    "auth": {
      "username": "apiuser",
      "password": "changeit",
      "type": "PASSWORD"
    }
  }
}
  • id - Any globally-unique (each system on CyVerse must be named distinctly) name that you choose to identify your system. Note that if another CyVerse user has already named a system 'Corral', or 'My Laptop', you will need to choose another name.
  • name - Arbitrary text, more of a short description than a name.
  • type - 'Storage', indicates that we will not run jobs on this sytem.
  • description - Arbitrary text.
  • host - The hostname or IP address that you would use to SSH into the machine (e.g. corral.tacc.utexas.edu).

######Please see the Agave API Systems Tutorial for a full explanation of these fields.

Option 2 - Authenticate with SSH keys:

Similar to above, create a text file on your system called my-system.json (or another name that you prefer) in an easily accessible directory.

You will need a public/private key pair on your the system that you are registering with Agave.

Log in to remote system:

$ ssh foobar@corral.tacc.utexas.edu

Create a directory for your SSH keys, if one does not already exist:

$ mkdir ~/.ssh

Generate new key-pair (unless you already have one you would like to use). You will be prompted for a filename, then a passphrase; you may choose the default filename ('id_rsa'), leave the passphrase blank.

$ cd ~/.ssh
$ ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

Copy and paste your public and private keys into the system description below. You can print you keys to the screen using 'cat':

$ cat id_rsa
$ cat id_rsa.pub

When you paste into the JSON system description file, you may want to remove line feeds, in other words, turn this:

x9XxWDwpY1gJl7c8ppxraLXc7/i5NSsOFwWHi3WsGfah0fVN+/CixQKkgQDTISxP
4WJgoaJxfzm7rwqbD7CPrOuBQXN5OFikQX0y//TZ9df5OsX1xxM406Jhzaj8zwJV

into this:

x9XxWDwpY1gJl7c8ppxraLXc7/i5NSsOFwWHi3WsGfah0fVN+/CixQKkgQDTISxP4WJgoaJxfzm7rwqbD7CPrOuBQXN5OFikQX0y//TZ9df5OsX1xxM406Jhzaj8zwJV

my-system.json:

{
  "id": "foobar-corral",
  "name": "Foobar's Corral Storage System",
  "type": "STORAGE",
  "description": "My storage system using SFTP to store data for testing",
  "storage": {
    "host": "storage.example.com",
    "port": 22,
    "protocol": "SFTP",
    "rootDir": "/",
    "homeDir": "/home/foobar",
    "auth": {
      "publicKey": "ssh-rsa AAAAB3NzaC ... RtUC7CY4SBw== foobar@tacc.utexas.edu",
      "privateKey": "-----BEGIN RSA PRIVATE KEY-----\nM.oI+n8Y ... lqU0g==\n-----END RSA PRIVATE KEY-----",
      "type": "SSHKEYS"
    }
  }
}

#####Next Step - Upload System Definition File