Skip to main content

Vault Storage User Guide

Overview​

The RDSM Vault storage can be made accessible by the following

  1. Windows shares (SMB)
  2. NFS shares
  3. SFTP
  4. rsync
  5. Aspera
  6. Globus

Once an NFS or Windows share has been mapped to a machine, it can be accessed similarly to local storage and will support standard actions like copying files. etc. SFTP and rsync shares are usually accessed via the command line or through a client depending upon the user's preference and operating system.

Windows shares / SMB​

SMB shares can be mapped on virtually any operating system and are best used for user or group shares. SMB shares utiliseuser-based authentication are secured by unique secure access groups linked to each share which contain a list authorised Monash IDs. Because users need to enter their personal username and password when mapping a share, it is recommended that users only mount SMB shares on their own machines rather than shared machines such as instrument PCs. If a share does need to be mounted on a shared machine then the RDSM team can be contacted in order to discuss the most appropriate options.

Access to the shares is controlled by a user group for SMB shares (via the University GroupAdmin access group control)

NFS shares​

NFS mounts are used by non-Windows operating systems such as Linux and macOS and are accessible only to specific machines with pre-approved hostnames or IP addresses. Because NFS shares do not require user-based authentication, they are often used to make remote storage available to machines hosting services or applications such as web servers.

SFTP​

SFTP is a secure file transfer protocol that operates via SSH. Once a secure connection has been established to the Vault service, data can be transferred between the Vault and another machine as required using SFTP or SSH-FS. SFTP is generally used by Linux and macOS machines and is also used by MASSIVE, MonARCH and other HPC services. Linux and macOS machines natively support SFTP and a client can be used if SFTP/SSH-FS access is required on Windows. Depending on the requirements of the Collection Owner, each users may have access to the entire shared fileset, or only a specific path within the fileset folder structure.

rsync​

The rsync program is an application that runs natively on Linux and macOS and is primarily used to replicate file and folder trees and maintain a duplicate copy. This is commonly used as a way to back up a collection of data and is often scheduled to run at specific times during the day. Access to Vault shares is available using rsync over SSH and it should be noted that rsync modules are not currently supported.

rclone​

The rclone program operates in a similar fashion to rsync, however it is more recent and supports the ability perform multi-threaded writes, and to write direct to SMB end-points. Writing direct to the SMB end-point (vault or market) avoids having to mount the storage and can result in much higher throughput. The program is available on Windows, macOS and Linux either through their packaging system or from https://rclone.org. On the massive HPC system it can be accessed through the use of the command module add rclone

Accessing Vault Shares​

The process for mapping Vault storage varies depending upon the storage type and the operating system being used and therefore mapping instructions for the three most common environments (Windows, macOS, Linux) have been provided below. Note that NFS shares can be mounted on Windows machines that run specific versions (Windows Vista, Windows 7 and Windows 10), but there are a number of technical issues. As a result, it is not recommended but the RDSM team can provide assistance if NFS access on Windows is required.

Accessing Vault shares on Windows via SMB​

  • Open the Windows Explorer.
  • Right-click on "Computer" on the left-hand side.
  • Select "Add a network location" and then click on "Next".
  • Select "Choose a custom network location" and then click on "Next".
  • Enter the provided URL (e.g. \\vault-v2.erc.monash.edu.au\<SomeShare>) and then click on "Next".
  • Enter your Monash ID and password (if prompted). Note that if you are not using a Monash SOE (Monash-supplied and configured) machine, you may need to add "MONASH" to your Monash username (e.g. MONASH\jsmith).
  • Enter a name of your choice, e.g. "RDS SomeShare" and click on "Next".
  • Review the information presented and then "Finish".
  • The newly-mapped share will then appear as a folder in the Windows Explorer.

Accessing Vault shares on macOS via SMB​

  • Access the "Go" menu which is available at the top of the screen from the desktop.
  • Click on "Connect to Server".
  • Enter the provided URL (e.g. smb://vault-v2.erc.monash.edu.au/<SomeShare>) into the textbox.
  • Click on "Connect".
  • Enter your Monash ID and password details (if required) and click on "Connect". Note that if you are not using a Monash SOE (Monash-supplied and configured) machine, you may need to add "MONASH" to your Monash username (e.g. MONASH\jsmith).
  • The newly-mapped share will then appear as a folder in the macOS Finder.

Accessing Vault shares on Linux via SMB​

Linux SMB Using X​

  • The mapping process via X is essentially the same as macOS.

Linux SMB from the command line​

  • SMB shares can be mapped using the "mount" command which uses the following syntax: mount -t cifs //vault-v2.erc.monash.edu/<SomeShare> /<LocalDirectory> -o user=<Username>,domain=MONASH,vers=3.0.
  • Note that the each share needs to be mapped to a local directory (/<LocalDirectory> above) which should be created before the above command is used.
  • Enter your Monash password and press Enter.
  • Unless an error message is displayed, the share has been mapped properly and will be available via the nominated directory (/<LocalDirectory>).
  • The above command should work for most Linux distributions, but there are other commands that can be used such as smbmount. If you encounter any issues, please contact the RDSM team.
  • Shares can be removed by using the following command: umount /<LocalDirectory>.

Accessing Vault shares on Windows via NFS​

  • If NFS access on Windows is required, please contact the RDSM team.

Accessing Vault shares on macOS via NFS​

  • Access the "Go" menu which is available at the top of the screen from the desktop.
  • Click on "Connect to Server".
  • Enter the provided URL (e.g. nfs://vault-v2.erc.monash.edu:/<SomeShare>) into the textbox.
  • Click on "Connect".
  • The newly-mapped share will then appear as a folder in the macOS Finder.

Accessing Vault shares on Linux via NFS​

Linux NFS Using X​

  • The mapping process via X is essentially the same as macos.

Linux NFS from the command line​

  • NFS shares can be mounted using the "mount" command which uses the following syntax: mount -t nfs //vault-v2.erc.monash.edu:/<SomeShare> /<LocalDirectory>.
  • Note that the each share needs to be mapped to a local directory (/<LocalDirectory> above) which should be created before the above command is used.
  • Unless an error message is displayed, the share has been mapped properly and will be available via the nominated directory (<LocalDirectory>). If you encounter any issues, contact the RDSM team.
  • Shares can be removed by using the following command: umount /<LocalDirectory>.

Accessing Vault shares via rsync or SFTP​

In order to gain access to a Vault share via rsync or SFTP, it is necessary to create an SSH key. If you don't already have an SSH key, please see below. Once you have an SSH key, email MeRC-RDSM-Support@monash.edu, provide the share name and the MonashID that should be granted access and attach your public SSH key.

For users that need to browse content from the desktops, at least two GUI clients are known to work effectively. This approach is generally good enough to support browsing folder structures and uploading/downloading desktop files. Users of wireless connections should take their connection speed into account when moving large quantities of data.

Additional steps need to be performed by RDS administrative staff so that the Vault share appears for the user when they login via SFTP or rsync. The Owner/PI of the storage allocation should contact the RDSM team to enable this for each new account. By default, users are generally granted access to the entire share. Access can be restricted to a specific sub-folder tree within the fileset. Please ensure that a sub-folder path does already exist if restricted access is required and the path is specified in the job ticket. The SFTP/rsync service on the Vault runs within a captive shell and as such attempting a SSH command line session WILL NOT work as you expect. Users should keep to the requested protocol and recommended clients.

Security restrictions implemented on the Vault storage include authentication failure lockout (Fail2Ban) that may be triggered by the users client presenting a series of bad passwords or multiple SSH keys when none have been set up.

While off campus, Monash users are required to use SSH keys and/or the Monash VPN.

For routine and repeated access beyond the Monash domain, external users need to have their sites explicit IP Address added to the "White List" of permitted sites. The Owner/PI of the storage allocation should contact the RDSM team to enable this.

FileZilla​

Settingvalue
ProtocolSFTP
Hostvault-v2.erc.monash.edu
Port22
UserMONASH\<MonashID>

WinSCP​

File protocol : SFTP Host name : vault-v2.erc.monash.edu Port number: 22 User name : MONASH\\<MonashID> Password: <SSH key>

Linux Command Line​

Users with SSH access to Linux hosts, including the M3 DTN and login nodes or NeCTAR VMs, can use the command line interactively to launch SFTP or rsync applications and use scripts to archive, locate and recall large quantities of data. These methods have slightly different syntax depending on the OS version of Linux.

The SFTP/rsync service on the Vault runs within a captive shell and as such attempting a SSH command line session on Vault WILL NOT work as you expect. Users should keep to the requested protocol.

There are issues around the use of the backslash character as part of the MONASH domain declaration because the backslash is also used as a text escape character. There are examples below that use one, two and even three backslash characters in a row to get the desired folder/file path interpreted properly.

Linux SFTP application command line examples​

Note that because the required separator between the MONASH domain and the account name is a backslash "\", it is necesary to escape this for the terminal.

Start SFTP session with the Vault

(requires two backslashes MONASH\\<MonashID>)

LocalHost:~ $ sftp MONASH\\<MonashID>@vault-v2.erc.monash.edu

SFTP put data to Vault from M3 login or dtn node

sftp> put <local-files> /home/MONASH\\<MonashID>/<share>/vault/<path>/

SFTP get data from Vault to M3 login or DTN node

(requires three backslashes MONASH\<MonashID>)

sftp> get /home/MONASH\\\<MonashID>/<sharename>/vault/<path>/

Using SSH Keys

Accounts created for SFTP and rsync on the Vault should be accessed through the use of SSH keys.

SSH keys have a number of advantages and are considered to be more secure than passwords.

In particular, SSH keys allow you to run scripts on your local terminal that will speed up repetitive actions.

Creating SSH Keys

Users should generate a specific key pair (public and private). eg. ssh-keygen -t rsa

The public key (e.g.: id_rsa.pub) is to be placed in the authorized_keys file in the .ssh folder in their accounts home directory. That allows login and use of any collection they have linked using the corresponding undisclosed private key.

The authorized_keys file in the .ssh folder of the users home directory is a special case that can be uploaded and downloaded by the user for their autonomous management of secure public keys.

Start SFTP session with the Vault

LocalHost:~ $ sftp MONASH\\<MonashID>@vault-v2.erc.monash.edu

SFTP get current authorized_keys file

sftp> cd .ssh
sftp> pwd
Remote working directory: /home/MONASH\<MonashID>/.ssh
sftp> ls

authorized_keys

sftp> get /home/MONASH\\<MonashID>/.ssh/authorized_keys
Fetching /home/MONASH\<MonashID>/.ssh/authorized_keys to authorized_keys
/home/MONASH\<MonashID>/.ssh/authorized_keys 100% 403 194.0KB/s 00:00

SFTP put updated current authorized_keys file

sftp> cd .ssh
sftp> pwd
Remote working directory: /home/MONASH\<MonashID>/.ssh
sftp> ls

authorized_keys

sftp> put authorized_keys

Uploading authorized_keys to /home/MONASH\<MonashID>/.ssh/authorized_keys

authorized_keys 100% 403 310.9KB/s 00:00}

If you are unable to access your home directory and .ssh folder, your public key should be emailed in plain text or uploaded as an attachment to RDSM Support either through an existing ticket in Freshdesk or via the Data Dashboard Support Ticket function.

Linux rsync Examples

  • Push to Vault using rsync** (generic, requires two backslashes MONASH\\<MonashID>)

rsync -aHWv --stats --progress /<local-folder-path>/ MONASH\\<MonashID>@vault-v2.erc.monash.edu:<sharename>/vault/<path>

  • RSync Push to Vault from QNAP

rsync -aHWv --stats --progress /share/CACHEDEV1\_DATA/ MONASH\\sdar0001@vault-v2.erc.monash.edu:sensilab/vault/

  • RSync Pull from Vault to QNAP

rsync -aHWv --stats --progress MONASH\\sdar0001@vault-v2.erc.monash.edu:sensilab/vault/sensiLab/Media/Photo/SensiLab/RawMedia/2018\_09\_27/ /share/CACHEDEV1\_DATA/Public/Restored/

  • RSync Pull from Vault to m3-dtn

`rsync -aHWv --stats --progress MONASH\sdar0001@vault-v2.erc.monash.edu:sensilab/vault/sensiLab/Media/Photo/SensiLab/RawMedia/2018_09_21/ /scratch/rdsmtest/2018_09_29

  • Using rsync with ssh key pairs

rsync -aHWv -e "ssh -i /home/ubuntu/.ssh/id\_rsa" --stats --progress /<local-folder-path>/ MONASH\\<MonashID>@vault-v2.erc.monash.edu:<sharename>/vault/<path>

Example: rsync -aHWv -e "ssh -i /home/ubuntu/.ssh/id\_rsa" rsynctest MONASH\\dlam@vault-v2.erc.monash.edu:BMH-archive/vault/<path>

Accessing Vault shares using SSH Keys​

Users with SFTP or rsync shares on Vault-v2 can request access to their shares through the use of SSH keys. SSH keys can be used to authorise access to Vault shares and can be used in place of Monash ID and password authentication, which is useful for things like cronjobs or scripts which need to run without any user interaction.

  • Vault-v2 SSH key access configuration setup

When access to the Vault service via SSH key is requested, the following should be noted:

  1. The account you use to login to Vault-v2 must have an associated home directory in order that you can log in and see the home directory contents. Your home directory is not a location to which you can write directly like your Vault share.
  2. A link to the requested share is placed in your home directory, so you can use the same account and any valid key contained therein for several shares.
  3. The configuration is replicated to the four nodes of the cluster, so even if your session is load-balanced for performance it will work normally.
  4. The account used must have rights to the share and root must be a member of the secure access group (RDS-$group-vault) that has rights to the share. Accounts can be added using the Group Admin tool by Owners on the Group.
  5. Changes to the Vault configuration (e.g. new shares and new users) require the service to restart and all shares are automatically re-linked (using bind) to the home directories that they are configured to access. This may interrupt a session in progress so we recommend planning ahead.
  • Process

The first step is to generate a SSH key pair (public and private) if they do not already exist. One method is to use the commandline tools in Linux/macOS (ssh-keygen -t rsa) but there are other methods. Once generated, the public key (e.g.: id_rsa.pub) must be added to the authorized_keys file in the .ssh folder in the relebant account's home directory. Doing so will allow users to log in and access any collection that has been linked to the corresponding key and upload/download data through the use of autonomous secure public key management.

  • SFTP "get" current authorized_keys file
sftp> cd .ssh
sftp> pwd
Remote working directory: /home/MONASH\\<MonashID>/.ssh

sftp> ls

authorized_keys

sftp> get /home/MONASH\\<MonashID>/.ssh/authorized_keys

Fetching /home/MONASH\<MonashID>/.ssh/authorized_keys to authorized_keys

/home/MONASH\<MonashID>/.ssh/authorized_keys                      100% 403 194.0KB/s 00:00

SFTP "put" updated current authorized_keys file

sftp> cd .ssh

sftp> pwd

Remote working directory: /home/MONASH\<MonashID>/.ssh

sftp> ls

authorized_keys

sftp> put authorized_keys

Uploading authorized_keys to /home/MONASH\<MonashID>/.ssh/authorized_keys

authorized_keys

100% 403 310.9KB/s  00:00

If you are unable to access your home directory and .ssh folder, the public key will need to be emailed in plain text or uploaded as an attachment to RDSM Support either through an existing ticket in Fresh Desk or via the Data Dashboard "Support Ticket" function.

  • Push to Vault using ssh key pairs

rsync -aHWv -e "ssh -i /home/ubuntu/.ssh/id\_rsa" --stats --progress /<local-folder-path>/ MONASH\\<MonashID>@vault-v2.erc.monash.edu:<sharename>/vault/<path>

Example: rsync -aHWv -e "ssh -i /home/ubuntu/.ssh/id\_rsa" rsynctest MONASH\\dlam@vault-v2.erc.monash.edu:BMH-archive/vault/<path>

rsync interaction with SMB and NFS shares​

Owing to the way that the permissions are managed on the vault filesystems there are edge cases where data written via one protocol cannot be read by another. The most common case is data written over a Windows fileshare and causing problems when being read via rsync(1). Folders can end up being created with Unix permissions "000", makeing them unwritable. A workaround for this can be to use the following options to rsync to force the permissions on the destination:

--no-p --no-g --chmod=ugo=rwX

(for detailed explanation search for "--perms" in the rsync(1) man page)

Accessing Vault shares via rclone​

The rclone utility can be used on client systems for copying files to and from the vault storage without requiring the storage to be mounted. It is available for Windows, Macintosh and Linux platforms either through the local package manager or from [https://rclone.org/]. HPC users should use it from one of the m3-dtn systems, accessed via module add rclone. Configuration and use is from a command line, a typical configuration will be generated via the command rclone config create vault smb --all then answering the following prompts (options "port" and "spn" can be left at their default).' Note, use your Monash account name and password

Option host.
SMB server hostname to connect to.
E.g. "eample.com".
Enter a value.
host> vault.erc.monash.edu
Option user.
SMB username.
Enter a string value. Press Enter for the default (fred).
user> jsmith

Option pass.
SMB password.
Choose an alternative below. Press Enter for the default (n).
y) Yes, type in my own password
g) Generate random password
n) No, leave this optional password blank (default)
y/g/n> y
Enter the password:
password:
Confirm the password:
password:

Option domain.
Domain name for NTLM authentication.
Enter a string value. Press Enter for the default (WORKGROUP).
domain> MONASH

:
[market]
type = smb
host = vault.erc.monash.edu
user = jsmith
pass = \*\*\* ENCRYPTED \*\*\*
domain = MONASH

The fileshares can then be accessed via commands such as rclone lsf vault:Share-Name, eg rclone ls market:R-Eng-xyz or rclone copy vault:Cat-Videos/Experiment1 .