Storage Products
A number of storage products are available to researchers at Monash University including:
Market | Vault | Computational | Object | Archive |
|---|
Each of the storage products above can be customised to accommodate specific needs depending upon the data that needs to be stored. Additional storage services include:
- Aspera
- Bridges
- Store.Monash (MyTardis)
Storage Product Overview
Each storage product supports different usecases and has both advantages and disadvantages. New storage allocations can be requested via the Data Dashboard.
Market
Security Classification
Market storage (also known as Market-file) is a disk-based storage product that is designed to support research data that is accessed regularly (usually every day) by researchers. Market storage closely resembles the Monash University S:\ Drive and is suitable for data that changes frequently and needs to be available for daily work but is too large to be stored on a local machine. Market storage can be presented as a desktop share/network drive (SMB) or can be mounted on a server (NFS).
Market storage can be provided for large collections, but for very large collections (more than 10TB) a combination of Market and Vault storage is recommended. For such collections, it is recommended that a smaller Market allocation should be used for data that is currently being used or analysed while older, unused data should be stored in a Vault allocation and moved to Market as required. Market storage can also be made accessible to the Internet (not recommended for sensitive data) using Aspera, a web-based service that is useful for sharing large amounts of data with external collaborators. Note that Market storage has less overall capacity than Vault storage and allocations are assigned individual quotas. Amendments and extensions for Market allocations can be requested through the Data Dashboard.
Vault
Security Classification
Vault storage is a tape-based product that is best used to store data that is not accessed frequently or needs to be retained for long periods of time. It is best used for archiving or for backups. Vault storage can be presented as a desktop share/network drive(SMB), can be mounted on a server (NFS) or can be accessed via SFTP or rsync.
Vault is commonly used to store large collections of data that is accessed infrequently, research data archives that need to be retained or backups of data that is stored locally. Vault allocations are often used alongside large Market allocations because data that is no longer required can be moved to Vault while data that needs to be analysed can be moved to the Market allocation as required.
Due to the way in which the Vault stores data it is recommended that users with large numbers of files (tens of thousands or more), store them as discrete collections (e.g. datasets) within an allocation using compression tools such as ZIP or TAR. This ensures that the data can be retrieved from tape more quickly and efficiently as well as reduce wait times. Large amounts of data can be manually recalled by the RDS team if necessary which is known as a "bulk recall". Bulk recalls are uncommon and are assessed on a case-by-case basis.
Vault storage can be made accessible to the Internet (not recommended for critical or sensitive data) using Aspera, which is useful for sharing large amounts of data with external collaborators. There is a significant amount of Vault capacity available and allocations are assigned individual quotas. Amendments and extensions for Vault allocations can be requested through the Data Dashboard.
Computational
Security Classification
Computational storage is a disk-based storage product that is often used when running parallel computational jobs on the Research Cloud. Research Cloud is a Victorian node of the Nectar service. Computational storage can only be used by Nectar VMs (Virtual Machines) and, as a result, the storage is inextricably linked to specific VMs called "instances" which are created for specific projects. Nectar VMs can be requested through the Nectar Dashboard while Computational storage should be requested through the Data Dashboard. Additional information about Computational storage can be found here.
Object
Security Classification
Object storage is a disk-based product that is organised differently to more traditional storage and is generally recommended for unstructured collections of data that will be written once and then read many times in an unpredictable fashion. Nectar's Introduction to object storage page provides a good summary of Object storage's features as well as related information.
Archive - MURDA
Security Classification
Archive storage, also known as the Monash University Research Data Archive (MURDA), is a tape-based product that is designed to store static collections of research data that will not be actively used but need to be retained for a specific period of time. Archive storage is designed to support the long-term retention of data produced by research projects and help to ensure compliance with any data sharing agreements, ethics agreements or other encumbrances.