Skip to main content

Australian Synchrotron building Archive for a quick recovery

· 2 min read
Stephen Dart

Archive Advantages​

The Australian Synchrotron Beamline Science IT team have established an archiving process to accommodate large collections of experiment data. Rather than storing millions of separate files, whole experiments are grouped into manageable bundles (~5TB) using tools such as gzip, tar and SquashFS. This provides many advantages;

  • Recall of content is usually just a few bundles with few delays and not millions of files with millions of delays,
  • Data retains internal consistency with any databases enclosed,
  • The data bundle can be exported to the investigator at a remote location,
  • Removal to an archive frees online storage for new experiments.

The preferred tool has been SquashFS because the process results in an archive image that can be mounted on a Linux system, or have files extracted by a freely available utility (7-zip) on a MS-Windows desktop.

Getting Started​

Split your data into the data you need to do the day to day work, and other data that you will inevitably need later. The day-to-day has to be online and at a performance that allows good use of the compute facilities. The not-so-everyday is candidate for archiving and your preferred retrieval method should help you choose what form that archiving should take.

The collection owner usually has a technical contact that can bundle the content by their preferred method prior to being archived in the Vault storage service.

Collaboration of Australian Synchrotron and Monash eResearch​

The Australian Synchrotron has been reliant on the VicNode RDS storage operated the Monash University Operating Centre. Monash runs an IBM SONAS disk front-end with a TSM/HSM back-end tape system for its Vault service. Its general purpose nature does require some planning for archival use.

Planning your detailed overall data management can consume a large amount of time. Using the straight forward technique like that adopted by the Australian Synchrotron Beamline Science IT team described here can make your future a lot less painful.