NFS Backups

Introduction

While TIG maintains both drive redundancy (RAID) and snapshots of most NFS filesystems, that is sometimes insufficient for data preservation: a server might experience a catastrophic multi-disk failure, or a user might not notice that an important file was corrupted before the last snapshot containing the correct data expires. For this reason, we take periodic backups to tape, and send some of these backups to a physical off-site storage facility. Filesystems in the archival storage tier are backed up weekly, and those in the production tier are backed up nightly between 10 PM and 6 AM Eastern Time.

Backups cannot begin on a new filesystem until a significant amount of data has been written. How much is “significant” depends on how the data in the filesystem is being used, and in particular how much of the initial data is expected to change after it is first written. If too much of the initial data changes or is deleted, backups will automatically cease and require administrator intervention to resume (usually after taking a new initial backup). We recommend waiting until at least 40% of your quota is used, but this is not a hard requirement and backups can be started at any time so long as some permanent data is being stored.

Requesting a restore

If you delete or overwrite a file on NFS, and you are certain it existed in the desired form for at least 24 hours, you can request the backup team restore it from tape by sending email to help@csail. In your request, make sure to include the full pathname of the file or files that you want restored, and as close to an exact date range as you are able to identify from when the file last existed.

Before requesting a restore, please check for snapshots in the hidden .zfs/snapshot directory below the mount point of the filesystem to see if the version of the file you need is available there.

Limits on tape backups

Due to the costs of tape storage and the difficulty of reading obsolete tape formats, we cannot promise to be able to restore data more than two years old. In addition, date resolution is reduced for older backups: daily incremental tapes are recycled quickly; weekly tapes are kept longer; and monthly and full tapes are kept for at least two years. Only monthly and full tapes have copies sent to off-site storage (although one copy is retained in Stata to reduce the latency of restore requests).