Difference: ImagerTechnicalInformation (32 vs. 33)

Snapshot Failure Due To Large Data Changes

modified: 10 Nov 2008

• The following are answers from Mike Sanderson (tech staff) about the snapshot failure.
• When there is a snapshot failure due to large data change, what exactly is lost? If a snapshot fails, the entire snapshot will not be created. We take hourly, daily, and weekly snapshots. We retain 25 hourly, 8 daily, and 3 weekly snapshots. We currently have weekly snapshots from Nov 8, Nov 1, and Oct 25 run at 2:00 AM on netapp2. If the next weekly snapshot at 2:00 AM Nov 15 fails, we will retain the snapshots from Nov 8, Nov 1, and Oct 25. To clarify the problem we were having a little, it was not the snapshots in /ubc/cs/research that we were having problems with. The issue was on netapp3 - the filer dedicated to retaining weekly snapshot data. The intent was to retain 52 weeks of snapshots on netapp3, effectively replacing our tape backups with online disk-based backups. The combination of large changes in snapshot volume and disk space increases on netapp1 and netapp2 that were not mirrored on netapp3 resulted in 100% usage of the disk space on netapp3 - regular space and snapshot space. By deleting the oldest snapshots, we were able to do the transfers of the newest snapshots. Because the retention is so large, it takes much longer for the large snapshots to roll off the end and the space to be recovered.
• Note that (almost) all non -raid* filesystems in /ubc/cs/research/* share the same snapshot space as .../research/imager. The snapshot space is based on the total size of a "volume" - think of it as an entire disk - which we split up into research group space via quotas.
  • 1.3T 670G 565G 55% /ubc/cs/research/imager/.snapshot
  • 635G 213G 423G 34% /ubc/cs/home/.snapshot
• Would it be possible to get a directory that does not have snapshot utilities? The RAID space (disk space not on the filers) was intended for this purpose, though there are problems with this space. If NetApp filer space is needed because of performance needs or some other reason, this can be discussed. Tech staff will need some lead time with it however. We are approaching the available capacity of netapp2. Mike will be meeting with Bill this week to discuss a non-budgeted purchase of additional space.