StorReduce is integrated with Amazon Glacier through Amazon S3. When StorReduce is configured to work with Glacier we say that it is running in “Glacier Mode”. This mode of operation is different from StorReduce’s normal mode of operation in a number of ways. The document covers these differences and how to use StorReduce with Glacier.
What is Amazon Glacier?
Amazon Glacier is a secure, durable, and extremely low-cost cloud storage service for data archiving and long-term backup. Please see the AWS web site for more information.
How Glacier Mode Works
When StorReduce’s Glacier Mode is enabled, StorReduce uses Amazon S3 as a staging area, all deduplicated data is first written to S3 and then a S3 Lifecycle Policy is used to transition (or tier) that data to Glacier after a configurable number of days in S3.
After the data has been tiered to Glacier, it can not be accessed until it is copied back into S3. The process of copying the day from Glacier into S3 is called restoring the data. When an object is restored back into S3, it remains in S3 for a configurable time, during which is can be accessed. After this period, the copy in S3 is deleted leaving the copy in Glacier as the only copy.
The process of restoring data from Glacier can take minutes to hours to days. The expected time to restore is configurable when a restore is started, StorReduce currently supports Expedited, Standard and Bulk restores which each have a different AWS cost profile. Please see the AWS web site for more details.
In order to restore an object you must either request a restore through the StorReduce Dashboard, use software like the AWS CLI to request a restore, or make a S3 API POST Object Restore call of type restore to a StorReduce endpoint.
Configuring Glacier Mode
Glacier Mode is enabled by checking the Enable Glacier Mode checkbox in the Storage section of the Settings tab on the StorReduce Dashboard, as shown below.
Once Glacier Mode is enabled and data has been stored through StorReduce, it should not be disabled without consulting StorReduce support.
Operating in Glacier Mode
After the StorReduce server or cluster has been restarted, you will notice a number of differences.
In the Browse Data tab on the StorReduce dashboard, objects will have a Storage Class of GLACIER.
In the Object Properties sheet for each object visible in the Browse Data tab, there is a extra field called Restore State that tells you if the object is accessible or not. See Restore States below.
Next to the Restore State field there is a button labelled Restore. If the button is bright blue then you can start restoring an object by clicking on it. See Restoring Objects below.
If you try to download an object, the download may fail with an InvalidObjectState error.
The Restore State of an object tells you if an object is currently accessible. It has these states.
Available: the object has not yet transtioned to Glacier, it is accessible (e.g., available for download) now, however it may transition to Glacier at anytime.
In Glacier: the object is in Glacier, you must restore it in order to access it.
Restore in progress: the object is being restored. This process may take sometime.
Available until DATE: the object can be downloaded until DATE. DATE is in the GMT or UTC timezone. Some unspecified time after DATE the available copy will be deleted, but the copy of the object in Glacier will remain.
Working: StorReduce is determining the state of the object, this process will usually take seconds, but for larger objects it could take minutes.
Error: there was an error while determining the state of an object, please try again later.
If you restore a "folder" then all "child" objects in that folder will be restored. If that folder contains a large number of objects then this could potentially be very expensive. Please check the fees on the AWS website and remember that restoring object in StorReduce may result in multiple objects being restored from Glacier.
The section outlines how to restore an object or objects via the StorReduce Dashboard, please see below for information on restoring object using the S3 API.
Starting the Restore Process
In order to restore an object, StorReduce has to first tell S3 to start the restore process for an object. This section explains how start the restore process for one or more objects.
A single object can be restored by clicking the Restore button in the Object Properties sheet of the Browse Data tab.
Multiple objects, including “folders” containing multiple “child” objects can be restored by checking the checkboxes next to the objects or folders and then selecting the Restore Selected Items from Glacier item in the Actions menu.
Both options will bring up a dialog like this:
In this dialog you can select the Tier, or speed, with which to restore the objects (please see the AWS web site for more details) and the number of days that the restored copy of the object should be kept accessible in S3 before deletion. When you have selected these parameters, click the Restore button.
After clicking Restore you will see a dialog like the one below, please read it carefully:
If you wish to continue to push the Restore button, otherwise push Cancel or Back. Back will take you back to the previous dialog.
If you push Restore you will see a progress dialog. The progress dialog indicates how many objects StorReduce has started the restore process for and also gives you the option to cancel the process. If the process is cancelled, the objects that have alreaqdy been processed will continue through the restore process the objects that have not yet been processed will remain in Glacier.
When all the objects have been processed, a final dialog telling you how many objects were restored will be displayed.
Monitoring the Restore Process
Once the restore process has been started the object or objects should move into the Restore in progress state. When they are restored and available for download then they will be in the Available until DATE state.
Using the S3 API to Restore Objects
Developers can use the S3 API or AWS CLI to restore objects through StorReduce in the same way that they would with Amazon S3. There are two important opetations:
To check the status of an object, use the HEAD Object API call or the
aws s3api head-objectCLI command and consult the
X-AMZ-Restoreheaders returned in the response as describe in the S3 API documentation.
Notable differences between StorReduce’s S3 API and Amazon’s
When a bucket is listed using the GET Bucket (List
API call and StorReduce is in Glacier mode then all objects will have
StorageClass value of
GLACIER. Use the HEAD
API to find the actual status of each object.