StorReduce Cluster Edition Deployment Guide

Introduction

The StorReduce Cluster Edition is designed for large scale-out deployments of StorReduce (up to 100s of PBs) that supports 10s of GB/s of throughput with high availability.

This guide will step through setting up a multi-server cluster using the StorReduce Cluster Edition.

Cluster Sizing

Depending on hardware spec and deduplication ratio, a single StorReduce server is capable of writing data at up to 2GB/s and recovering data at up to 1GB/s.

A StorReduce cluster is comprised of an odd number of equally specced StorReduce servers, N_SERVERS, which must be greater than or equal to 3.

To size a cluster, first identify the required throughput and logical data volume and work backwards. Throughput is dictated by available CPU cores, disk IOPs and network bandwidth, and volume is dictated by the available amount of SSD storage for the deduplication index.

CPU Requirements

To work out the number of required servers:

<desired_throughput> / <throughput_per_core> / <cores_per_server> = N_SERVERS

Use the table below to approximate the <throughput_per_core> variable:

Dedupe Percent Throughput per core*
98% 68 MB/s
95% 60 MB/s
90% 47 MB/s
80% 33 MB/s
50% 16 MB/s
*Tested on an Intel Xeon E5-2686 v4 @ 2.30GHz

For example, to achieve 5GB/s of write throughput at a 90% dedupe rate using servers with 36 cores each:

5000MB / 47MB / 36 = 2.9 (rounded up to 3)

SSD Requirements

StorReduce requires approximately 0.1% of the total logical data volume as SSD index storage. In a cluster this storage is divided amongst all servers.

To work out the amount of SSD per server:

<desired_logical_data_volume> * 0.001 / N_SERVERS = SSD_PER_SERVER

For example, to store 10PB of logical data (data before dedupe) using the 3 server cluster calculated in the previous step:

10000TB * 0.001 / 3 = 3.4TB

Network Requirements

There are three distinct network segments in a StorReduce cluster - client, management, and object storage. Client network traffic represents the data being uploaded by client software before dedupe (e.g. NetBackup), management traffic is the internal communication and distribution of work between servers, and object storage traffic represents the data being written to the object store after dedupe.

Depending on the size of the cluster and required throughput these segments may all run over the same physical network interfaces, or they may each run on dedicated interfaces. For high-performance clusters we recommend the use of dedicated interfaces.

Client network bandwidth

The client network bandwidth handles data before dedupe (e.g. from NetBackup), therefore is must be capable of handling the overall desired throughput. This traffic should be load balanced amongst all servers in the cluster.

To work out the required client network bandwidth per server:

<desired_throughput> / N_SERVERS = CLIENT_BANDIWDTH_PER_SERVER

For example, using the 3 server cluster calculated in the previous steps:

5GB / 3 = 1.67 GB/s

To achieve 5GB/s this example would require 2x 10GbE interfaces per server.

Management network bandwidth

Unlike the client network, the management network transmits data after deduplication, therefore it needs much less bandwidth.

To work out the required management network bandwidth per server:

(<desired_throughput> / <estimated_dedupe_ratio> / N_SERVERS) * N_SERVERS-1 = MANAGEMENT_BANDIWDTH_PER_SERVER

For example, using the 3 server cluster calculated in the previous steps (90% dedupe is a 10:1 ratio):

(5GB / 10 / 3) * 2 = 0.33 GB/s

This example would require a single 10GbE interface to support 333 MB/s of management traffic per server. In this scenario there is enough bandwidth remaining on the 2x 10GbE client interfaces above to use for management traffic.

Object Storage network bandwidth

Similar to the management network, the object storage network transmits data after deduplication.

To work out the required object storage network bandwidth per server:

<desired_throughput> / <estimated_dedupe_ratio> / N_SERVERS = OBJECT_STORAGE_BANDIWDTH_PER_SERVER

For example, using the 3 server cluster calculated in the previous steps (90% dedupe is a 10:1 ratio):

5GB / 10 / 3 = 0.17 GB/s

This example would require a single 10GbE interface to support 170 MB/s of object storage traffic per server. In this scenario there is enough bandwidth remaining on the 2x 10GbE client interfaces above to use for both management traffic and object storage traffic.

Namespace partitioning and High Availability

Once the cluster has been sized the next consideration is how to distribute data amongst the servers in a highly available way.

Inside each StorReduce server there are a number of shards, N_SHARDS. Each shard represents a logical partition of the namespace - if a server contains a shard then that server is responsible for that part of the namespace.

When the number of servers in a cluster is increased the cluster rebalances the shards across the cluster so that each server is responsible for approximately the same number of shards. The number of shards N_SHARDS is fixed when the cluster is created; it cannot be changed after the cluster is created.

Before you create a cluster you must decide on the number shards that the cluster will have. StorReduce reccommends that in most cases the number of shards should be 4 times the number of servers in the cluster. E.g.,

N_SHARDS = 4 * N_SERVERS

For example, if N_SERVERS = 3 we recommend N_SHARDS = 12; if N_SERVERS = 9 we reccommend N_SHARDS = 36.

Each shard is contained in a primary server. Additionally, a shard can also be replicated to one or more secondary servers so that if the primary server fails, one of the secondary servers can take over that portion of the namespace and the cluster can continue to operate. The number of shard replicas is N_REPLICAS.

A StorReduce cluster will continue to function if up to N_REPLICAS servers fail and the number of operational servers is greater than or equal to N_SERVERS divided by 2 and rounded up to the nearest integer.

The constraint on the size of N_REPLICAS is the amount of local SSD that can be allocated to each server. Every replica increases the amount of SSD required for the cluster. E.g., a cluster with N_REPLICAS = 1 requires twice as much SSD as a cluster with N_REPLICAS = 0.

Increasing N_REPLICAS also increases the amount CPU, network bandwidth and IOPS consumed by the cluster.

Therefore, if N_SERVERS is 3 then N_REPLICAS can be 1 or 0. If N_SERVERS greater than 3 then N_REPLICAS should in most cases be 0, 1 or 2.

Firewall Configuration

By default StorReduce server uses the following TCP ports.

Port(s) Reason Who/What will connect
22 SSH Administrators
80 S3 API over HTTP Administrators & users
443 S3 API over HTTPS Administrators & users
8080 StorReduce adminstration dashboard over HTTPS Administrators & users
8095-8098 Intra-cluster communications Servers in the cluster
2379-2380 Intra-cluster communications Servers in the cluster

Deployment

Once the cluster has been sized and N_SERVERS, N_SHARDS and N_REPLICAS have been determined the servers can be deployed.

Prerequisites

Before deploying StorReduce Cluster Edition you will require the following:

  • A static IP address for each of the StorReduce server appliances.

  • A DNS name configured to DNS Round Robin between each of the static IP addresses assigned to the servers, or a load balancer which sits infront of the cluster.

  • A bucket configured in a supported Object Store either on-premises or on-cloud

  • A set of access keys that are used to authenticate to the bucket

  • License key information for your server which should have been provided by a StorReduce representative when issuing your free trial.

  • Details for the optional Amazon KMS or a KMIP-compatible hardware appliance for data encryption should be provisioned.

Amazon EC2

After requesting a free trial please follow the deployment guide for deploying on Amazon EC2:

StorReduce Cluster Edition - Amazon EC2 Deployment Guide

After completing the guide please return here to continue the configuration steps below.

VMware OVA

After requesting a free trial please follow the deployment guide for deploying on VMware using an OVA file:

StorReduce Cluster Edition - On-Premises VMware Deployment Guide

After completing the guide please return here to continue the configuration steps below.

Bare metal or Custom OS

After requesting a free trial please follow the deployment guide for deploying on deploying a StorReduce in a bare metal or custom OS environment without using one of our standard virtual appliances:

StorReduce Cluster Edition - Custom Deployment Guide

After completing the guide please return here to continue the configuration steps below.

Configuration

Configure the first server

StorReduce clusters are masterless, no server has any special role or is any different from any other server. However, when creating a new cluster one server must be fully configured first, and then additional servers are added to the cluster to expand it.

  1. On the first server in the cluster:

    sudo storreducectl server init
    
  2. Follow the wizard to allocate ports and other configuration settings for StorReduce. If you change the ports from the defaults, please update the firewall accordingly. We recommend using all default ports. The output of the wizard should look like this:

    Enter the IP address that other StorReduce servers will use to connect to this server (--cluster_listen_interface): 172.31.0.197
    Enter the beginning of the port range that other servers will connect to this server on. The port range will cover the specifed port plus the next 3 ports (--cluster_listen_port) [8095]:
    Enter the ETCD client port that other servers will connect to this server on (--config_server_client_port) [2379]:
    Enter the ETCD peer port that other servers will connect to this server on (--config_server_peer_port) [2380]:
    Enter the S3 HTTP port that S3 clients will connect to this server on (--http_port) [80]:
    Enter the S3 HTTPS port that S3 clients will connect to this server on (--https_port) [443]:
    Enter the Admin API HTTPS port that StorReduce admins will connect to this server on (--admin_port) [8080]:
    Please enter the number of shards. This CANNOT be changed later (--dev_n_shards) [36]:
    Please enter the number of shard replicas. This CAN be changed later (--n_shard_replicas) [2]: 1
    
    StorReduce will be configured with the following settings:
    
    ### Cluster ###
    
    Shard Count (--dev_n_shards): 36
    Shard Replica Count (--n_shard_replicas): 1
    
    ### Network ###
    
    IP Address (--cluster_listen_interface): 172.31.0.197
    Cluster Port #1 (--cluster_listen_port): 8095
    Cluster Port #2: 8096
    Cluster Port #3: 8097
    Cluster Port #4: 8098
    ETCD Client Port (--config_server_client_port): 2379
    ETCD Peer Port (--config_server_peer_port): 2380
    HTTP Port (--http_port): 80
    HTTPS Port (--https_port): 443
    Admin API HTTPS Port (--admin_port): 8080
    
    ### Host Directories ###
    
    SRDB: /mnt/srdb
    SRKEYDB: /mnt/srkeydb
    Lib: /var/lib/storreduce
    Run: /var/run
    Etc: /etc/storreduce
    Log: /var/log/storreduce
    
    WARNING Please confirm you wish to initialize StorReduce
    Please type "init storreduce" and push enter to proceed: init storreduce
    
  3. Browse to the first server on port 8080. For example:

    https://172.31.0.197:8080

  4. You will receive a warning about the server’s certificate being invalid. Initially the StorReduce server will use a self-signed certificate for the ‘localhost’ domain, which can be replaced later. Continue on past the browser warnings.

    SSL Warning Screenshot

  5. Log in: You should see the Login dialog for StorReduce. Log in with the User Name of root, with the initial password suitable for You can to change the root password for StorReduce after the server is configured.

    StorReduce Login Screenshot

  6. On the initial screen, select ‘Configure StorReduce’. You will be taken to the Initial Settings screen.

    StorReduce Landing Screenshot

  7. In the License Options section please enter the license information obtained when you requested your free trial.

    StorReduce License Screenshot

  8. Under the Storage section, configure your backend bucket for your chosen object store. The screenshot below shows an Amazon S3 bucket being configured:

    StorReduce Bucket Screenshot

  9. Optionally configure Amazon KMS or KMIP-based data encryption.

  10. Under the Network section enter the DNS name for the server in the Hostname field:

    StorReduce Hostname Screenshot

  11. All required settings have now been entered. Click on the Save Settings & Restart Server button at the bottom of the page. The page should automatically reload after 10 seconds (it can sometimes take a little longer).

    StorReduce Save Settings Screenshot

    StorReduce Restart Screenshot

  12. After the first server has restarted login and browse to the Cluster screen. Copy the Cluster Discovery Token that appears (referred in subsequent steps as <CLUSTERTOKEN>)

    StorReduce Cluster Token Screenshot


Add additional servers to the cluster
  1. On all the other servers that have not yet been configured, run:

    sudo storreducectl server init <CLUSTERTOKEN>
    
  2. Follow the wizard to allocate ports and other configuration settings for StorReduce. If you change the ports from the defaults, please update the firewall accordingly.

Rebalance the cluster

After all servers have been added to the cluster, click the Rebalance Cluster button on the Cluster tab of the StorReduce Dashbaord, or SSH to any server and run:

sudo storreducectl cluster rebalance

Wait 2 minutes and the cluster should be fully functional.

Next Steps

After deploying a StorReduce scale-out cluster the StorReduce Monitor edition should also be deployed to provide centralized monitoring a log aggreation for the cluster.

See the StorReduce Monitor Edition deployment guide for more information.

Grafana Screenshot