StorReduce Monitor Edition Deployment Guide

Introduction

The StorReduce Monitor appliance is an optional component of a StorReduce scale-out cluster that provides centralized log management and statistics aggregation. While the StorReduce Monitor appliance is not an essential component of a scale-out cluster we highly recommend it is used as it greatly simplifies management tasks.

Functionality

Log aggregation

The log aggregation function leverages the Elastic ELK (Elasticsearch - Logstash - Kibana) stack.

StorReduce writes logs into a local directrory (/var/log/storreduce by default) on each server in the cluster. An elastic Filebeat daemon forwards the lines of the logs to a Logstash server on the Monitor server. Logstash transforms each log line and inserts it into an Elasticsearch instance. A Kibana server provides a dashboard for viewing and searching the logs.

All components of the log aggregation function (Logstash, Elasticsearch and Kibana) are run inside Docker containers within the Monitor appliance. Filebeat runs inside a Docker container on each StorReduce server.

Statistics aggregation

Statistics aggregation is powered by Prometheus, a timeseries database, and Grafana, an analytics and monitoring system.

A Prometheus server running on the Monitor server periodically pulls statistics from each StorReduce server in the cluster and stores them in its database.

A Grafana server, also running on the Monitor server, provides a dashboard for viewing the statistics.

Prometheus and Grafana also run in Docker within the Monitor appliance.

Deployment

The StorReduce Monitor Appliance is currently available for use on Amazon EC2 or on-premises via a VMware OVA file. Please contact us to request a free trial for your preferred platform.

Prerequisites

You should have previously deployed and configured a StorReduce scale-out cluster using the StorReduce Cluster Edition.

Firewall setup

If the deployment has a firewall, please configure it to allow TCP connections on the following ports:

Port Service Who/What will connect Source Destination
22 SSH Administrators
3000 Grafana (HTTP) Administrators
5601 Kibana (HTTP) Administrators
5044 Logstash Servers in the cluster StorReduce Server(s) Monitor Server
9090 Prometheus Servers in the cluster StorReduce Server(s) Monitor Server
8096 Prometheus Servers in the cluster Monitor Server StorReduce Server(s)
2379 etcd Servers in the cluster Monitor Server StorReduce Server(s)
Default setup

All components inside the StorReduce Monitor appliance are pre-configured to store persistent data (including databases etc.) on a single large disk mounted at /mnt/storage. This data should be backed up periodically, although it is not considered critical data and the StorReduce cluster will continue to function without it.

Amazon EC2

Please use the following guide to deploy a StorReduce Monitor Appliance on Amazon EC2:

StorReduce Monitor Appliance Amazon EC2 Deployment Guide

After completing the guide please return here to continue the configuration steps below.

VMware OVA

Please use the following guide to deploy a StorReduce Monitor Appliance on VMware using an OVA file:

StorReduce Monitor Appliance VMware Deployment Guide

After completing the guide please return here to continue the configuration steps below.

Configuration

After deploying a StorReduce Monitor appliance using the instructions above it must be configured to connect to the StorReduce cluster.

Configure the Monitor Server

SSH to the Monitor server and run the following command:

sudo storreduce-monitor --initial_cluster_discovery_token="<CLUSTERTOKEN>"

Where <CLUSTERTOKEN> is the cluster discovery token found in the Cluster tab of the StorReduce dashboard (shown below).

Cluster Discovery Token Screenshot

This command allows the Monitor server to connect to the etcd server that runs inside of the StorReduce cluster, which is then used to discover all StorReduce servers in the cluster. This list of servers is used by Prometheus to periodically scrape statistics from each server.

Configure StorReduce Servers

Please note the following steps must be performed on every StorReduce server in the cluster.

SSH to each StorReduce server and run the following commands:

sudo yum install storreduce-monitor
sudo storreduce-filebeat install "<MONITOR_IP>:5044"
sudo storreducectl server flags set stats_server_address "<MONITOR_IP>:9090"
sudo storreducectl server restart

Where <MONITOR_IP> is the IP address (or hostname) of the Monitor server. Please note it is important to include the ports when running these commands.

Using the Monitor Appliance

Using Grafana

Grafana allows you to view near real-time performance metrics of the StorReduce cluster.

Browse to http://<MONITOR_IP>:3000 (where <MONITOR_IP> is the IP address (or hostname) of the Monitor server) and log in using the default credentials of user admin and password admin. Please make sure to change the password via the Admin -> Users screen.

After changing the password clicked the Dashboards dropdown at the top of the screen and select the StorReduce dashboard.

Grafana Screenshot

Using Kibana

Kibana provides a dashboard for viewing and searching the aggregated logs from every server in the StorReduce cluster.

Browse to http://<MONITOR_IP>:5601 (where <MONITOR_IP> is the IP address (or hostname) of the Monitor server) and log in with default user elastic and password changeme.

On the first configuration screen that loads confirm the pattern of filebeat-* and then browse to the discover tab. From here you can begin to search logs.

Kibana Index Configuration Screenshot

Kibana Discover Screenshot