Scale-out Monitor Server Deployment Guide

Introduction

This document outlines the requirements and instructions for setting up the Monitor Server for a StorReduce Cluster.

The Monitor Server is an optional part of a StorReduce Cluster. As its name suggests it is used to monitor a StorReduce Cluster, but unlike other systems it is not necessary for controlling the cluster nor is it an essential part of the cluster.

The functions performed by the monitor server are:

  1. Log aggregation. It aggregates the logs from all servers in the cluster, providing a central location for viewing and searching logs.

  2. Statistics aggregation. It aggregates statistics from all the cluster and displays them in a dashboard. Statistics are also displayed on the StorReduce dashboard.

Log aggregation

The log aggregation function leverages the Elastic ELK (Elasticsearch - Logstash - Kibana) stack.

Logs are written into a local directrory (/var/log/storreduce by default) on each server in the cluster. An elastic filebeat daemon forwards the lines of the logs to a logstash server on the monitor server. logstash transforms each log line and inserts it into an elasticsearch instance. A kibana server provides a dashboard for viewing and searching the logs.

In a typical installation filebeat, logstash, elasticsearch and kibana all run in Docker containers with corresponding names.

Statistics aggregation

Statistics aggregation is powered by Prometheus, a timeseries database, and Grafana, an analytics and monitoring system.

A prometheus server running on the Monitor Server periodically pulls statistics from each StorReduce server in the cluster and stores them in its database.

A grafana server, also running on the Monitor Server, provides a dashboard for viewing the statistics.

Typically prometheus and grafana also run in Docker containers with corresponding names.

Prerequisites

The prerequisites for the Monitor Server are:

  • A Linux Server (physical or VM), Running RedHat Enterprise Linux 7+, CentOS 7+, Amazon Linux or similar.
  • Docker installed and running on the Linux Server
  • At least 8GB RAM
  • At least 40GB of disk space

as well as a functioning StorReduce cluster.

Installation

Firewall configuration

If the system has a firewall, please configure it to allow TCP connections to the following ports:

Port Service Who/What will connect
22 SSH Administrators
80 Grafana Administrators
81 Kibana Administrators
5044 Logstash Servers in the cluster
9090 Prometheus Servers in the cluster

Install ntpd if it is not already installed

(If you are using the StorReduce VM, please skip to this section).

Determine if ntpd is installed.

sudo systemctl status ntpd 

if using a systemd based distribution or:

sudo server ntpd status

on an older distribution.

If NTP is not installed, please install it.

Install the StorReduce YUM Repository

(If you are using the StorReduce VM, please skip to this section).

To install the StorReduce YUM repository:

sudo curl --output /etc/yum.repos.d/storreduce.repo \
  https://storreduce-yum.s3-us-west-2.amazonaws.com/storreduce.repo

Install the Monitor package

On the Monitor Server please run:

sudo yum clean all && sudo yum -y install storreduce-monitor

These commands install the storreduce-monitor RPM which contains scripts and configuration for the Monitor Server.

Then run:

storreduce-install-monitor

This command will pull the appropriate docker containers from hub.docker.com and docker.elastic.co and connect them using a host only network. It also configures Elasticsearch to store logs from Filebeat.

Finally, run:

sudo storreduce-monitor --initial_cluster_discovery_token=<CLUSTERTOKEN>

where <CLUSTERTOKEN> is the cluster discovery token found in the Cluster tab of the StorReduce dashboard (shown below).

Cluster Discovery Token Screenshot

The dashboard can be found at https://<SERVER-IP-ADDRESS>:8080/, where <SERVER-IP-ADDRESS> is the IP address of any of the servers in the cluster.

The storreduce-monitor command fetches the IP addresses of the servers in the cluster and then configures Prometheus so that it can retrieve statistics from them. If you add more servers to the cluster, please run the command again to update Prometheus.

Install Filebeat on each server in the cluster

On the monitor server, take a copy of the filebeat configuration file template:

cp /usr/share/storreduce/filebeat/filebeat.yml.template /tmp/filebeat.yml

Edit the file and replace HOSTNAME with the hostname (or IP address) of the monitor server.

Copy filebeat.yml and /usr/share/storreduce/filebeat/storreduce-install-filebeat to each server in the cluster.

scp /tmp/filebeat.yml /usr/share/storreduce/filebeat/storreduce-install-filebeat <SERVER>:/tmp

where <SERVER> is the IP address or hostname of each server.

On each server in the cluster run:

sh /tmp/storreduce-install-filebeat /tmp/filebeat.yml

This script will pull, configure and run the Filebeat docker container.

Tell StorReduce about the monitor VM

Run these commands on every server in the cluster:

sudo storreducectl server flags set stats_server_address <HOSTNAME>:9090
sudo storreducectl server restart

Where <HOSTNAME> is the hostname (or IP address) of the monitor server.

Configure the Kibana dashboard

Browse to port 81 on the Monitor Server to view the Kibana dashboard. From here all StorReduce logs can be searched.

Login with user elastic, password changeme.

On the first configuration screen that loads confirm the default pattern of filebeat-* and then browse to the discover tab. From here you can begin to search logs.

Configure the Grafana dashboard

Browse to port 80 on the monitor server to view the Grafana dashboard. From here, StorReduce server health and metrics can be monitored across the cluster.

Login using the default credentials of user admin, password admin and make sure to first change the password via the Admin -> Users screen.

Once the default password has been changed we must tell Grafana about Prometheus.

Browse to Data Sources -> Add Data Source and enter the following information:

Field Value
Name StorReduce
Type Prometheus
Url http://prometheus:9090/metrics/proxy
Access Proxy

Note that proxy should be selected, and the host name should always be “prometheus”.

Grafana Data Source Settings

Then click Save & Test.

Next, we must import the standard StorReduce dashboard. Browse to Dashboards -> Import, click Upload .json file. A sample dashboard can be found in /usr/share/storreduce/grafana/grafana-dashboard.json on the monitor server.

Select the StorReduce as the data source.

You should then be able to browse to Dashboards -> StorReduce and see statistics about the monitor server.