Redis-Sentinel cluster on CentOS 7

Disclaimer

I have tried left right and the center for a good source to explain this from head to toe and failed to find any, so decided to put up this walk-through, share the whys and hows and perhaps it comes handy for anyone. hope it takes off some burden and useful.

  • Most of the examples are run from the compiled binary redis/redis-sentinel while I wanted a properly installed service by the repositories and controlled by systemd.
  • I needed some sort of protection at the firewall level, later explain how I achieved it.
  • I needed all the kernel parameters incorporated ,either by manipulating the sysconfig or yet in later stages grub-hooks. In case of the latter I had to restrain to self-composed systemd script which Ill provide here along the reason.
  • Almost always examples are run with several instances on the same node without a real separated role groups. A matter that needs extra-fine granulation and production disaster avoidance. A great source to explain and eye-opening is comments from #The_real_bill in here.

Environment

The environment I run in is CentOS 7 with firewalld and selinux enabled. The packages at the time of installation is grabbed from epel repository. The redis servers are configured with 100GB RAM and sentinels are really tiny with 1 core + 1G RAM. All the settings are done in vanilla installation and a word of caution, if you messed up the premission and port not able to bound to or the service stopped starting, please do reinstall a fresh instance as I learned it the hard way and the time I spent was too dear.

The diagram

Configuration

 

Record the IPs and if needed nmtui the static settings. Get both redis server packages installed by:

# yum install redis

Add the necessary firewall rules:

# firewall-cmd –new-zone=redis –permanent

# firewall-cmd –zone=redis –add-source={x.x.x.x/32,x.x.x.x/32,…} –permanent

# firewall-cmd –zone=redis –add-service=redis –permanent

# firewall-cmd –add-port=26379/tcp –permanent –zone=redis

# firewall-cmd –reload

# firewall-cmd –list-all –zone=redis

Check with the above if all the rules are set and OK-Look for the redis entry in the services list and ports and IPs in the relative parts.

Now time for the config changes for redis

On redis1 server, edit the below:

bind 127.0.0.1 10.0.0.53

#protected-mode yes <–comment out this to allow communication

On redis2 server:

bind 127.0.0.1 10.0.0.53

#protected-mode yes <–comment out this to allow communication

slaveof 10.0.0.53 6379

On both servers start the service and check the replication status and also test if you see the replication taking effects:

# systemctl start redis && systemctl status redis

# redis-cli info replication && redis-cli set foo bar && redis-cli get foo

Now if the replication info is what you expect and master and slave roles are already there appearing, you are done with the Master-Slave redis config. Time to jump to the redis-sentinel configuration.

Minimal configuration for the redis-sentinel based on the above diagram IP info can be set as:

bind 10.0.0.74             <- this is the tricky bit that costs me a  lot of debugging even in case of sentinel and redis on the same server do not put the loopback interface here
port 26379             <- that is the default and we keep it that way
sentinel monitor mymaster 10.0.0.53 6379 2           <- some clarification mymaster is the default cluster naming …the tricky bit is that we do not provide loopback ip here even if that is the sentinel on the master node…I learned this the hard way and after a long debug
sentinel down-after-milliseconds mymaster 6000                <-announcement agreement for the master re-election after 6 seconds
sentinel failover-timeout mymaster 6000      <- in a nutshell it will fail over after only 6 seconds,since the servers are on the same LAN 6 seconds pretty fine
supervised systemd               <- change this to the systemd

Now there are some other lines added to the above file after starting the redis-sentinel like the id number and the node view-point of itself along the rest of the nodes and also the redis nodes roles i.e who is master whois slave and whois master and consensus of the redis-sentinel nodes.

Here is an example after redis-sentinel service started on all nodes and convergence done:

bind 10.0.0.74
port 26379
dir “/tmp”
sentinel myid 801d529ccc1ded9762c096faff3d66ea5ee1ce04
sentinel monitor mymaster 10.0.0.53 6379 2
sentinel down-after-milliseconds mymaster 6000
sentinel failover-timeout mymaster 6000
logfile “/var/log/redis/sentinel.log”
supervised systemd
sentinel config-epoch mymaster 2
sentinel leader-epoch mymaster 2
sentinel known-slave mymaster 10.0.0.54 6379
sentinel known-sentinel mymaster 10.0.0.76 26379 b2466fa5d687182e96decca1076f2d2d4ae7d781
sentinel known-sentinel mymaster 10.0.0.75 26379 c61da1e439edca0b67cb6c177ebd07b0fa16b9ac
sentinel current-epoch 2

The last bits and tricks:

Install the tuned package and change the server profile so that they are throughput intensive load responsive if on VM environment. This is imperative as the benchmarking of the servers proves many folds improvement.

# yum install tuned

# tuned-adm profile throughput-performance

The above commands essentially re-edit and fine-tune performance of the server. check redhat document for further details

Create a systemd service and enable it to disable hugepagefile issue that affects the performance of redis:

# vim /etc/systemd/system/hpf_disable.service

And the content reads:

[Unit]
Description=Disable Transparent Huge Pages (THP)

[Service]
Type=simple
ExecStart=/bin/sh -c “echo ‘never’ > /sys/kernel/mm/transparent_hugepage/enabled && echo ‘never’ > /sys/kernel/mm/transparent_hugepage/defrag”

[Install]
WantedBy=multi-user.target

Enable and start the service:

systemctl enable hpf_disable && systemctl start hpf_disable

Last word: essentially there are two kernel parameters in case you did not go the tuned package settings which are :

vm.swappiness = 10
vm.overcommit_memory = 1

The HAproxy minimal config for the backend redis as a sample is:

backend BE_redis
mode tcp
option tcp-check
option tcpka
tcp-check connect
tcp-check send PING\r\n
tcp-check expect string +PONG
tcp-check send info\ replication\r\n
tcp-check expect string role:master
tcp-check send QUIT\r\n
tcp-check expect string +OK
fullconn 30000
server red0 10.0.0.53:6379 weight 1 check inter 3000 rise 3 fall 3
server red110.0.0.54:6379 weight 1 check inter 3000 rise 3 fall 3 backup

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.