Redis 3.2集群环境搭建和测试

2017-03-06 阅读: Redis

Redis 3开始支持了Cluster模式,增强了Redis的水平扩展能力,Redis Cluster的节点分片通过hash slot实现,每个节点上的键都属于16384(0~16383)个slots中的一个,每个节点负责处理一部分slots。 Redis Cluster采用无中心节点方式实现,无需proxy代理,客户端直接与redis集群的每个节点连接,根据同样的hash算法计算出key对应的slot,然后直接在slot对应的Redis上执行命令。

Redis安装

环境准备

  • CentOS 7.2
    • redis1 192.168.61.51 6379(master) 6380(slave)
    • redis2 192.168.61.52 6379(master) 6380(slave)
    • redis3 192.168.61.53 6379(master) 6380(slave)
  • 修改内核参数vm.overcommit_memory = 1

    vi /etc/sysctl.conf
    vm.overcommit_memory = 1
      
    sysctl -p
    

编译安装Redis

wget http://download.redis.io/releases/redis-3.2.8.tar.gz
tar -zxvf redis-3.2.8.tar.gz
cd redis-3.2.8
make
make install PREFIX=/usr/local/redis

mkdir /usr/local/redis/etc
cp redis.conf /usr/local/redis/etc/

测试启动:

/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis.conf

集群初始化

配置

redis_6379.conf

daemonize yes
pidfile /var/run/redis_6379.pid
dir /var/lib/redis/redis_6379  
logfile /var/log/redis/redis_6379.log
appendonly yes
bind 192.168.61.51
# bind 192.168.61.52
# bind 192.168.61.53
port 6379

cluster-enabled yes
cluster-config-file nodes-6379.conf
cluster-node-timeout 15000
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage yes

redis_6380.conf

daemonize yes
pidfile /var/run/redis_6380.pid
dir /var/lib/redis/redis_6380  
logfile /var/log/redis/redis_6380.log
appendonly yes
bind 192.168.61.51
# bind 192.168.61.52
# bind 192.168.61.53
port 6380

cluster-enabled yes
cluster-config-file nodes-6380.conf
cluster-node-timeout 15000
cluster-slave-validity-factor 10
cluster-migration-barrier 1
cluster-require-full-coverage yes

集群相关的配置:

  • cluster-enabled 是否开启集群配置
  • cluster-config-file 集群节点集群信息配置文件,每个节点都有一个,由redis生成和更新,配置时避免名称冲突
  • cluster-node-timeout 集群节点互连超时的阀值,单位毫秒
  • cluster-slave-validity-factor 进行故障转移时,salve会 申请成为master。有时slave会和master失联很久导致数据较旧,这样的slave不应该成为master。这个配置用来判断slave是否和master失联时间过长。
groupadd redis
useradd -g redis -d /var/lib/redis -s /sbin/nologin redis
mkdir -p /var/log/redis
mkdir -p /var/lib/redis/redis_6379
mkdir -p /var/lib/redis/redis_6380
chown -R redis:redis /var/log/redis /var/lib/redis

redis_6379.service

[Unit]
Description=redis
After=network.target

[Service]
Type=forking
User=redis
ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_6379.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target

redis_6380.service

[Unit]
Description=redis
After=network.target

[Service]
Type=forking
User=redis
ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_6380.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target
systemctl enable redis_6379
systemctl enable redis_6380
systemctl start redis_6379
systemctl start redis_6380

主节点

配置成功后,在各个主机上查看redis进程,各个redis节点都运行在cluster模式下:

 ps -ef | grep redis
redis     2656     1  0 15:09 ?        00:00:00 /usr/local/redis/bin/redis-server 192.168.61.51:6379 [cluster]
redis     2666     1  0 15:09 ?        00:00:00 /usr/local/redis/bin/redis-server 192.168.61.51:6380 [cluster]

使用cluster meet添加主节点:

./redis-cli -h 192.168.61.51 -p 6379 -c

cluster meet 192.168.61.52 6379
cluster meet 192.168.61.53 6379

查看集群状态:

cluster info

cluster_state:fail  #集群状态
cluster_slots_assigned:0 #分配的slot数
cluster_slots_ok:0       #正确的slot数
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3    #当前的节点数
cluster_size:0
cluster_current_epoch:2
cluster_my_epoch:0
cluster_stats_messages_sent:171
cluster_stats_messages_received:171

上面显示cluster的状态是fail,是由于没有分配slot,需要把16384分配到3个Node上,编写addslots脚本:

addslots.sh

#!/bin/bash
for ((i=$2;i<=$3;i++))
do
   /usr/local/redis/bin/redis-cli -h $1 -p 6379 CLUSTER ADDSLOTS $i
done

分别在redis1,redis2,redis3三个主机上执行:

./addslots.sh redis1 0 5461
./addslots.sh redis2 5462 10922
./addslots.sh redis3 10923 16383

再次查看集群状态:

CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:3
cluster_size:3
cluster_current_epoch:2
cluster_my_epoch:0
cluster_stats_messages_sent:3495
cluster_stats_messages_received:3495

从节点

先添加从节点到集群中:

cluster meet 192.168.61.51 6380
cluster meet 192.168.61.52 6380
cluster meet 192.168.61.53 6380
CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6  #所有节点数6个
cluster_size:3  #主节点数3个
cluster_current_epoch:5
cluster_my_epoch:0
cluster_stats_messages_sent:4388
cluster_stats_messages_received:4388

查看各节点id:

CLUSTER NODES
5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 192.168.61.53:6380 master - 0 1488959931731 5 connected
8f60e94a47da98286795b2fbbcae1b3aea75e123 192.168.61.51:6380 master - 0 1488959933751 3 connected
bd503b887831e642ac850af3fdd6e27c27086db7 192.168.61.51:6379 myself,master - 0 0 0 connected 0-5461
956f9f95b96bb02842609cea5dcda0cf89bfb78e 192.168.61.53:6379 master - 0 1488959935772 2 connected 10923-16383
877a1fad6a6c344a69707a335d2dcf94afeecbbd 192.168.61.52:6380 master - 0 1488959934761 4 connected
7290136f19a26c877c31f565068ea02badf11339 192.168.61.52:6379 master - 0 1488959936786 1 connected 5462-10922

下面分配从节点,主从关系如下:

master 192.168.61.51 6379  -> slave 192.168.61.52 6380
master 192.168.61.52 6379  -> slave 192.168.61.53 6380
master 192.168.61.53 6379  -> slave 192.168.61.51 6380
./redis-cli -h 192.168.61.51 -p 6380 -c
CLUSTER REPLICATE 5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68
 

./redis-cli -h 192.168.61.52 -p 6380 -c
CLUSTER REPLICATE bd503b887831e642ac850af3fdd6e27c27086db7

./redis-cli -h 192.168.61.53 -p 6380 -c
CLUSTER REPLICATE 7290136f19a26c877c31f565068ea02badf11339
 CLUSTER NODES
877a1fad6a6c344a69707a335d2dcf94afeecbbd 192.168.61.52:6380 slave bd503b887831e642ac850af3fdd6e27c27086db7 0 1488960443865 4 connected
7290136f19a26c877c31f565068ea02badf11339 192.168.61.52:6379 master - 0 1488960447913 1 connected 5462-10922
956f9f95b96bb02842609cea5dcda0cf89bfb78e 192.168.61.53:6379 master - 0 1488960444893 2 connected 10923-16383
5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 192.168.61.53:6380 myself,slave 7290136f19a26c877c31f565068ea02badf11339 0 0 5 connected
8f60e94a47da98286795b2fbbcae1b3aea75e123 192.168.61.51:6380 slave 5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 0 1488960446907 5 connected
bd503b887831e642ac850af3fdd6e27c27086db7 192.168.61.51:6379 master - 0 1488960445904 0 connected 0-5461

到这一步Redis 3的高可用和分配集群搭建完毕。

基于ruby的集群管理工具

安装ruby

安装ruby:

wget https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.0.tar.gz
tar -zxvf ruby-2.4.0.tar.gz
cd ruby-2.4.0
./configure prefix=/usr/local/ruby
make
make install

echo 'export PATH="$PATH:/usr/local/ruby/bin"' >> /etc/profile
source /etc/profile

ruby -v
ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-linux]

安装rubygems:

wget https://rubygems.org/rubygems/rubygems-2.6.10.tgz
tar -zxvf rubygems-2.6.10.tgz
cd rubygems-2.6.10
ruby setup.rb

gem -v
2.6.10

安装zlib:

wget http://www.zlib.net/zlib-1.2.11.tar.gz
tar -zxvf zlib-1.2.11.tar.gz
cd zlib-1.2.11
./configure --prefix=/usr/local/zlib
make
make install

安装 ruby-zlib:

cd ruby-2.4.0/ext/zlib
ruby ./extconf.rb --with-zlib-dir=/usr/local/zlib
make
make install

上面的命令在make时报make: *** No rule to make target /include/ruby.h', needed by zlib.o'. Stop.。修改Makefile文件:

ORIG_SRCS = /root/source/ruby-2.4.0/ext/zlib/zlib.c
SRCS = $(ORIG_SRCS)
OBJS = /root/source/ruby-2.4.0/ext/zlib/zlib.o

安装redis gem:

gem sources --add http://rubygems.org
gem sources --remove https://rubygems.org/
gem sources list

gem q -rn redis
gem install redis

redis源码下的redis-trib.rb即为集群管理工具:

./redis-trib.rb
Usage: redis-trib <command> <options> <arguments ...>

  create          host1:port1 ... hostN:portN
                  --replicas <arg>
  check           host:port
  info            host:port
  fix             host:port
                  --timeout <arg>
  reshard         host:port
                  --from <arg>
                  --to <arg>
                  --slots <arg>
                  --yes
                  --timeout <arg>
                  --pipeline <arg>
  rebalance       host:port
                  --weight <arg>
                  --auto-weights
                  --use-empty-masters
                  --timeout <arg>
                  --simulate
                  --pipeline <arg>
                  --threshold <arg>
  add-node        new_host:new_port existing_host:existing_port
                  --slave
                  --master-id <arg>
  del-node        host:port node_id
  set-timeout     host:port milliseconds
  call            host:port command arg arg .. arg
  import          host:port
                  --from <arg>
                  --copy
                  --replace
  help            (show this help)

参考

标题:Redis 3.2集群环境搭建和测试
本文链接:https://blog.frognew.com/2017/03/redis-3.2-cluster-install-and-test.html
转载请注明出处。

目录