Redis 3.2集群环境搭建和测试

2017-03-06

Redis 3开始支持了Cluster模式，增强了Redis的水平扩展能力，Redis Cluster的节点分片通过hash slot实现，每个节点上的键都属于16384(0~16383)个slots中的一个，每个节点负责处理一部分slots。 Redis Cluster采用无中心节点方式实现，无需proxy代理，客户端直接与redis集群的每个节点连接，根据同样的hash算法计算出key对应的slot，然后直接在slot对应的Redis上执行命令。

Redis安装 #

环境准备 #

CentOS 7.2
- redis1 192.168.61.51 6379(master) 6380(slave)
- redis2 192.168.61.52 6379(master) 6380(slave)
- redis3 192.168.61.53 6379(master) 6380(slave)

修改内核参数vm.overcommit_memory = 1

1vi /etc/sysctl.conf
2vm.overcommit_memory = 1
3
4sysctl -p

编译安装Redis #

1wget http://download.redis.io/releases/redis-3.2.8.tar.gz
2tar -zxvf redis-3.2.8.tar.gz
3cd redis-3.2.8
4make
5make install PREFIX=/usr/local/redis
6
7mkdir /usr/local/redis/etc
8cp redis.conf /usr/local/redis/etc/

测试启动：

1/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis.conf

集群初始化 #

配置 #

redis_6379.conf

 1daemonize yes
 2pidfile /var/run/redis_6379.pid
 3dir /var/lib/redis/redis_6379  
 4logfile /var/log/redis/redis_6379.log
 5appendonly yes
 6bind 192.168.61.51
 7# bind 192.168.61.52
 8# bind 192.168.61.53
 9port 6379
10
11cluster-enabled yes
12cluster-config-file nodes-6379.conf
13cluster-node-timeout 15000
14cluster-slave-validity-factor 10
15cluster-migration-barrier 1
16cluster-require-full-coverage yes

redis_6380.conf

 1daemonize yes
 2pidfile /var/run/redis_6380.pid
 3dir /var/lib/redis/redis_6380  
 4logfile /var/log/redis/redis_6380.log
 5appendonly yes
 6bind 192.168.61.51
 7# bind 192.168.61.52
 8# bind 192.168.61.53
 9port 6380
10
11cluster-enabled yes
12cluster-config-file nodes-6380.conf
13cluster-node-timeout 15000
14cluster-slave-validity-factor 10
15cluster-migration-barrier 1
16cluster-require-full-coverage yes

集群相关的配置：

cluster-enabled 是否开启集群配置
cluster-config-file 集群节点集群信息配置文件,每个节点都有一个,由redis生成和更新,配置时避免名称冲突
cluster-node-timeout 集群节点互连超时的阀值，单位毫秒
cluster-slave-validity-factor 进行故障转移时,salve会申请成为master。有时slave会和master失联很久导致数据较旧，这样的slave不应该成为master。这个配置用来判断slave是否和master失联时间过长。

1groupadd redis
2useradd -g redis -d /var/lib/redis -s /sbin/nologin redis
3mkdir -p /var/log/redis
4mkdir -p /var/lib/redis/redis_6379
5mkdir -p /var/lib/redis/redis_6380
6chown -R redis:redis /var/log/redis /var/lib/redis

redis_6379.service

 1[Unit]
 2Description=redis
 3After=network.target
 4
 5[Service]
 6Type=forking
 7User=redis
 8ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_6379.conf
 9Restart=on-failure
10
11[Install]
12WantedBy=multi-user.target

redis_6380.service

 1[Unit]
 2Description=redis
 3After=network.target
 4
 5[Service]
 6Type=forking
 7User=redis
 8ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_6380.conf
 9Restart=on-failure
10
11[Install]
12WantedBy=multi-user.target

1systemctl enable redis_6379
2systemctl enable redis_6380
3systemctl start redis_6379
4systemctl start redis_6380

主节点 #

配置成功后，在各个主机上查看redis进程，各个redis节点都运行在cluster模式下：

1 ps -ef | grep redis
2redis     2656     1  0 15:09 ?        00:00:00 /usr/local/redis/bin/redis-server 192.168.61.51:6379 [cluster]
3redis     2666     1  0 15:09 ?        00:00:00 /usr/local/redis/bin/redis-server 192.168.61.51:6380 [cluster]

使用cluster meet添加主节点:

1./redis-cli -h 192.168.61.51 -p 6379 -c
2
3cluster meet 192.168.61.52 6379
4cluster meet 192.168.61.53 6379

查看集群状态：

 1cluster info
 2
 3cluster_state:fail  #集群状态
 4cluster_slots_assigned:0 #分配的slot数
 5cluster_slots_ok:0       #正确的slot数
 6cluster_slots_pfail:0
 7cluster_slots_fail:0
 8cluster_known_nodes:3    #当前的节点数
 9cluster_size:0
10cluster_current_epoch:2
11cluster_my_epoch:0
12cluster_stats_messages_sent:171
13cluster_stats_messages_received:171

上面显示cluster的状态是fail，是由于没有分配slot，需要把16384分配到3个Node上，编写addslots脚本：

addslots.sh

1#!/bin/bash
2for ((i=$2;i<=$3;i++))
3do
4   /usr/local/redis/bin/redis-cli -h $1 -p 6379 CLUSTER ADDSLOTS $i
5done

分别在redis1,redis2,redis3三个主机上执行:

1./addslots.sh redis1 0 5461
2./addslots.sh redis2 5462 10922
3./addslots.sh redis3 10923 16383

再次查看集群状态：

 1CLUSTER INFO
 2cluster_state:ok
 3cluster_slots_assigned:16384
 4cluster_slots_ok:16384
 5cluster_slots_pfail:0
 6cluster_slots_fail:0
 7cluster_known_nodes:3
 8cluster_size:3
 9cluster_current_epoch:2
10cluster_my_epoch:0
11cluster_stats_messages_sent:3495
12cluster_stats_messages_received:3495

从节点 #

先添加从节点到集群中：

1cluster meet 192.168.61.51 6380
2cluster meet 192.168.61.52 6380
3cluster meet 192.168.61.53 6380

 1CLUSTER INFO
 2cluster_state:ok
 3cluster_slots_assigned:16384
 4cluster_slots_ok:16384
 5cluster_slots_pfail:0
 6cluster_slots_fail:0
 7cluster_known_nodes:6  #所有节点数6个
 8cluster_size:3  #主节点数3个
 9cluster_current_epoch:5
10cluster_my_epoch:0
11cluster_stats_messages_sent:4388
12cluster_stats_messages_received:4388

查看各节点id:

1CLUSTER NODES
25daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 192.168.61.53:6380 master - 0 1488959931731 5 connected
38f60e94a47da98286795b2fbbcae1b3aea75e123 192.168.61.51:6380 master - 0 1488959933751 3 connected
4bd503b887831e642ac850af3fdd6e27c27086db7 192.168.61.51:6379 myself,master - 0 0 0 connected 0-5461
5956f9f95b96bb02842609cea5dcda0cf89bfb78e 192.168.61.53:6379 master - 0 1488959935772 2 connected 10923-16383
6877a1fad6a6c344a69707a335d2dcf94afeecbbd 192.168.61.52:6380 master - 0 1488959934761 4 connected
77290136f19a26c877c31f565068ea02badf11339 192.168.61.52:6379 master - 0 1488959936786 1 connected 5462-10922

下面分配从节点，主从关系如下：

1master 192.168.61.51 6379  -> slave 192.168.61.52 6380
2master 192.168.61.52 6379  -> slave 192.168.61.53 6380
3master 192.168.61.53 6379  -> slave 192.168.61.51 6380

1./redis-cli -h 192.168.61.51 -p 6380 -c
2CLUSTER REPLICATE 5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68
3 
4
5./redis-cli -h 192.168.61.52 -p 6380 -c
6CLUSTER REPLICATE bd503b887831e642ac850af3fdd6e27c27086db7
7
8./redis-cli -h 192.168.61.53 -p 6380 -c
9CLUSTER REPLICATE 7290136f19a26c877c31f565068ea02badf11339

1 CLUSTER NODES
2877a1fad6a6c344a69707a335d2dcf94afeecbbd 192.168.61.52:6380 slave bd503b887831e642ac850af3fdd6e27c27086db7 0 1488960443865 4 connected
37290136f19a26c877c31f565068ea02badf11339 192.168.61.52:6379 master - 0 1488960447913 1 connected 5462-10922
4956f9f95b96bb02842609cea5dcda0cf89bfb78e 192.168.61.53:6379 master - 0 1488960444893 2 connected 10923-16383
55daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 192.168.61.53:6380 myself,slave 7290136f19a26c877c31f565068ea02badf11339 0 0 5 connected
68f60e94a47da98286795b2fbbcae1b3aea75e123 192.168.61.51:6380 slave 5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 0 1488960446907 5 connected
7bd503b887831e642ac850af3fdd6e27c27086db7 192.168.61.51:6379 master - 0 1488960445904 0 connected 0-5461

到这一步Redis 3的高可用和分配集群搭建完毕。

基于ruby的集群管理工具 #

安装ruby #

安装ruby：

 1wget https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.0.tar.gz
 2tar -zxvf ruby-2.4.0.tar.gz
 3cd ruby-2.4.0
 4./configure prefix=/usr/local/ruby
 5make
 6make install
 7
 8echo 'export PATH="$PATH:/usr/local/ruby/bin"' >> /etc/profile
 9source /etc/profile
10
11ruby -v
12ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-linux]

安装rubygems:

1wget https://rubygems.org/rubygems/rubygems-2.6.10.tgz
2tar -zxvf rubygems-2.6.10.tgz
3cd rubygems-2.6.10
4ruby setup.rb
5
6gem -v
72.6.10

安装zlib：

1wget http://www.zlib.net/zlib-1.2.11.tar.gz
2tar -zxvf zlib-1.2.11.tar.gz
3cd zlib-1.2.11
4./configure --prefix=/usr/local/zlib
5make
6make install

安装 ruby-zlib:

1cd ruby-2.4.0/ext/zlib
2ruby ./extconf.rb --with-zlib-dir=/usr/local/zlib
3make
4make install

上面的命令在make时报make: *** No rule to make target /include/ruby.h', needed by zlib.o'. Stop.。修改Makefile文件：

1ORIG_SRCS = /root/source/ruby-2.4.0/ext/zlib/zlib.c
2SRCS = $(ORIG_SRCS)
3OBJS = /root/source/ruby-2.4.0/ext/zlib/zlib.o

安装redis gem：

1gem sources --add http://rubygems.org
2gem sources --remove https://rubygems.org/
3gem sources list
4
5gem q -rn redis
6gem install redis

redis源码下的redis-trib.rb即为集群管理工具：

 1./redis-trib.rb
 2Usage: redis-trib <command> <options> <arguments ...>
 3
 4  create          host1:port1 ... hostN:portN
 5                  --replicas <arg>
 6  check           host:port
 7  info            host:port
 8  fix             host:port
 9                  --timeout <arg>
10  reshard         host:port
11                  --from <arg>
12                  --to <arg>
13                  --slots <arg>
14                  --yes
15                  --timeout <arg>
16                  --pipeline <arg>
17  rebalance       host:port
18                  --weight <arg>
19                  --auto-weights
20                  --use-empty-masters
21                  --timeout <arg>
22                  --simulate
23                  --pipeline <arg>
24                  --threshold <arg>
25  add-node        new_host:new_port existing_host:existing_port
26                  --slave
27                  --master-id <arg>
28  del-node        host:port node_id
29  set-timeout     host:port milliseconds
30  call            host:port command arg arg .. arg
31  import          host:port
32                  --from <arg>
33                  --copy
34                  --replace
35  help            (show this help)