Redis基礎必備

redis 全稱：
remote dictionary server

存儲系統：

RDBMS關係型數據庫: MYSQL / SQLSERVER

NOSQL非關係型數據庫: REDIS/MEMCACHED/MongoDB/HBase

NEWSQL: NewSQL是對所有新型可擴展、高性能數據庫的簡稱NewSQL能夠結合傳統關係型數據庫和NoSQL的優勢，且容易橫向擴展

redis與memcached的區別 

如果簡單地比較Redis與Memcached的區別，大多數都會得到以下觀點：
1 Redis不僅僅支持簡單的k/v（key、value）類型的數據，同時還提供list，set，hash等數據結構的存儲。
2 Redis支持數據的備份，即master-slave模式的數據備份，redis通過SENTINEL哨兵來進行主從的切換

redis在3.0 開始支持redis cluster （集群），但是可能並不完善

3 Redis支持數據的持久化，可以將內存中的數據保持在磁盤中，重啟的時候可以再次加載進行使用。
在Redis中，並不是所有的數據都一直存儲在內存中的。這是和Memcached相比一個最大的區別（我個人是這麼認為的）

另外：redis單線程，memcached多線程（可以使用多核cpu）

 

Redis組件：

1. redis-server redis的服務端

2. redis-cli redis的客戶端

command line interface 

3. redis-benchmark（壓力測試工具）

4. redis-check-dump & redis-check-aof（檢查redi持久化工具） 

corrupted RDB/AOF file utilities 

redis 持久化：

Redis只會緩存所有的key的信息，如果Redis發現內存的使用量超過了某一個閥值，將觸發swap的操作，Redis根據“swappability = age*log(size_in_memory)”計算出哪些key對應的value需要swap到磁盤。然後再將這些key對應的value持久化到磁盤中，同時在內存中清除。這種特性使得Redis可以保持超過其機器本身內存大小的數據。當然，機器本身的內存必須要能夠保持所有的key，畢竟這些數據是不會進行swap操作的。

同時由於Redis將內存中的數據swap到磁盤中的時候，提供服務的主線程和進行swap操作的子線程會共享這部分內存，所以如果更新需要swap的數據，Redis將阻塞這個操作，直到子線程完成swap操作後才可以進行修改。

 

Redis 併發相關介紹：

1 Million small key → string value pairs use ~100MB of memory

Single threaded – but cpu should not to be the bottleneck

Average linux system can deliver even 500K request persecond（50w併發）

如果較好的硬件經過測試支持百萬的併發是完全沒有問題的

Redis持久化的兩種方式(Redis Persistence):

參考官網：

https://redis.io/topics/persistence

 

1. Snapshotting快照RDB

 

The RDB persistence performs point-in-time snapshots of your dataset at specified intervals.

RDB持久性執行時間點快照的數據集在指定的時間間隔。二進制格式顯示為dump.rdb

2.AOF(append only file )

the AOF persistence logs every write operation received by the server

If you wish, you can disable persistence at all, if you want your data to just exist as long as the server is running.

 

The general indication is that you should use both persistence methods if you want a degree of data safety comparable to what PostgreSQL can provide you.

·一般的跡象是,您應該同時使用這兩種持久性方法如果你想一定程度的數據安全與PostgreSQL能給你的一樣

If you care a lot about your data, but still can live with a few minutes of data loss in case of disasters, you can simply use RDB alone.

·如果你非常關心你的數據,但仍然可以忍受幾分鐘的數據丟失在災害的情況下,您可以簡單地使用RDB孤單。

There are many users using AOF alone, but we discourage it since to have an RDB snapshot from time to time is a great idea for doing database backups, for faster restarts, and in the event of bugs in the AOF engine.

有許多用戶使用AOF獨自,但是我們阻止它從一個RDB快照時間進行數據庫備份是一個偉大的想法,更快的重新啟動,在發生錯誤的AOF引擎

 

在redis服務器啟動用於恢復數據時，會優先使用aof

Redis安裝：

 

 

Make test 的一個報錯：

[err]: Test replication partial resync: ok psync (diskless: yes, reconnect: 1) in tests/integration/replication-psync.tcl

Expected condition '[s -1 sync_partial_ok] > 0' to be true ([s -1 sync_partial_ok] > 0)

 

解決方式：

http://www.voidcn.com/blog/chenggong2dm/article/p-6097574.html

 

修改redis配置文件

 

Redis 啟動命令：

 

 

 

Redis-cli 命令行工具使用說明：

 

 

 

 

 

redis清空數據庫操作：

Redis對事務的支持：

事務：一組相關的操作是原子性的，要麼都執行，要麼都不執行

一組事務，要麼成功；要麼撤回。

Mysql 事務簡介：http://www.runoob.com/mysql/mysql-transaction.html

 

redis通過multi  exec  watch 等命令實現事務功能

開始事務：multi命令開始事務

在事務進行過程中，其他對redis的操作都會被阻塞

Exec：一併返回

 

例子：

 

 

Redis的複製：

一個master可以有多個slave

一個slave還可以再有slave，所以支持鏈式複製

Master 以非阻塞方式同步數據至salve ，master可以同時處理多個slave的操作

 

修改配置文件兩處：

 

 

我們再增加一個從服務器，為後面的sentinel課程打下基礎10.0.140.86

 

REDIS安全簡介

1).禁止一些高危命令

修改 redis.conf 文件，添加

rename-command FLUSHALL ""

rename-command CONFIG   ""

rename-command EVAL     ""

來禁用遠程修改 DB 文件地址

 

2).以低權限運行 Redis 服務

為 Redis 服務創建單獨的用戶和家目錄，並且配置禁止登陸

 

3).為 Redis 添加密碼驗證

修改 redis.conf 文件，添加

requirepass mypassword

 

4).禁止外網訪問 Redis

修改 redis.conf 文件，添加或修改

bind 10.0.140.84

使得 Redis 服務只在當前主機可用

 

 

Redis哨兵sentinel

Redis主從有一個問題，如果我們主服務器掛掉了，slave服務器只讀，這樣就會出現問題

Redis採用了sentinel機制來解決這個問題

 

Redis Sentinel provides high availability for Redis. In practical terms this means that using Sentinel you can create a Redis deployment that resists without human intervention to certain kind of failures. 您可以創建一個複述,部署,抗拒某種失敗而無需人工干預

Redis Sentinel also provides other collateral（並行的） tasks such as monitoring, notifications and acts as a configuration provider for clients.

This is the full list of Sentinel capabilities at a macroscopical level (i.e. the big picture):

· Monitoring. Sentinel constantly(持續的) checks if your master and slave instances(實例) are working as expected.

· Notification. Sentinel can notify the system administrator, another computer programs, via an API, that something is wrong with one of the monitored Redis instances.

· Automatic failover.(自動切換) If a master is not working as expected, Sentinel can start a failover process where a slave is promoted to master, the other additional slaves are reconfigured to use the new master, and the applications using the Redis server informed about the new address to use when connecting.

· Configuration provider. Sentinel acts as a source of authority for clients service discovery: clients connect to Sentinels in order to ask for the address of the current Redis master responsible for a given service. If a failover occurs, Sentinels will report the new address.客戶端配置哨兵的地址

· You only need to specify the masters to monitor, giving to each separated master (that may have any number of slaves) a different name. There is no need to specify slaves, which are auto-discovered. Sentinel will update the configuration automatically with additional information about slaves只需告知哨兵master的地址就可以，無需告訴slave的地址，sentinel會自動發現

· 

· Sentinels stay connected with other Sentinels in order to reciprocally 相互地check the availability of each other, and to exchange messages. However you don't need to configure a list of other Sentinel addresses in every Sentinel instance you run(你不需要在哨兵實例中配置哨兵實例的列表), as Sentinel uses the Redis instances Pub/Sub capabilities 能力in order to discover the other Sentinels that are monitoring the same masters and slaves.

· Sentinels need to be elected leader for the failover and be authorized to proceed. This only happens with the vote of the majority of the Sentinel processes. 所以我們最少可以用1臺sentinel，或者我們可以使用3臺

· # Slaves are auto-discovered, so you don't need to specify slaves in

· # any way. Sentinel itself will rewrite this configuration file adding

· # the slaves using additional configuration options.

· # Also note that the configuration file is rewritten when a

· # slave is promoted to master.

 

Sentinel的配置

Sentinel當前最新的穩定版本稱為Sentinel 2(與之前的Sentinel 1區分開來）。隨著redis2.8的安裝包一起發行。安裝完Redis2.8後，可以在redis2.8/src/裡面找到Redis-sentinel的啟動程序。

配置文件：/app/zpy/redis/conf/sentinel.conf

二進制文件：/app/zpy/redis/bin/redis-sentinel

 

運行sentinel有兩種方式：

第一種：redis-sentinel /path/to/sentinel.conf

第二種：redis-server /path/to/sentinel.conf --sentinel

以上兩種方式，都必須指定一個sentinel的配置文件sentinel.conf，如果不指定，將無法啟動sentinel。

sentinel默認監聽26379端口，所以運行前必須確定該端口沒有被別的進程佔用。

修改sentinel的配置文件：

整理配置文件：

egrep -v "#|^$" sentinel.conf >> a.conf

 

啟動sentinel：

cd /app/zpy/redis/bin/

./redis-sentinel ../conf/sentinel.conf

Ss -tnl  查看26379 端口

./redis-cli  -h 10.0.150.43 -p 26379

Sentinel簡單命令：

sentinel切換實驗：

殺掉redis-master 

再去看上述幾條命令的返回結果

Redis-cluster：

3.0以後開始支持cluster

分佈式數據庫，通過分片機制進行數據分佈，clustering內的每個節點僅持有數據庫的一部分數據

Redis cluster每一個節點都可以作為客戶端的接入節點

每個節點只持有一部分數據，但每個節點都有全局視角

例子， 比如有10000個key，一共5臺redis節點，每臺都可以被客戶端接入，但是每臺可能只存儲2000個key，每臺都有一個類似index索引的東西記錄了這10000個key是如何分佈的

 

我們來看一下官網的解釋：

Redis Cluster provides a way to run a Redis installation where data is automatically sharded across multiple Redis nodes.

Redis Cluster also provides some degree of availability during partitions, that is in practical terms the ability to continue the operations when some nodes fail or are not able to communicate. However the cluster stops to operate in the event of larger failures (for example when the majority of masters are unavailable).

So in practical terms, what you get with Redis Cluster?

· The ability to automatically split your dataset among multiple nodes.

· The ability to continue operations when a subset of the nodes are experiencing failures or are unable to communicate with the rest of the cluster.

官網關於cluster port的說明：

Redis Cluster TCP ports

Every Redis Cluster node requires two TCP connections open. The normal Redis TCP port used to serve clients, for example 6379, plus the port obtained by adding 10000 to the data port, so 16379 in the example.

This second high port is used for the Cluster bus（集群總線）, that is a node-to-node communication channel using a binary protocol. The Cluster bus is used by nodes for failure detection, configuration update, failover authorization and so forth. Clients should never try to communicate with the cluster bus port, but always with the normal Redis command port, however make sure you open both ports in your firewall, otherwise Redis cluster nodes will be not able to communicate.

The command port and cluster bus port offset is fixed and is always 10000.

 

If you don't open both TCP ports, your cluster will not work as expected.

The cluster bus uses a different, binary protocol, for node to node data exchange, which is more suited to exchange information between nodes using little bandwidth and processing time.

 

 

Redis Cluster data sharding（分片）（redis cluster的數據分片機制）

 

Redis Cluster does not use consistent（一致的） hashing, but a different form of sharding（分片） where every key is conceptually (概念性的)part of what we call an hash slot.

翻譯說明：Redis cluster 不會使用常用的哈希算法，而是提出了分片機制，這種分片機制把一個key叫做一個hash slot（哈希槽位）

 

There are 16384 hash slots in Redis Cluster, and to compute what is the hash slot of a given key, we simply take the CRC16 of the key modulo 16384.

Every node in a Redis Cluster is responsible for a subset of the hash slots, so for example you may have a cluster with 3 nodes, where:

翻譯說明：

這裡有16384個hash槽位在redis cluster裡，為了計算出一個key的hash槽位，我們使用CRC16校驗機制

每一個節點都是16384的子集，例如

· Node A contains hash slots from 0 to 5500.

· Node B contains hash slots from 5501 to 11000.

· Node C contains hash slots from 11001 to 16383.

This allows to add and remove nodes in the cluster easily. For example if I want to add a new node D, I need to move some hash slot from nodes A, B, C to D. Similarly（類似的） if I want to remove node A from the cluster I can just move the hash slots served by A to B and C. When the node A will be empty I can remove it from the cluster completely.

翻譯說明：這樣增加或者減少節點就變得容易了，例如如果我想增加一個D節點，我需要從node A B C 裡面移除一部分hash slot到D，類似的，如果我想移除A從集群中，我只需要把A的hash slot 移動到B C .當A空了的時候，我就可以把它從集群移除了

 

Because moving hash slots from a node to another does not require to stop operations, adding and removing nodes, or changing the percentage of hash slots hold by nodes, does not require any downtime.

翻譯說明：

因為從一個節點移除hash slot到另一個節點並不需要停機操作，增加或者移除節點和改變某一個節點擁有hash slot的比例都不需要停機

 

 

Redis Cluster master-slave model

In order to remain available when a subset of master nodes are failing or are not able to communicate with the majority of nodes, Redis Cluster uses a master-slave model where every hash slot has from 1 (the master itself) to N replicas (N-1 additional slaves nodes).

翻譯說明：為了保證少數redis節點宕機或者不能與多數節點通訊時整個cluster還可以正常使用，redis-cluster使用了master-slave模式，每一個hash slot 具有N個副本，n-1個 slave node

 

In our example cluster with nodes A, B, C, if node B fails the cluster is not able to continue, since we no longer have a way to serve hash slots in the range 5501-11000.

However when the cluster is created (or at a latter time) we add a slave node to every master, so that the final cluster is composed of A, B, C that are masters nodes, and A1, B1, C1 that are slaves nodes, the system is able to continue if node B fails.

Node B1 replicates B, and B fails, the cluster will promote node B1 as the new master and will continue to operate correctly.

However note that if nodes B and B1 fail at the same time Redis Cluster is not able to continue to operate.

翻譯說明：

在我們上面的實驗裡如果NODEB故障了，cluster就不能繼續工作了，因為5501-1000這個範圍的hash-slot就丟失了

但是當創建集群的時候我們為每個master-node添加一個slave，A B C是master node ，A1 B1 C1 是slave node ，如果B 掛掉了，仍然沒有問題

Node B1 複製B ，如果B掛掉了，系統會提升B1為新的master

但是如果B和B1都掛掉了，那就不行了

 

Redis Cluster consistency guarantees

Redis集群一致性的保證

 

Redis Cluster is not able to guarantee strong consistency（一致性）.

In practical terms ，this means that under certain conditions it is possible that Redis Cluster will lose writes

that were acknowledged（承認） by the system to the client.

The first reason why Redis Cluster can lose writes is because it uses asynchronous（異步） replication.

This means that during writes the following happens:

Your client writes to the master B.

The master B replies（回覆） OK to your client.

The master B propagates （傳播）the write to its slaves B1, B2 and B3.

As you can see B does not wait for an acknowledge （確認）from B1, B2, B3 before replying to the client,

since this would be a prohibitive（禁止） latency（潛在） penalty（處罰） for Redis, so if your client writes something, B acknowledges the write,

but crashes（崩潰了） before being able to send the write to its slaves, one of the slaves (that did not receive the write) can be promoted to master, losing the write forever.

This is very similar to what happe

總結：這個是無法避免的，並且只會有很小概率發生

 

 

 

redis-cluster選舉:容錯

(1)領著選舉過程是集群中所有master參與,如果半數以上master節點與故障節點通信超過(cluster-node-timeout),認為該節點故障，自動觸發故障轉移操作.

(2):什麼時候整個集群不可用(cluster_state:fail)? 

    a:如果集群任意master掛掉,且當前master沒有slave.集群進入fail狀態,也可以理解成集群的slot映射[0-16383]不完成時進入fail狀態. ps : redis-3.0.0.rc1加入cluster-require-full-coverage參數,默認關閉,打開集群兼容部分失敗.

    b:如果集群超過半數以上master掛掉，無論是否有slave集群進入fail狀態.

  ps:當集群不可用時,所有對集群的操作做都不可用，收到((error) CLUSTERDOWN The cluster is down)錯誤

 

Redis Cluster configuration parameters

Cp redis.conf  redis-cluster.conf

cluster-enabled yes    集群開關，默認是不開啟集群模式。

cluster-config-file nodes-6379.conf  集群配置文件的名稱，每個節點都有一個集群相關的配置文件，持久化保存集群的信息。這個文件並不需要手動配置，這個配置文件有 Redis生成並更新，每個Redis集群節點需要一個單獨的配置文件，請確保與實例運行的系統中配置文件名稱不衝突。

cluster-node-timeout 15000 節點互連超時的閥值。集群節點超時毫秒數。即節點與集群其他節點斷開多長時間將被認定為超時。建議稍微大一點

cluster-slave-validity-factor 10 在進行故障轉移的時候，全部slave都會請求申請為master，但是有些slave可能與master斷開連接一段時間了，導致數據過於陳舊，這樣的slave不應該被提升為master。該參數就是用來判斷slave節點與master斷線的時間是否過長。判斷方法是：比較slave斷開連接的時間和(node-timeout * slave-validity-factor)+ repl-ping-slave-period如果節點超時時間為三十秒, 並且slave-validity-factor為10,假設默認的repl-ping-slave-period是10秒，即如果超過310秒slave將不會嘗試進行故障轉移

cluster-migration-barrier 1 master的slave數量大於該值，slave才能遷移到其他孤立master上，如這個參數若被設為2，那麼只有當一個主節點擁有2個可工作的從節點時，它的一個從節點才會嘗試遷移。

cluster-require-full-coverage yes 默認情況下，集群全部的slot有節點負責，集群狀態才為ok，才能提供服務。設置為no，可以在slot沒有全部分配的時候提供服務。不建議打開該配置，這樣會造成分區的時候，小分區的master一直在接受寫請求，而造成很長時間數據不一致。

 

 

 

Yum install ruby  → 啟動集群的時候需要ruby

yum install rubygems 

gem source -l 查看當前ruby源

gem sources --remove https://rubygems.org/

gem sources --remove http://rubygems.org/

gem sources -a https://ruby.taobao.org/

gem install redis --version 3.2.0

 

redis-trib 位於 Redis 源碼的 src 文件夾中， 它是一個 Ruby 程序， 這個程序通過向實例發送特殊命令來完成創建新集群， 檢查集群， 或者對集群進行重新分片（reshared）等工作。這裡通過create命令來創建集群，指定replicas=1，即每一個主實例有一個從實例。redis-trib 會打印出一份預想中的配置給你看， 如果你覺得沒問題的話， 就可以輸入 yes ， redis-trib 就會將這份配置應用到集群當中,讓各個節點開始互相通訊

 

啟動redis-cluster

./redis-server ../conf/redis-cluster.conf

root      4379     1  0 15:51 ?        00:00:00 ./redis-server 10.0.140.84:6379 [cluster]

 

./redis-trib.rb create --replicas 1 10.0.140.84:6379 10.0.140.85:6379 10.0.140.86:6379 10.0.140.87:6379 10.0.140.88:6379 10.0.140.89:6379

 

10.0.140.84:6379> set dailiang 333

-> Redirected to slot [15929] located at 10.0.140.87:6379

OK

10.0.140.87:6379> get dailiang

"333"

10.0.140.87:6379>

 

為節點分配slot：redis-trib.rb reshard 192.168.72.100:7006