Go
by smallnest

Redka - 父亲是Redis，母亲是SQLite

Redka旨在使用SQLite重新实现Redis的优秀部分,同时保持与Redis API的兼容性。

有意思的特性:

  • 数据不必完全装载在内存中。
  • 支持ACID事务。
  • 使用SQL视图以便于内省和报告。
  • 同时提供进程内(Go API)和独立(RESP)服务器。
  • 兼容Redis的命令和线路协议。

Redia 并不期望完全达到 Redis 那么高的性能，但是性能上也并不会落后很多。它的主要功能还是 SQL + Redis 的功能，集成了两种优秀产品 SQLite 和 Redis 的盛世美颜。

支持的 Redis 命令

字符串

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Command      Go API                 Description
-------      ------                 -----------
DECR         DB.Str().Incr          Decrements the integer value of a key by one.
DECRBY       DB.Str().Incr          Decrements a number from the integer value of a key.
GET          DB.Str().Get           Returns the value of a key.
GETSET       DB.Str().SetWith       Sets the key to a new value and returns the prev value.
INCR         DB.Str().Incr          Increments the integer value of a key by one.
INCRBY       DB.Str().Incr          Increments the integer value of a key by a number.
INCRBYFLOAT  DB.Str().IncrFloat     Increments the float value of a key by a number.
MGET         DB.Str().GetMany       Returns the values of one or more keys.
MSET         DB.Str().SetMany       Sets the values of one or more keys.
PSETEX       DB.Str().SetExpires    Sets the value and expiration time (in ms) of a key.
SET          DB.Str().Set           Sets the value of a key.
SETEX        DB.Str().SetExpires    Sets the value and expiration (in sec) time of a key.
SETNX        DB.Str().SetWith       Sets the value of a key when the key doesn't exist.

暂时不支持的命令

1
APPEND  GETDEL  GETEX  GETRANGE  LCS  MSETNX  SETRANGE  STRLEN  SUBSTR

列表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Command      Go API                      Description
-------      ------                      -----------
LINDEX       DB.List().Get               Returns an element by its index.
LINSERT      DB.List().Insert*           Inserts an element before or after another element.
LLEN         DB.List().Len               Returns the length of a list.
LPOP         DB.List().PopFront          Returns the first element after removing it.
LPUSH        DB.List().PushFront         Prepends an element to a list.
LRANGE       DB.List().Range             Returns a range of elements.
LREM         DB.List().Delete*           Removes elements from a list.
LSET         DB.List().Set               Sets the value of an element by its index.
LTRIM        DB.List().Trim              Removes elements from both ends a list.
RPOP         DB.List().PopBack           Returns the last element after removing it.
RPOPLPUSH    DB.List().PopBackPushFront  Removes the last element and pushes it to another list.
RPUSH        DB.List().PushBack          Appends an element to a list.

暂时不支持的命令

1
2
BLMOVE  BLMPOP  BLPOP  BRPOP  BRPOPLPUSH  LMOVE  LMPOP
LPOS  LPUSHX  RPUSHX

Set

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Command      Go API                 Description
-------      ------                 -----------
SADD         DB.Set().Add           Adds one or more members to a set.
SCARD        DB.Set().Len           Returns the number of members in a set.
SDIFF        DB.Set().Diff          Returns the difference of multiple sets.
SDIFFSTORE   DB.Set().DiffStore     Stores the difference of multiple sets.
SINTER       DB.Set().Inter         Returns the intersection of multiple sets.
SINTERSTORE  DB.Set().InterStore    Stores the intersection of multiple sets.
SISMEMBER    DB.Set().Exists        Determines whether a member belongs to a set.
SMEMBERS     DB.Set().Items         Returns all members of a set.
SMOVE        DB.Set().Move          Moves a member from one set to another.
SPOP         DB.Set().Pop           Returns a random member after removing it.
SRANDMEMBER  DB.Set().Random        Returns a random member from a set.
SREM         DB.Set().Delete        Removes one or more members from a set.
SSCAN        DB.Set().Scanner       Iterates over members of a set.
SUNION       DB.Set().Union         Returns the union of multiple sets.
SUNIONSTORE  DB.Set().UnionStore    Stores the union of multiple sets.

暂时不支持的命令

1
SINTERCARD  SMISMEMBER

Hash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Command       Go API                  Description
-------       ------------------      -----------
HDEL          DB.Hash().Delete        Deletes one or more fields and their values.
HEXISTS       DB.Hash().Exists        Determines whether a field exists.
HGET          DB.Hash().Get           Returns the value of a field.
HGETALL       DB.Hash().Items         Returns all fields and values.
HINCRBY       DB.Hash().Incr          Increments the integer value of a field.
HINCRBYFLOAT  DB.Hash().IncrFloat     Increments the float value of a field.
HKEYS         DB.Hash().Keys          Returns all fields.
HLEN          DB.Hash().Len           Returns the number of fields.
HMGET         DB.Hash().GetMany       Returns the values of multiple fields.
HMSET         DB.Hash().SetMany       Sets the values of multiple fields.
HSCAN         DB.Hash().Scanner       Iterates over fields and values.
HSET          DB.Hash().SetMany       Sets the values of one or more fields.
HSETNX        DB.Hash().SetNotExists  Sets the value of a field when it doesn't exist.
HVALS         DB.Hash().Exists        Returns all values.

暂时不支持的命令

1
HRANDFIELD  HSTRLEN

Sorted Set

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Command           Go API                  Description
-------           ------                  -----------
ZADD              DB.ZSet().AddMany       Adds or updates one or more members of a set.
ZCARD             DB.ZSet().Len           Returns the number of members in a set.
ZCOUNT            DB.ZSet().Count         Returns the number of members of a set within a range of scores.
ZINCRBY           DB.ZSet().Incr          Increments the score of a member in a set.
ZINTER            DB.ZSet().InterWith     Returns the intersection of multiple sets.
ZINTERSTORE       DB.ZSet().InterWith     Stores the intersection of multiple sets in a key.
ZRANGE            DB.ZSet().RangeWith     Returns members of a set within a range of indexes.
ZRANGEBYSCORE     DB.ZSet().RangeWith     Returns members of a set within a range of scores.
ZRANK             DB.ZSet().GetRank       Returns the index of a member in a set ordered by ascending scores.
ZREM              DB.ZSet().Delete        Removes one or more members from a set.
ZREMRANGEBYRANK   DB.ZSet().DeleteWith    Removes members of a set within a range of indexes.
ZREMRANGEBYSCORE  DB.ZSet().DeleteWith    Removes members of a set within a range of scores.
ZREVRANGE         DB.ZSet().RangeWith     Returns members of a set within a range of indexes in reverse order.
ZREVRANGEBYSCORE  DB.ZSet().RangeWith     Returns members of a set within a range of scores in reverse order.
ZREVRANK          DB.ZSet().GetRankRev    Returns the index of a member in a set ordered by descending scores.
ZSCAN             DB.ZSet().Scan          Iterates over members and scores of a set.
ZSCORE            DB.ZSet().GetScore      Returns the score of a member in a set.
ZUNION            DB.ZSet().UnionWith     Returns the union of multiple sets.
ZUNIONSTORE       DB.ZSet().UnionWith     Stores the union of multiple sets in a key.

不支持的命令

1
2
3
BZMPOP  BZPOPMAX  BZPOPMIN  ZDIFF  ZDIFFSTORE  ZINTERCARD
ZLEXCOUNT  ZMPOP  ZMSCORE  ZPOPMAX  ZPOPMIN  ZRANDMEMBER
ZRANGEBYLEX  ZRANGESTORE  ZREMRANGEBYLEX  ZREVRANGEBYLEX

Key

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
Command    Go API                    Description
-------    ------                    -----------
DBSIZE     DB.Key().Len              Returns the total number of keys.
DEL        DB.Key().Delete           Deletes one or more keys.
EXISTS     DB.Key().Count            Determines whether one or more keys exist.
EXPIRE     DB.Key().Expire           Sets the expiration time of a key (in seconds).
EXPIREAT   DB.Key().ExpireAt         Sets the expiration time of a key to a Unix timestamp.
FLUSHDB    DB.Key().DeleteAll        Deletes all keys from the database.
KEYS       DB.Key().Keys             Returns all key names that match a pattern.
PERSIST    DB.Key().Persist          Removes the expiration time of a key.
PEXPIRE    DB.Key().Expire           Sets the expiration time of a key in ms.
PEXPIREAT  DB.Key().ExpireAt         Sets the expiration time of a key to a Unix ms timestamp.
RANDOMKEY  DB.Key().Random           Returns a random key name from the database.
RENAME     DB.Key().Rename           Renames a key and overwrites the destination.
RENAMENX   DB.Key().RenameNotExists  Renames a key only when the target key name doesn't exist.
SCAN       DB.Key().Scanner          Iterates over the key names in the database.
TTL        DB.Key().Get              Returns the expiration time in seconds of a key.
TYPE       DB.Key().Get              Returns the type of value stored at a key.

不支持的命令

1
2
3
COPY  DUMP  EXPIRETIME  MIGRATE  MOVE  OBJECT  PEXPIRETIME
PTTL  RESTORE  SORT  SORT_RO  TOUCH  TTL  TYPE  UNLINK
WAIT  WAITAOF

事务

1
2
3
4
5
Command    Go API                 Description
-------    ------                 -----------
DISCARD    DB.View / DB.Update    Discards a transaction.
EXEC       DB.View / DB.Update    Executes all commands in a transaction.
MULTI      DB.View / DB.Update    Starts a transaction.

与 Redis 不同，Redka 的事务完全符合 ACID 属性，在发生故障时会自动回滚。

不支持的命令

1
UNWATCH  WATCH

连接管理

1
2
3
4
Command    Go API                Description
-------    ------                -----------
ECHO       -                     Returns the given string.
PING       -                     Returns the server's liveliness response.

安装

你可以以独立的服务的形式安装。官方文档提供了方便的 Linux、macOS 的安装脚本，你也可以使用 docker 进行部署。

也可以在你的 Go 程序中当成一个库引用: go get github.com/nalgeon/redka，当然了你也必须引入 sqlite 的库，比如 github.com/mattn/go-sqlite3 或者 modernc.org/sqlite。

运行

你可以使用 redka [-h host] [-p port] [db-path] 启动一个 redka 服务，比如

1
2
3
./redka
./redka data.db
./redka -h 0.0.0.0 -p 6379 data.db

如果没有指定数据库文件，那么它就完全使用内存的方式。

如果当成一个库引入使用，你可以像这样在进程内使用:\

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
package main
import (
    "log"
    _ "github.com/mattn/go-sqlite3"
    "github.com/nalgeon/redka"
)
func main() {
    // Open or create the data.db file.
    db, err := redka.Open("data.db", nil)
    if err != nil {
        log.Fatal(err)
    }
    // Always close the database when you are finished.
    defer db.Close()
    // ...
}

如果是使用内存的方式，可以指定 sqlite 使用内存的方式打开:

1
2
// All data is lost when the database is closed.
redka.Open("file:redka?mode=memory&cache=shared")

然后代码中就可以直接调用：

1
2
3
4
5
6
7
8
db.Str().Set("name", "alice")
db.Str().Set("age", 25)
count, err := db.Key().Count("name", "age", "city")
slog.Info("count", "count", count, "err", err)
name, err := db.Str().Get("name")
slog.Info("get", "name", name, "err", err)

下面的代码是使用事务的例子：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
updCount := 0
err := db.Update(func(tx *redka.Tx) error {
    err := tx.Str().Set("name", "bob")
    if err != nil {
        return err
    }
    updCount++
    err = tx.Str().Set("age", 50)
    if err != nil {
        return err
    }
    updCount++
    return nil
})
slog.Info("updated", "count", updCount, "err", err)

性能

作者在 Apple M1 8-core CPU, 16GB RAM 的苹果机器和 Redis 进行了性能比较。
使用下面的场景进行比较(redis-benchmark -p 6379 -q -c 10 -n 1000000 -r 10000 -t get,set)：

  • 10 parallel connections
  • 1000000 requests
  • 10000 randomized keys
  • GET/SET commands

比 Redis 慢 2~5 倍，这也合理，毕竟底层采用一个关系数据库进行存储。

给我们的启发

Redka 使用 redcon 进行 Redis 命令的解析，然后针对每种大类的 Redis 命令，会有一个单独的表进行存储。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
rkey
---
id       integer primary key
key      text not null
type     integer not null    -- 1 string, 2 list, 3 set, 4 hash, 5 sorted set
version  integer not null    -- incremented when the key value is updated
etime    integer             -- expiration timestamp in unix milliseconds
mtime    integer not null    -- modification timestamp in unix milliseconds
len      integer             -- number of child elements
rstring
---
kid      integer not null    -- FK -> rkey.id
value    blob not null
rlist
---
kid      integer not null    -- FK -> rkey.id
pos      real not null       -- is used for ordering, but is not an index
elem     blob not null
rset
---
kid      integer not null    -- FK -> rkey.id
elem     blob not null
rhash
---
kid      integer not null    -- FK -> rkey.id
field    text not null
value    blob not null
rzset
---
kid      integer not null    -- FK -> rkey.id
elem     blob not null
score    real not null

既然它支持 sqlite, 我们可以把它 port 到其它的数据库上，比如 clickhouse，这样我们就可以支持巨量的数据了。

当然更深一步，我们还可以接入其他的 NoSQL 数据库，比如 RocksDB 等，类似的项目都有很多了。

这是一个很有意思的项目。