环境说明
| 服务器 | IP地址 | 服务 |
|---|---|---|
| 监控服务器 | 10.10.xxx.56 | prometheus/grafana |
| k8s-vip | 10.10.xxx.100 | redis集群 |
将redis_exporter部署在监控服务器上，对各环境redis进行监控。此处以监控k8s平台的redis集群为例。
1. redis_exporter
- 创建redis_exporter目录，下边创建docker-compose.yml文件如下：
version: '2'
services:
redis_exporter:
image: harbocto.xxx.com.cn/public/redis_exporter
container_name: redis_exporter
expose:
- "9121"
ports:
- "9121:9121"
restart: always
command: ["--redis.addr","redis://10.10.xxx.100:30020","--redis.password","xxxxxx","redis://10.10.xxx.100:30022","--redis.password","xxxxxx","redis://10.10.xxx.100:30024","--redis.password","xxxxxx"]
【附】如果你要在k8s上启动，注意k8s和docker-compose中 command的对镜像中ENTRYPOINT的覆盖方式是不同的，k8s需要如下写：
command: ["/redis_exporter"]
args: ["--redis.addr","redis://10.10.xxx.100:30020","--redis.password","xxxxxx","redis://10.10.xxx.100:30022","--redis.password","xxxxxx","redis://10.10.xxx.100:30024","--redis.password","xxxxxx"]
- 启动
# docker-compose up -d
[root@monitor redis_exporter]# docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------
redis_exporter /redis_exporter --redis.ad ... Up 0.0.0.0:9121->9121/tcp
-
查看 redis_exporter数据
如下图可见，exporter收集到了数据。
image.png
2. 配置prometheus
- 修改 prometheus.yml 文件，添加如下内容：
########################################
# redis #
########################################
- job_name: 'redis_exporter_targets'
static_configs:
- targets:
- redis://10.10.xxx.100:30020
- redis://10.10.xxx.100:30022
- redis://10.10.xxx.100:30024
metrics_path: /scrape
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 10.10.xxx.56:9121
## config for scraping the exporter itself
- job_name: 'redis_exporter'
static_configs:
- targets:
- 10.10.xxx.56:9121
重启prometheus
-
查看
image.png
3. n9e配置
3.1 导入指标释义
-
入口
image.png
导入指标释义文本如下：
我自己写的，有错误请指正
参考文档：redis官网相关文档
redis_active_defrag_running:活动碎片整理是否运行[lw]
redis_allocator_active_bytes:分配器活动字节[lw]
redis_active_allocated_bytes:活动分配的字节[lw]
redis_assocator_frag_bytes:关联碎片字节[lw]
redis_allocator_frag_ratio:分配器碎片比率[lw]
redis_allocator_resident_bytes:分配器常驻字节[lw]
redis_allocator_rss_bytes:分配器RSS字节[lw]
redis_allocator_rss_ratio:分配器RSS比率[lw]
redis_aof_current_rewrite_duration_sec:aof当前重写持续时间sec[lw]
redis_aof_enabled:是否启用aof[lw]
redis_aof_last_bgrewrite_status:最近一次AOF重写操作是否执行成功[lw]
redis_aof_last_cow_size_bytes:在执行AOF重写期间，分配给COW的大小[lw]
redis_aof_last_rewrite_duration_sec:最近一次AOF重写操作消耗的时间[lw]
redis_aof_last_write_status:aof上次写入状态[lw]
redis_aof_rewrite_in_progress:是否在进行AOF的重写操作[lw]
redis_aof_rewrite_scheduled:是否有AOF操作等待执行[lw]
redis_blocked_clients:被阻止的客户[lw]
redis_client_recent_max_input_buffer_bytes:客户端最近最大输入缓冲区字节[lw]
redis_client_recent_max_output_buffer_bytes:客户端最近最大输出缓冲区字节[lw]
redis_cluster_enabled:是否启用集群[lw]
redis_commands_duration_seconds_total:命令持续时间总秒数[lw]
redis_commands_processed_total:命令处理总数[lw]
redis_commands_total:命令总数[lw]
redis_config_maxclients:配置最大客户端[lw]
redis_config_maxmemory:配置最大内存[lw]
redis_connected_clients:连接的客户[lw]
redis_connected_slave_lag_seconds:连接的从节点延迟秒[lw]
redis_connected_slave_offset_bytes:连接的从节点偏移字节[lw]
redis_connected_slaves:连接的从节点[lw]
redis_connections_received_total:收到的连接总数[lw]
redis_cpu_sys_children_seconds_total:由后台进程消耗的系统CPU[lw]
redis_cpu_sys_seconds_total:由Redis服务器消耗的用户CPU[lw]
redis_cpu_user_children_seconds_total:由后台进程消耗的用户CPU[lw]
redis_cpu_user_seconds_total:由Redis服务消耗的用户CPU[lw]
redis_db_avg_ttl_seconds:db平均ttl秒[lw]
redis_db_keys:数据库key的数量[lw]
redis_db_keys_expiring:即将过期的key[lw]
redis_defrag_hits:碎片整理命中[lw]
redis_defrag_key_hits:碎片整理命中key[lw]
redis_defrag_key_misses:碎片整理未命中key[lw]
redis_evicted_keys_total:被驱逐的key总数[lw]
redis_expired_keys_total:过期key总数[lw]
redis_expired_stale_percentage:过期陈旧key占百分比[lw]
redis_expired_time_cap_reached_total:已达到总时间上限[lw]
redis_exporter_build_infor:redis_exporter信息[lw]
redis_exporter_last_scrape_connect_time_seconds:redis_exporter最后一次采集时间[lw]
redis_exporter_last_scrape_duration_seconds:redis_exporter次抓取持续时间秒[lw]
redis_exporter_last_scrape_error:redis_exporter次抓取错误[lw]
redis_exporter_scrape_duration_seconds_count:redis_exporter采集续时间秒数[lw]
redis_exporter_scrape_duration_seconds_sum:redis_exporter持续时间秒总和[lw]
redis_exporter_scrapes_total:redis_exporter抓取总数[lw]
redis_instance_info:实例信息[lw]
redis_keyspace_hits_total:键空间命中总数[lw]
redis_keyspace_misses_total:键空间未命中总数[lw]
redis_last_key_groups_scrape_duration_milliseconds:最后一个键组抓取持续时间毫秒[lw]
redis_last_slow_execution_duration_seconds:最后一个慢执行持续时间秒[lw]
redis_latest_fork_seconds:最新fork时间[lw]
redis_lazyfree_pending_objects:惰性删除或延迟释放的对象[lw]
redis_loading_dump_file:加载转储文件[lw]
redis_master_last_io_seconds_ago:master最后io过去时间[lw]
redis_master_repl_offset:主节点累加偏移量（判断主从是否同步）[lw]
redis_master_sync_in_progress:正在进行主同步[lw]
redis_mem_clients_normal:[lw]
redis_mem_clients_slaves:[lw]
redis_mem_fragmentation_bytes:内存碎片字节[lw]
redis_mem_fragmentation_ratio:内存碎片率[lw]
redis_mem_not_counted_for_eviction_bytes:内存不计入驱逐的字节数[lw]
redis_memory_max_bytes:内存最大字节[lw]
redis_memory_used_lua_bytes:lua脚本使用内存字节数[lw]
redis_memory_used_overhead_bytes:维护数据集的内部机制所需的内存开销[lw]
redis_memory_used_peak_bytes:内存使用峰值[lw]
redis_memory_used_rss_bytes:rss占用内存的字节数[lw]
redis_memory_used_scripts_bytes:脚本占用内存的字节数[lw]
redis_memory_used_startup_bytes:启动占用内存的字节数[lw]
redis_migrate_cached_sockets_total:[lw]
redis_net_input_bytes_total:网络input总数[lw]
redis_net_output_bytes_total:网络output总数[lw]
reids_process_id:进程号[lw]
redis_pubsub_channels:发布订阅频道[lw]
redis_pubsub_patterns:发布订阅模式[lw]
redis_rdb_bgsave_in_progress:[lw]
redis_rdb_changes_since_last_save:自上次保存以来的rdb更改[lw]
redis_rdb_current_bgsave_duration_sec:rdb当前bgsave持续时间[lw]
redis_rdb_last_bgsave_duration_sec:rdb上次bgsave持续时间[lw]
redis_rdb_last_bgsave_status:rdb上次bgsave状态[lw]
redis_rdb_last_cow_size_bytes:rdb上次cow的大小[lw]
redis_rdb_last_save_timestamp_seconds:rdb最后保存时间戳[lw]
redis_rejected_connections_total:拒绝的连接总数[lw]
redis_repl_backlog_first_byte_offset:复制起始偏移量[lw]
redis_repl_backlog_history_bytes:repl_backlog历史数据大小[lw]
redis_repl_backlog_is_active:repl_backlog是否开启[lw]
redis_replica_partial_resync_accepted:[lw]
redis_replica_partial_resync_denied:[lw]
redis_replica_resyncs_full:[lw]
redis_replication_backlog_bytes:[lw]
redis_second_repl_offset:[lw]
redis_slave_expires_tracked_keys:[lw]
redis_slave_info:从节点信息[lw]
redis_slave_priority:从节点优先级[lw]
redis_slave_repl_offset:从节点累加偏移量（判断主从是否同步）[lw]
redis_slowlog_last_id:慢查询日志最后一个的id[lw]
redis_slowlog_length:慢查询日志长度[lw]
redis_start_time_seconds:开始时间秒[lw]
redis_target_scrape_request_errors_total:[lw]
redis_up:运行时间[lw]
redis_uptime_in_seconds:正常运行时间[lw]
3.2 手动配置图表（方法一）
创建图表入口：
监控看图 > 监控大盘 > 新建大盘 > 新建大盘分组 > 新建图表
配置方式使用n9e（建议）
redis监控没有什么需要计算的，因此建议使用n9e方式监控，变量使用比prometheus方式灵活。
image.png
配置方式选择prometheus
变量可以根据需要定义
image.png
配置变量
image.png
3.3 导入模板（方法二）
image.png
模板内容如下：
[
{
"id": 0,
"name": "redis监控",
"tags": "",
"configs": "{\"tags\":[{\"tagName\":\"cluster\",\"key\":\"job\",\"value\":\"redis_k8s_pub\",\"prefix\":false,\"metric\":\"redis_memory_used_bytes\"},{\"tagName\":\"node\",\"key\":\"instance\",\"value\":\"redis://10.10.239.100:30020\",\"prefix\":false,\"metric\":\"redis_memory_used_bytes\"}]}",
"chart_groups": [
{
"id": 0,
"dashboard_id": 0,
"name": "Default chart group",
"weight": 0,
"charts": [
{
"id": 72,
"group_id": 15,
"configs": "{\"name\":\"客户端连接数\",\"mode\":\"prometheus\",\"link\":\"http://127.0.0.1:9090\",\"prome_ql\":[\"redis_connected_clients{job=\\\"$job\\\"}\"],\"layout\":{\"h\":2,\"w\":12,\"x\":0,\"y\":0,\"i\":\"0\"}}",
"weight": 0
},
{
"id": 73,
"group_id": 15,
"configs": "{\"name\":\"占用内存大小\",\"mode\":\"prometheus\",\"link\":\"http://127.0.0.1:9090\",\"prome_ql\":[\"redis_memory_used_bytes{job=\\\"$job\\\"}\"],\"layout\":{\"h\":2,\"w\":12,\"x\":12,\"y\":0,\"i\":\"1\"}}",
"weight": 0
},
{
"id": 74,
"group_id": 15,
"configs": "{\"name\":\"每分钟处理数据量\",\"mode\":\"prometheus\",\"link\":\"http://127.0.0.1:9090\",\"prome_ql\":[\"rate(redis_commands_processed_total{job=\\\"$job\\\"}[1m])\"],\"layout\":{\"h\":2,\"w\":12,\"x\":0,\"y\":2,\"i\":\"2\"}}",
"weight": 0
},
{
"id": 75,
"group_id": 15,
"configs": "{\"name\":\"缓存命中率\",\"mode\":\"prometheus\",\"link\":\"http://127.0.0.1:9090\",\"prome_ql\":[\"redis_keyspace_hits_total{job=\\\"$job\\\"}/(redis_keyspace_hits_total{job=\\\"$job\\\"}+redis_keyspace_misses_total{job=\\\"$job\\\"})\"],\"layout\":{\"h\":2,\"w\":12,\"x\":12,\"y\":2,\"i\":\"3\"}}",
"weight": 0
},
{
"id": 76,
"group_id": 15,
"configs": "{\"name\":\"网络IO\",\"mode\":\"prometheus\",\"link\":\"http://127.0.0.1:9090\",\"prome_ql\":[\"rate(redis_net_input_bytes_total{job=\\\"$job\\\"}[5m])\",\"rate(redis_net_output_bytes_total{job=\\\"$job\\\"}[5m])\"],\"layout\":{\"h\":2,\"w\":12,\"x\":0,\"y\":4,\"i\":\"4\"}}",
"weight": 0
},
{
"id": 81,
"group_id": 15,
"configs": "{\"name\":\"1分钟5条执行最多命令的次数\",\"mode\":\"prometheus\",\"link\":\"http://127.0.0.1:9090\",\"prome_ql\":[\"topk(5, irate(redis_commands_total{job=\\\"$job\\\"} [1m]))\"],\"layout\":{\"h\":2,\"w\":12,\"x\":12,\"y\":4,\"i\":\"5\"}}",
"weight": 0
},
{
"id": 83,
"group_id": 15,
"configs": "{\"name\":\"max_over_time\",\"mode\":\"prometheus\",\"link\":\"http://127.0.0.1:9090\",\"prome_ql\":[\"max(max_over_time(redis_uptime_in_seconds{job=\\\"$job\\\"}[5m]))\"],\"layout\":{\"h\":2,\"w\":12,\"x\":0,\"y\":6,\"i\":\"6\"}}",
"weight": 0
}
]
},
{
"id": 0,
"dashboard_id": 0,
"name": "Key",
"weight": 1,
"charts": [
{
"id": 271,
"group_id": 108,
"configs": "{\"name\":\"有效的key数量\",\"mode\":\"prometheus\",\"prome_ql\":[\"sum (redis_db_keys) - sum (redis_db_keys_expiring) \"],\"layout\":{\"h\":2,\"w\":8,\"x\":0,\"y\":0,\"i\":\"0\"}}",
"weight": 0
},
{
"id": 272,
"group_id": 108,
"configs": "{\"name\":\"过期的key数量\",\"mode\":\"prometheus\",\"prome_ql\":[\"sum (redis_db_keys_expiring{job=\\\"$job\\\",instance=\\\"$instance\\\"}) \"],\"layout\":{\"h\":2,\"w\":8,\"x\":8,\"y\":0,\"i\":\"1\"}}",
"weight": 0
},
{
"id": 273,
"group_id": 108,
"configs": "{\"name\":\"每个库里的key数量\",\"mode\":\"nightingale\",\"metric\":[\"redis_db_keys\"],\"tags\":{},\"layout\":{\"h\":2,\"w\":8,\"x\":16,\"y\":0,\"i\":\"2\"}}",
"weight": 0
}
]
}
]
}
]
【附录】
使用Grafana
如果用不使用n9e，也可以使用grafana绘图，方法如下：
模板
我使用763 这个模板
https://grafana.com/grafana/d
ashboards/763添加到grafana
（略）-
查看
image.png
