Caused by: org.springframework.data.redis.RedisConnectionFailureException: Could not get a resource from the pool; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67)
at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:41)
at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:37)
at org.springframework.data.redis.connection.jedis.JedisClusterConnection.convertJedisAccessException(JedisClusterConnection.java:3999)
at org.springframework.data.redis.connection.jedis.JedisClusterConnection.setEx(JedisClusterConnection.java:717)
at org.springframework.data.redis.core.DefaultValueOperations11.doInRedis(DefaultValueOperations.java:186)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:207)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:169)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:91)
at org.springframework.data.redis.core.DefaultValueOperations.set(DefaultValueOperations.java:182)
at io.renren.common.utils.SuppleDataUtils.provideIcc(SuppleDataUtils.java:107)
at io.renren.modules.job.task.TaskIcc.taskFour(TaskIcc.java:242)
... 10 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:53)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:226)
at redis.clients.jedis.JedisSlotBasedConnectionHandler.getConnectionFromSlot(JedisSlotBasedConnectionHandler.java:66)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:116)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:141)
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:60)
at redis.clients.jedis.BinaryJedisCluster.setex(BinaryJedisCluster.java:268)
at org.springframework.data.redis.connection.jedis.JedisClusterConnection.setEx(JedisClusterConnection.java:715)
... 18 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out
at redis.clients.jedis.Connection.connect(Connection.java:207)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:93)
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1767)
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:106)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at redis.clients.util.Pool.getResource(Pool.java:49)
... 28 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at redis.clients.jedis.Connection.connect(Connection.java:184)
... 35 common frames omitted
最近在本地测试通过springboot基础redis的方式连接redis集群,启动的时候没有报错。
到时当执行保存,查询到redis的操作的时候,报上面的错误。
首先看到错误信息,先是connect timed out后,再出现的Could not get a resource from the pool错误。一开始排查这个错误,由于使用的是默认连接池,debug的时候,也看到有装载连接池,为啥报错呢。本地通过redis-manger工具,连接集群地址也可以连接访问。从日志也看不出还有其他的错误信息。
于是自己写了main方法直接操作JedisCluster的方式来测试,出现的错误跟上面全文一样。用抓包工具wireshark过滤集群的地址,查看连接的消息也是正常的。
另外一个项目使用的reddison客户端,于是测试了deamo用reddison来连接集群地址,启动的时候,发现报如下错误
2019-11-06 20:21:41.143 INFO 19340 --- [ main] o.r.cluster.ClusterConnectionManager : Redis cluster nodes configuration got from 129.195.17.95/129.195.17.95:7001:
a5f080ab5a65840ba00ffd813860f1bf3882f24f 10.287.17.35:7006@17006 slave 0d213d690d4e101c1c9030c53e7d36cc0313e8ae 0 1573042900000 6 connected
b9b83af66f04231e4eec8e9848e50639621637d3 10.287.17.35:7005@17005 slave 2d2e758c95e9e4b86e1ec64f65ff90860319c642 0 1573042901733 5 connected
0d213d690d4e101c1c9030c53e7d36cc0313e8ae 10.287.17.35:7003@17003 master - 0 1573042899730 3 connected 10923-16383
f5e2805aecc1ad975e07e7c637049b686caaf4ef 10.287.17.35:7004@17004 slave 674d71329cab7e40e377db50a06d4a5a4699e4bf 0 1573042900731 4 connected
674d71329cab7e40e377db50a06d4a5a4699e4bf 10.287.17.35:7001@17001 myself,master - 0 1573042900000 1 connected 0-5460
2d2e758c95e9e4b86e1ec64f65ff90860319c642 10.287.17.35:7002@17002 master - 0 1573042898726 2 connected 5461-10922
2019-11-06 20:21:51.165 ERROR 19340 --- [sson-netty-1-10] o.r.cluster.ClusterConnectionManager : Can't connect to master: redis://10.287.17.35:7003 with slot ranges: [[10923-16383]]
2019-11-06 20:21:51.165 ERROR 19340 --- [sson-netty-1-11] o.r.cluster.ClusterConnectionManager : Can't connect to master: redis://10.287.17.35:7002 with slot ranges: [[5461-10922]]
2019-11-06 20:21:51.165 ERROR 19340 --- [sson-netty-1-12] o.r.cluster.ClusterConnectionManager : Can't connect to master: redis://10.287.17.35:7001 with slot ranges: [[0-5460]]
乖乖,恍然大悟;10.287.17.31地址是服务器的内网地址,本机是连不上内网地址的,只有通过外网地址连接。而我配置的是外网地址。然而,客户端在集群同步和心跳检查的时候,是拉取的集群信息,redis集群信息里面的节点的信息
配置的是内网地址。客户端就通过这个内网地址来同步信息了。
再次同wireshark过滤集群10.287.17.31,发现有大量的超时重传的包,也没有响应。