一次Memcached使用问题

收到群里反馈很多PHP请求执行超时,服务已经不可用。

查看服务器监控数据

发现 CPU 负载 在那段时间一直爬坡。 查询fpm slow 日志 发现请求都卡在了 memcache get 的方法上面。

发现出问题的请求有一个方法 频繁的读写一个key 这个 key 保存的内容 内存已经大于1MB, 一秒内有 500 多个请求,同时读写会导致memcache 的性能急剧下降。

自己写代码重现这个问题:
php 脚本 test-memcached.php:

<?php
$time_start = microtime(true);

$ac = new Memcached();
$ac->addServer('localhost', 11211);

#$ac->setOption(Memcached::OPT_SOCKET_SEND_SIZE, 1024*512);
#$ac->setOption(Memcached::OPT_SOCKET_RECV_SIZE, 1024*512);
#$ac->setOption(Memcached::OPT_RECV_TIMEOUT, 5);
#$ac->setOption(Memcached::OPT_SEND_TIMEOUT, 5);

$data = [];

for($i = 0; $i<30000; $i++){
  $data[] = md5($i);
}

$r = $ac->set('key', $data, 3600);
$set_time_end = microtime(true);
$set_time = $set_time_end - $time_start;
echo "Set Key Done in $set_time seconds \n";
$result = $ac->get('key');
$read_time_end = microtime(true);
$read_time = $read_time_end - $time_start;
echo "Read key Done in $read_time seconds \n";
$time_end = microtime(true);
$time = $time_end - $time_start;
echo "Did nothing in $time seconds\n";
?>

shell 测试脚本:

#!/bin/bash
date
for i in `seq 1 300`
do
{
 php test-memcached.php
 sleep 1
}&
done
wait #等待执行完成
date

分别修改 memcache key 保存内容的大小得到结果:

A:

for($i = 0; $i<30000; $i++){
  $data[] = md5($i);
}

B:

for($i = 0; $i<3; $i++){
  $data[] = md5($i);
}

测试结果:

A:

Did nothing in 18.769073009491 seconds
Did nothing in 14.950340032578 seconds
Did nothing in 19.823235034943 seconds
Did nothing in 20.014719009399 seconds
Did nothing in 20.894814014435 seconds
Did nothing in 20.827455997467 seconds
Did nothing in 15.345555067062 seconds
Did nothing in 18.984731197357 seconds
Did nothing in 21.08841586113 seconds

B:

Did nothing in 0.39238500595093 seconds
Did nothing in 0.42710494995117 seconds
Did nothing in 0.34245300292969 seconds
Did nothing in 0.084210872650146 seconds
Did nothing in 0.41426110267639 seconds
Did nothing in 0.12554693222046 seconds
Did nothing in 0.10487604141235 seconds
Did nothing in 0.60212993621826 seconds
Did nothing in 0.054439067840576 seconds
Did nothing in 0.11286902427673 seconds
Did nothing in 0.69583892822266 seconds
Did nothing in 0.14684200286865 seconds
Did nothing in 0.082473993301392 seconds
Did nothing in 0.34351587295532 seconds
Did nothing in 0.10698294639587 seconds


看结果可以得知 但 memcache 保存的内容变大的时候速度会非常慢,所以使用memcache 的时候要注意缓存值得大小。

memcache 缓存值最大大小是1M 引用 Oracle 网站的一个 FAQ

The default maximum object size is 1MB. In memcached 1.4.2 and later, you can change the maximum size of an object using the -I command line option.

For versions before this, to increase this size, you have to re-compile memcached. You can modify the value of the POWER_BLOCK within the slabs.c file within the source.

In memcached 1.4.2 and higher, you can configure the maximum supported object size by using the -I command-line option. For example, to increase the maximum object size to 5MB:

$ memcached -I 5m

If an object is larger than the maximum object size, you must manually split it. memcached is very simple: you give it a key and some data, it tries to cache it in RAM. If you try to store more than the default maximum size, the value is just truncated for speed reasons.

出现服务不可用的一个重要原因也有写缓存的次数和读缓存次数一样多。

最后修改测试脚本 依然保存很大的缓存值,但是只写入一次,测试并发的读取,基本都可以在1秒左右的读取到数据。

结论:
1.使用memcache的要注意缓存内容的大小,不要保存太大的内容在一个key上
2.缓存是读的次数远远大于写的次数,如果代码中发现写缓存的次数和读缓存的次数一样的时候要提高警惕
3.千万不要在数据库事务未提交的时候删除缓存

QA:
1.为啥使用MD5字符串?
因为memcache会自动压缩内容,如果用普通重复字符串测试效果不明显,线上出现问题的场景也是MD5串的缓存值,可以开启和关闭压缩选项?

测试代码下载