Lucene Query Syntax

Keyboard Matching

Search for word “­foo­” in the title field
title: foo

Search for phrase “foo bar” in the title field
title: “foo bar”

Search for phrase “foo bar” in the title field AND the phrase “­quick fox” in the body field.
title­:”foo bar” AND body:”quick fox”

Search for either the phrase “foo bar” in the title field AND the phrase “­quick fox” in the body field, or the word “­fox­” in the title field.
(titl­e:”foo bar” AND body:”quick fox”) OR title:fox

Search for word “­foo­” and not “­bar­” in the title field.
title:foo -title­:bar

Wildcard matching

Search for any word that starts with “­foo­” in the title field.
title­:foo*

Search for any word that starts with “­foo­” and ends with bar in the title field.
title­:fo­o*bar

Note that Lucene doesn’t support using a symbol as the first character of a *search.

Proximity matching

Search for “foo bar” within 4 words from each other.
“foo bar”~4

Range Searches

Range Queries allow one to match documents whose field(s) values are between the lower and upper bound specified by the Range Query. Range Queries can be inclusive or exclusive of the upper and lower bounds. Sorting is done lexico­gra­phi­cally.
mod_d­ate­:[2­0020101 TO 20030101]

Boosts

Query-time boosts allow one to specify which terms/­clauses are “more import­ant­”. The higher the boost factor, the more relevant the term will be, and therefore the higher the corres­ponding document scores.

A typical boosting technique is assigning higher boosts to title matches than to body content matches:
(titl­e:foo OR title:­bar­)^1.5 (body:foo OR body:bar)

Boolean Operators

To search for all transa­ctions except MySQL transa­ctions:
NOT type: mysql

To search for all MySQL SELECT queries with large attach­ments:
mysql.me­thod: SELECT AND mysql.s­ize: [10000 TO *]

Lucene also supports parent­heses to group sub queries.
To search for either INSERT or UPDATE MySQL queries with a respon­setime greater or equal with 30ms:
(mysq­l.m­ethod: INSERT OR mysql.m­ethod: UPDATE) AND respon­set­ime:[30 TO *]

git rebasing/merging forked repo

添加 upstream

git remote add upstream https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git

查看 upstream

# origin    https://github.com/YOUR_USERNAME/YOUR_FORK.git (fetch)
# origin    https://github.com/YOUR_USERNAME/YOUR_FORK.git (push)
# upstream  https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git (fetch)
# upstream  https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git (push)
git fetch upstream

rebase 或者 merge

git rebase upstream/master
# or
git merge upstream/master

Used Resources
https://help.github.com/articles/configuring-a-remote-for-a-fork/
https://help.github.com/articles/syncing-a-fork/

How do I install Java on Mac OSX allowing version switching?

原文链接

note: These solutions work for various versions of Java including Java 8 and the new Java 13, and for any other previous Java version covered by the listed version managers. This includes alternative JDK’s from OpenJDK, Oracle, IBM, Azul, Amazon Correto, Graal and more. Easily work with Java 7, Java 8, Java 9, Java 10, Java 11, Java 12, and Java 13!

You have a few options of how to do the installation as well as manage JDK switching. Installation can be done by Homebrew, SDKMANJabba, or a manual install. Switching can be done by JEnvSDKMANJabba, or manually by setting JAVA_HOME. All of these are described below.


Installation

First, install Java using whatever method you prefer including Homebrew, SDKMAN or a manual install of the tar.gz file. The advantages of a manual install is that the location of the JDK can be placed in a standardized location for Mac OSX.

Install with SDKMAN

This is a simple model in that it handles both installation and version switching, with a caveat that it installs the JDK into a non-standard directory.

<see below “Installing and Switching versions with SDKMAN”>

Install using Jabba

This is also a simple model in that both installation and version switching are handled by the same tool. The installations are made to a non-standard directory.

<see below “Installing and Switching versions with Jabba”>

Install manually from OpenJDK download page:

  1. Download OpenJDK for Mac OSX from http://jdk.java.net/ (for example Java 13)
  2. Unarchive the OpenJDK tar, and place the resulting folder (i.e. jdk-13.jdk) into your /Library/Java/JavaVirtualMachines/ folder since this is the standard and expected location of JDK installs. You can also install anywhere you want in reality.

Install with Homebrew

The version of Java available in Homebrew Cask previous to October 3, 2018 was indeed the Oracle JVM. Now however, it has now been updated to OpenJDK. Be sure to update Homebrew and then you will see the lastest version available for install.

  1. install Homebrew if you haven’t already. Make sure it is updated:
    brew update
  2. Add the casks tap, if you haven’t already (or you are not seeing older Java versions anymore with step #3):
    brew tap homebrew/cask-versions

    and for the AdoptOpenJDK versions, add that tap:

    brew tap adoptopenjdk/openjdk

    These casks change their Java versions often, and there might be other taps out there with additional Java versions.

  3. Look for installable versions:
    brew search java   

    or for AdoptOpenJDK versions:

    brew search jdk     
  4. Check the details on the version that will be installed:
    brew cask info java

    or for the AdoptOpenJDK version:

    brew cask info adoptopenjdk
  5. Install a specific version of the JDK such as java11adoptopenjdk8, or just java or adoptopenjdk for the current. For example:
    brew cask install java

    You can use the fully qualified path to older versions as well:

    brew cask install homebrew/cask-versions/java11

And these will be installed into /Library/Java/JavaVirtualMachines/ which is the traditional location expected on Mac OSX.

Other installation options:

Some other flavours of openJDK are:

Azul Systems Java Zulu certified builds of OpenJDK can be installed by following the instructions on their site.

Zulu® is a certified build of OpenJDK that is fully compliant with the Java SE standard. Zulu is 100% open source and freely downloadable. Now Java developers, system administrators, and end users can enjoy the full benefits of open source Java with deployment flexibility and control over upgrade timing.

Amazon Correto OpenJDK builds have an easy to use an installation package for version 8 or version 11 (other versions are coming), and installs to the standard /Library/Java/JavaVirtualMachines/ directory on Mac OSX.

Amazon Corretto is a no-cost, multiplatform, production-ready distribution of the Open Java Development Kit (OpenJDK). Corretto comes with long-term support that will include performance enhancements and security fixes. Amazon runs Corretto internally on thousands of production services and Corretto is certified as compatible with the Java SE standard. With Corretto, you can develop and run Java applications on popular operating systems, including Linux, Windows, and macOS.


Where is my JDK?!?!

To find locations of previously installed Java JDK’s installed at the default system locations, use:

/usr/libexec/java_home -V

Matching Java Virtual Machines (6):
13, x86_64: “OpenJDK 13” /Library/Java/JavaVirtualMachines/openjdk-13.jdk/Contents/Home 12, x86_64: “OpenJDK 12” /Library/Java/JavaVirtualMachines/jdk-12.jdk/Contents/Home
11, x86_64: “Java SE 11” /Library/Java/JavaVirtualMachines/jdk-11.jdk/Contents/Home
10.0.2, x86_64: “Java SE 10.0.2” /Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents/Home
9, x86_64: “Java SE 9” /Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home
1.8.0_144, x86_64: “Java SE 8” /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home

You can also report just the location of a specific Java version using -v. For example for Java 13:

/usr/libexec/java_home -v 13

/Library/Java/JavaVirtualMachines/jdk-13.jdk/Contents/Home

Knowing the location of the installed JDK’s is also useful when using tools like JEnv, or adding a local install to SDKMAN, or linking a system JDK in Jabba — and you need to know where to find them.

If you need to find JDK’s installed by other tools, check these locations:

  • SDKMAN installs to ~/.sdkman/candidates/java/
  • Jabba installs to ~/.jabba/jdk

Switching versions manually

The Java executable is a wrapper that will use whatever JDK is configured in JAVA_HOME, so you can change that to also change which JDK is in use.

For example, if you installed or untar’d JDK 13 to /Library/Java/JavaVirtualMachines/jdk-13.jdk if it is the highest version number it should already be the default, if not you could simply set:

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-13.jdk/Contents/Home

And now whatever Java executable is in the path will see this and use the correct JDK.

Using the /usr/libexec/java_home utility as previously described helps you to create aliases or to run commands to change Java versions by identifying the locations of different JDK installations. For example, creating shell aliases in your .profile or .bash_profile to change JAVA_HOME for you:

export JAVA_8_HOME=$(/usr/libexec/java_home -v1.8)
export JAVA_9_HOME=$(/usr/libexec/java_home -v9)
export JAVA_10_HOME=$(/usr/libexec/java_home -v10)
export JAVA_11_HOME=$(/usr/libexec/java_home -v11)
export JAVA_12_HOME=$(/usr/libexec/java_home -v12)
export JAVA_13_HOME=$(/usr/libexec/java_home -v13)

alias java8='export JAVA_HOME=$JAVA_8_HOME'
alias java9='export JAVA_HOME=$JAVA_9_HOME'
alias java10='export JAVA_HOME=$JAVA_10_HOME'
alias java11='export JAVA_HOME=$JAVA_11_HOME'
alias java12='export JAVA_HOME=$JAVA_12_HOME'
alias java13='export JAVA_HOME=$JAVA_13_HOME'

# default to Java 13
java13

Then to change versions, just use the alias.

java8
java -version

java version “1.8.0_144”

Of course, setting JAVA_HOME manually works too!


Switching versions with JEnv

JEnv expects the Java JDK’s to already exist on the machine and can be in any location. Typically you will find installed Java JDK’s in /Library/Java/JavaVirtualMachines/. JEnv allows setting the global version of Java, one for the current shell, and a per-directory local version which is handy when some projects require different versions than others.

  1. Install JEnv if you haven’t already, instructions on the site http://www.jenv.be/ for manual install or using Homebrew.
  2. Add any Java version to JEnv (adjust the directory if you placed this elsewhere):
    jenv add /Library/Java/JavaVirtualMachines/jdk-13.jdk/Contents/Home
  3. Set your global version using this command:
    jenv global 13

You can also add other existing versions using jenv add in a similar manner, and list those that are available. For example Java 8:

jenv add /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home 
jenv versions

See the JEnv docs for more commands. You may now switch between any Java versions (Oracle, OpenJDK, other) at any time either for the whole system, for shells, or per local directory.

To help manage JAVA_HOME while using JEnv you can add the export plugin to do this for you.

$ jenv enable-plugin export
  You may restart your session to activate jenv export plugin echo export plugin activated

The export plugin may not adjust JAVA_HOME if it is already set, so you may need to clear this variable in your profile so that it can be managed by JEnv.

You can also use jenv exec <command> <parms...> to run single commands with JAVA_HOME and PATH set correctly for that one command, which could include opening another shell.


Installing and Switching versions with SDKMAN

SDKMAN is a bit different and handles both the install and the switching. SDKMAN also places the installed JDK’s into its own directory tree, which is typically ~/.sdkman/candidates/java. SDKMAN allows setting a global default version, and a version specific to the current shell.

  1. Install SDKMAN from https://sdkman.io/install
  2. List the Java versions available to make sure you know the version ID
    sdk list java
  3. Install one of those versions, for example, Java 13:
    sdk install java 13.0.0-open 
  4. Make 13 the default version:
    sdk default java 13.0.0-open

    Or switch to 13 for the session:

    sdk use java 13.0.0-open

When you list available versions for installation using the list command, you will see a wide variety of distributions of Java:

sdk list java

And install additional versions, such as JDK 8:

sdk install java 8.0.181-oracle

SDKMAN can work with previously installed existing versions. Just do a local install giving your own version label and the location of the JDK:

sdk install java my-local-13 /Library/Java/JavaVirtualMachines/jdk-13.jdk/Contents/Home

And use it freely:

sdk use java my-local-13

More information is available in the SDKMAN Usage Guide along with other SDK’s it can install and manage.

SDKMAN will automatically manage your PATH and JAVA_HOME for you as you change versions.


Installing and Switching versions with Jabba

Jabba also handles both the install and the switching. Jabba also places the installed JDK’s into its own directory tree, which is typically ~/.jabba/jdk.

  1. Install Jabba by following the instructions on the home page.
  2. List available JDK’s
    jabba ls-remote
  3. Install Java JDK 12
    jabba install openjdk@1.12.0
  4. Use it:
    jabba use openjdk@1.12.0

You can also alias version names, link to existing JDK’s already installed, and find a mix of interesting JDK’s such as GraalVM, Adopt JDK, IBM JDK, and more. The complete usage guide is available on the home page as well.

Jabba will automatically manage your PATH and JAVA_HOME for you as you change versions.

Memcached 总结

总结 memcached 教程 [必看]

Memcached 二三事 [版权归 原文链接,本文修改一些引用资源地址]

实际应用Memcached时,我们遇到的很多问题都是因为不了解其内存分配机制所致,下面就让我们以此为开端来开始Memcached之旅吧!官方wiki

为了规避内存碎片问题,Memcached采用了名为SlabAllocator的内存分配机制。内存以Page为单位来分配,每个Page分给一个特定长度的Slab来使用,每个Slab包含若干个特定长度的Chunk。实际保存数据时,会根据数据的大小选择一个最贴切的Slab,并把数据保存在对应的Chunk中。如果某个Slab没有剩余的Chunk了,系统便会给这个Slab分配一个新的Page以供使用,如果没有Page可用,系统就会触发LRU机制,通过删除冷数据来为新数据腾出空间,这里有一点需要注意的是:LRU不是全局的,而是针对Slab而言的。

一个Slab可以有多个Page,这就好比在古代一个男人可以娶多个女人;一旦一个Page被分给某个Slab后,它便对Slab至死不渝,犹如古代那些贞洁的女人。但是女人的数量毕竟是有限的,所以一旦一些男人娶得多了,必然另一些男人就只剩下咽口水的份儿,这在很大程度上增加了社会的不稳定因素,于是乎我们要解放女性。

好在Memcached已经意识到解放女性的重要性,新版本中Page可以调配给其它的Slab:

memcached -o slab_reassign,slab_automove

换句话说:女人可以改嫁了!这方面,其实Memcached的儿子Twemcache革命得更彻底,他甚至写了一篇大字报,以事实为依据,痛斥老子的无能,有兴趣的可以继续阅读:Random Eviciton vs Slab Automove

了解Memcached内存使用情况的最佳工具是:Memcached-tool。如果我们发现某个Slab的Evicted不为零,则说明这个Slab已经出现了LRU的情况,这通常是个危险的信号,但也不能一概而论,需要结合Evict_Time来做进一步判断。Replacing the cache replacement algorithm in memcached

在Memcached的使用过程中,除了会遇到内存分配机制相关的问题,还有很多稀奇古怪的问题等着你呢,下面我选出几个有代表性的问题来逐一说明:

Cache失效后的拥堵问题

通常我们会为两种数据做Cache,一种是热数据,也就是说短时间内有很多人访问的数据;另一种是高成本的数据,也就说查询很很耗时的数据。当这些数据过期的瞬间,如果大量请求同时到达,那么它们会一起请求后端重建Cache,造成拥堵问题,就好象在北京上班做地铁似的,英文称之为:stampeding herd,老外这里的用词还是很形象的。

一般有如下几种解决思路可供选择:

首先,我们可以主动更新Cache。前端程序里不涉及重建Cache的职责,所有相关逻辑都由后端独立的程序(比如CRON脚本)来完成,但此方法并不适应所有的需求。

其次,我们可以通过加锁来解决问题。以PHP为例,伪代码大致如下:

<?php
function query()
{
    $data = $cache->get($key);

    if ($cache->getResultCode() == Memcached::RES_NOTFOUND) {
        if ($cache->add($lockKey, $lockData, $lockExpiration)) {
            $data = $db->query();
            $cache->set($key, $data, $expiration);
            $cache->delete($lockKey);
        } else {
            sleep($interval);
            $data = query();
        }
    }

    return $data;
}
?>

不过这里有一个问题,代码里用到了sleep,也就是说客户端会卡住一段时间,就拿PHP来说吧,即便这段时间非常短暂,也有可能堵塞所有的FPM进程,从而使服务中断。于是又有人想出了柔性过期的解决方案,所谓柔性过期,指的是设置一个相对较长的过期时间,或者干脆不再直接设置数据的过期时间,取而代之的是把真正的过期时间嵌入到数据中去,查询时再判断,如果数据过期就加锁重建,如果加锁失败,不再sleep,而是直接返回旧数据,以PHP为例,伪代码大致如下:

<?php
function query()
{
    $data = $cache->get($key);

    if (isset($data['expiration']) && $data['expiration'] < $now) {
        if ($cache->add($lockKey, $lockData, $lockExpiration)) {
            $data = $db->query();
            $data['expiration'] = $expiration;
            $cache->set($key, $data);
            $cache->delete($lockKey);
        }
    }

    return $data;
}
?>

问题到这里似乎已经圆满解决了,且慢!还有一些特殊情况没有考虑到:设想一下服务重启;或者某个Cache里原本没有的冷数据因为某些情况突然转换成热数据;又或者由于LRU机制导致某些键被意外删除,等等,这些情况都可能会让上面的方法失效,因为在这些情况里就不存在所谓的旧数据,等待用户的将是一个空页面。

好在我们还有Gearman这根救命稻草。当需要更新Cache的时候,我们不再直接查询数据库,而是把任务抛给Gearman来处理,当并发量比较大的时候,Gearman内部的优化可以保证相同的请求只查询一次后端数据库,以PHP为例,伪代码大致如下:

<?php

function query()
{
    $data = $cache->get($key);
    //说明:如果多个并发请求的$unique参数一样,那么实际上Gearman只会请求一次。
    if ($cache->getResultCode() == Memcached::RES_NOTFOUND) {
        $data = $gearman->do($function, $workload, $unique);
        $cache->set($key, $data, $expiration);
    }

    return $data;
}
?>

Multiget的无底洞问题

Facebook在Memcached的实际应用中,发现了Multiget无底洞问题,具体表现为:出于效率的考虑,很多Memcached应用都已Multiget操作为主,随着访问量的增加,系统负载捉襟见肘,遇到此类问题,直觉通常都是通过增加服务器来提升系统性能,但是在实际操作中却发现问题并不简单,新加的服务器好像被扔到了无底洞里一样毫无效果。

为什么会这样?让我们来模拟一下案发经过,看看到底发生了什么:

我们使用Multiget一次性获取100个键对应的数据,系统最初只有一台Memcached服务器,随着访问量的增加,系统负载捉襟见肘,于是我们又增加了一台Memcached服务器,数据散列到两台服务器上,开始那100个键在两台服务器上各有50个,问题就在这里:原本只要访问一台服务器就能获取的数据,现在要访问两台服务器才能获取,服务器加的越多,需要访问的服务器就越多,所以问题不会改善,甚至还会恶化。

不过,作为被告方,Memcached官方开发人员对此进行了辩护:

请求多台服务器并不是问题的症结,真正的原因在于客户端在请求多台服务器时是并行的还是串行的!问题是很多客户端,包括Libmemcached在内,在处理Multiget多服务器请求时,使用的是串行的方式!也就是说,先请求一台服务器,然后等待响应结果,接着请求另一台,结果导致客户端操作时间累加,请求堆积,性能下降。

如何解决这个棘手的问题呢?只要保证Multiget中的键只出现在一台服务器上即可!比如说用户名字(user:foo:name),用户年龄(user:foo:age)等数据在散列到多台服务器上时,不应按照完整的键名(user:foo:name和user:foo:age)来散列的,而应按照特殊的键(foo)来散列的,这样就保证了相关的键只出现在一台服务器上。以PHP的 Memcached客户端为例,有getMultiByKey和setMultiByKey可供使用。

Nagle和DelayedAcknowledgment的延迟问题

老实说,这个问题和Memcached没有半毛钱关系,任何网络应用都有可能会碰到这个问题,但是鉴于很多人在写Memcached程序的时候会遇到这个问题,所以还是拿出来聊一聊,在这之前我们先来看看Nagle和DelayedAcknowledgment的含义:

先看看Nagle:

假如需要频繁的发送一些小包数据,比如说1个字节,以IPv4为例的话,则每个包都要附带40字节的头,也就是说,总计41个字节的数据里,其中只有1个字节是我们需要的数据。为了解决这个问题,出现了Nagle算法。它规定:如果包的大小满足MSS,那么可以立即发送,否则数据会被放到缓冲区,等到已经发送的包被确认了之后才能继续发送。通过这样的规定,可以降低网络里小包的数量,从而提升网络性能。

再看看DelayedAcknowledgment:

假如需要单独确认每一个包的话,那么网络中将会充斥着无数的ACK,从而降低了网络性能。为了解决这个问题,DelayedAcknowledgment规定:不再针对单个包发送ACK,而是一次确认两个包,或者在发送响应数据的同时捎带着发送ACK,又或者触发超时时间后再发送ACK。通过这样的规定,可以降低网络里ACK的数量,从而提升网络性能。

Nagle和DelayedAcknowledgment虽然都是好心,但是它们在一起的时候却会办坏事。下面我们举例说说Nagle和DelayedAcknowledgment是如何产生延迟问题的:

客户端需要向服务端传输数据,传输前数据被分为ABCD四个包,其中ABC三个包的大小都是MSS,而D的大小则小于MSS,交互过程如下:

首先,因为客户端的ABC三个包的大小都是MSS,所以它们可以耗无障碍的发送,服务端由于DelayedAcknowledgment的存在,会把AB两个包放在一起来发送ACK,但是却不会单独为C包发送ACK。

接着,因为客户端的D包小于MSS,并且C包尚未被确认,所以D包不会立即发送,而被放到缓冲区里延迟发送。

最后,服务端触发了超时阈值,终于为C包发送了ACK,因为不存在未被确认的包了,所以即便D包小于MSS,也总算熬出头了,可以发送了,服务端在收到了所有的包之后就可以发送响应数据了。

说到这里,假如你认为自己已经理解了这个问题的来龙去脉,那么我们尝试改变一下前提条件:传输前数据被分为ABCDE五个包,其中ABCD四个包的大小都是MSS,而E的大小则小于MSS。换句话说,满足MSS的完整包的个数是偶数个,而不是前面所说的奇数个,此时又会出现什么情况呢?答案我就不说了,留给大家自己思考。

知道了问题的原委,解决起来就简单了:我们只要设置socket选项为TCP_NODELAY即可,这样就可以禁用Nagle,以PHP为例:

<?php
$memcached->setOption(Memcached::OPT_TCP_NODELAY, true);
?>

如果大家意犹未尽,可以继续浏览:TCP Performance problems caused by interaction between Nagle’s Algorithm and Delayed ACK。

memcached 启动参数 和 stat 参数详解

/usr/local/bin/memcached -u root -c 50000 -t 8 -m 256 -l 192.168.1.101 -p 11211 -d
-p <num>      监听的TCP端口(默认: 11211)
-U <num>      监听的UDP端口(默认: 11211, 0表示不监听)
-s <file>     用于监听的UNIX套接字路径(禁用网络支持)
-a <mask>     UNIX套接字访问掩码,八进制数字(默认:0700)
-l <ip_addr>  监听的IP地址。(默认:INADDR_ANY,所有地址)
-d            作为守护进程来运行
-r            最大核心文件限制
-u <username> 设定进程所属用户。(只有root用户可以使用这个参数)
-m <num>      所有slab class可用内存的上限,以MB为单位。(默认:64MB)
              (译者注:也就是分配给该memcached实例的内存大小。)
-M            内存用光时报错。(不会删除数据)
-c <num>      最大并发连接数。(默认:1024)
-k            锁定所有内存页。注意你可以锁定的内存上限。
              试图分配更多内存会失败的,所以留意启动守护进程时所用的用户可分配的内存上限。
              (不是前面的 -u <username> 参数;在sh下,使用命令"ulimit -S -l NUM_KB"来设置。)
-v            提示信息(在事件循环中打印错误/警告信息。)
-vv           详细信息(还打印客户端命令/响应)
-vvv          超详细信息(还打印内部状态的变化)
-h            打印这个帮助信息并退出
-i            打印memcached和libevent的许可
-P <file>     保存进程ID到指定文件,只有在使用 -d 选项的时候才有意义
-f <factor>   不同slab class里面的chunk大小的增长倍率。(默认:1.25)
              (译者注:每个slab class里面有相同数量个slab page,每个slab page里面有chunk,且在当前slab class内的chunk大小固定。
              而不同slab class里的chunk大小不一致,具体差异就是根据这个参数的倍率在增长,直到分配的内存用尽。)
-n <bytes>    chunk的最小空间(默认:48)
              (译者注:chunk数据结构本身需要消耗48个字节,所以一个chunk实际消耗的内存是n+48。)
-L            尝试使用大内存页(如果可用的话)。提高内存页尺寸可以减少"页表缓冲(TLB)"丢失次数,提高运行效率。
              为了从操作系统获得大内存页,memcached会把全部数据项分配到一个大区块。
-D <char>     使用 <char> 作为前缀和ID的分隔符
              这个用于按前缀获得状态报告。默认是":"(冒号)
              如果指定了这个参数,则状态收集会自动开启;如果没指定,则需要用命令"stats detail on"来开启。
-t <num>      使用的线程数(默认:4)
-R            每个连接可处理的最大请求数
-C            禁用CAS
-b            设置后台日志队列的长度(默认:1024)
-B            绑定协议 - 可能值:ascii,binary,auto(默认)
-I            重写每个数据页尺寸。调整数据项最大尺寸
echo -e "stats\nquit\n" | /usr/bin/nc 192.168.1.101 11211
STAT pid 20487               ## memcache 进程PID
STAT uptime 1977637         ## 自memcache启动以来,服务器运行秒数
STAT time 1461202739       ## 服务器当前unix时间戳
STAT version 1.4.21      ## memcache 服务器版本
STAT libevent 1.4.13-stable      ## libevent 版本
STAT pointer_size 64      ##  架构(32 或 64 位)
STAT rusage_user 150.835069      ## 进程累计用户时间
STAT rusage_system 249.086133      ## 进程累计系统时间
STAT curr_connections 10      ## 当前打开连接数
STAT total_connections 5509      ## 自memcache启动以来,打开的连接总数
STAT connection_structures 11      ## 服务器分配的连接结构数
STAT reserved_fds 40      ## 
STAT cmd_get 8913248      ## 自memcache启动以来,执行get命令总数
STAT cmd_set 123382      ## 自memcache启动以来,执行set命令总数
STAT cmd_flush 0      ## 自memcache启动以来,执行flush命令总数
STAT cmd_touch 0      ## 自memcache启动以来,执行touch_all命令总数
STAT get_hits 8913074      ## 自memcache启动以来,get命中次数
STAT get_misses 174      ## 自memcache启动以来,get未命中次数
STAT delete_misses 0      ## 自memcache启动以来,delete未命中次数
STAT delete_hits 0      ## 自memcache启动以来,delete命中次数
STAT incr_misses 0      ## 自memcache启动以来,incr未命中次数
STAT incr_hits 0      ## 自memcache启动以来,incr命中次数
STAT decr_misses 0      ## 自memcache启动以来,decr未命中次数
STAT decr_hits 0      ## 自memcache启动以来,decr命中次数
STAT cas_misses 0      ## 自memcache启动以来,cas未命中次数
STAT cas_hits 0      ## 自memcache启动以来,cas命中次数
STAT cas_badval 0      ## 使用擦拭次数
STAT touch_hits 0      ## 自memcache启动以来,touch命中次数
STAT touch_misses 0      ## 自memcache启动以来,touch未命中次数
STAT auth_cmds 0      ##
STAT auth_errors 0      ##
STAT bytes_read 111225505      ## memcached服务器从网络读取的总的字节数
STAT bytes_written 3621054898      ## memcached服务器发送到网络的总的字节数
STAT limit_maxbytes 33554432      ## memcached服务缓存允许使用的最大字节数(分配的内存数)
STAT accepting_conns 1      ## 目前接受的链接数
STAT listen_disabled_num 0      ##
STAT threads 8      ## 被请求的工作线程的总数量
STAT conn_yields 0      ## 连接操作主动放弃数目
STAT hash_power_level 16      ##
STAT hash_bytes 524288      ##
STAT hash_is_expanding 0      ##
STAT malloc_fails 0      ## 
STAT bytes 384154      ## 存储item字节数(当前存储占用的字节数)
STAT curr_items 856      ## item个数(当前存储的数据总数)
STAT total_items 123382      ## item总数(启动以来存储的数据总数)
STAT expired_unfetched 0      ##
STAT evicted_unfetched 0      ##
STAT evictions 0      ## LRU释放的对象数目。为了给新的数据项目释放空间,从缓存移除的缓存对象的数目。比如超过缓存大小时根据LRU算法移除的对象,以及过期的对象
STAT reclaimed 0      ## 已过期的数据条目来存储新数据的数目
STAT crawler_reclaimed 0      ##
STAT lrutail_reflocked 0      ##
END

redis持久化——RDB、AOF

原文

redis支持两种持久化方案:RDB和AOF。

RDB

RDB持久化是redis默认的,用来生成某一个时间点的数据快照;RDB是一个经过压缩的二进制文件,采用RDB持久化时服务器只会保存一个RDB文件(维护比较简单);

  • 每次进行RDB持久化时,redis都是将内存中完成的数据写到文件中,不是增量的持久化(写脏数据)
  • 写RDB文件时,先把内存中数据写到临时文件,然后替换原来的RDB文件;

1、RDB文件的生成:
RDB持久化可以手动执行,也可以按照服务器的配置定期执行。
1)save和bgsave命令:(手动用于生成RDB文件的命令)

  • save:会阻塞redis服务器进程,直到创建RDB文件完毕为止;(在此期间进程不能处理任何请求)
  • bgsave:fork一个子进程来创建RDB文件,父进程可以继续处理命令请求;

2)自动执行:
redis服务器允许用户通过设置配置文件save选项,让服务器每隔一段时间自动执行一次bgsave命令。如下:配置文件中的save选项
save 900 1
save 300 10
save 60 10000

服务器内部维护了一个dirty计数器和lastsave属性:

  • dirty:记录了距上次成功执行了save或bgsave命令之后,数据库修改的次数(写入、删除、更新等);
  • lastsave:unix时间戳,记录了上一次成功执行save或bgsave命令的时间;

redis服务器的周期性操作函数serverCron默认每100毫秒执行一次,该函数的一个主要作用就是检查save选项所设置的保存条件是否满足,如果满足就执行bgsave命令。检查的过程:根据系统当前时间、dirty和lastsave属性的值来检验保存条件是否满足。

补充:

  • 在执行bgsave期间,客户端发送的save、bgsave命令会被服务器拒绝执行(防止产生竞争);
  • 如果bgsave在执行,bgrewriteaof命令会被延迟到bgsave执行完毕后再执行;
  • 如果bgrewriteaof在执行,bgsave命令也会被延迟到bgrewriteaof命令执行完毕后再执行(bgsave和bgrewriteaof都是通过子进程完成的,不存在冲突,主要是考虑性能);

2、RDB文件的载入:

  • redis并没有专门的命令去载入RDB文件,只有在服务器启动的时候检测到RDB文件存在就会自动执行载入。
  • 如果redis启用了AOF持久化功能,那么服务器优先使用AOF文件还原数据。
  • 当服务器载入RDB文件时,会一直处于阻塞状态,直到载入完毕为止。
  • 载入时RDB文件时,系统会自动检查、如果是过期键不会加载到数据库中。

3、其他:
1)redis会根据配置文件中rdbcompression属性决定保存RDB文件时是否进行压缩存储(默认为yes),如果打开了压缩功能,redis会判断字符串长度>=20字节则压缩,否则不压缩存储;
2)redis自带了RDB文件检查工具redis-check-dump;
3)使用od命令打印RDB文件:[root@centOS1 dir]# od -oc dump.rdb
4)RDB文件名通过配置文件dbfilename dump.rdb指定,存放位置通过配置文件dir /var/lib/redis/ 指定;

AOF

有上面分析可知:RDB方式持久化的颗粒比较大,当服务器宕机时,到上次save或bgsave后的所有数据都会丢失。而AOF的持久化颗粒比较细,当服务器宕机后,只有宕机之前没来得AOF的操作数据会丢失。

1、AOF实现:

1)AOF持久化是通过保存redis服务器所执行的写命令来记录数据库状态的;被写入AOF文件的所有命令都是以Redis的命令请求协议格式保存的(Redis的请求协议是纯文本的)。服务器在启动时,通过载入AOF文件、并执行其中的命令来还原服务器状态。

2)AOF文件名通过配置文件appendfilename appendonly.aof 指定,存放位置通过配置文件dir /var/lib/redis/ 指定;

3)AOF步骤:

  • 命令追加:服务器在执行玩一个写命令后,会以协议的格式把其追加到aof_buf缓冲区末尾;
  • 文件写入:redis服务器进程就是一个事件循环,在每次事件循环结束,会根据配置文件中的appednfsync属性值决定是否将aof_buf中的数据写入到AOF文件中;
  • 文件同步:将内存缓冲区的数据写到磁盘;(由于OS的特性导致)

补充:

为了提高文件写入效率,现代OS中通常会把写入数据暂时保存在一个内存的缓冲区里面,等到缓冲区满或超时,一次性把其中的内容再写到磁盘。

4)appendfsync选项:

  • always:将aof_buf中所有内容写入、同步到AOF文件;
  • everysec:将aof_buf中所有内容写到AOF文件,如果上次同步AOF文件时间距当前时间超过1s,那么对AOF文件同步;(由专门线程负责)
  • no:将aof_buf中所有内容写入AOF文件,合适同步根据操作系统决定;

5)AOF持久化效率和安全性:(appendfsync选项控制)
always每次都要写入、同步,所以其安全性最高,效率是最慢的;everysec效率也足够快,也安全性也可以得到保证;no效率最高,但安全性比较差。

2、AOF文件载入:
AOF文件中记录了数据库中所有写操作的命令,所以服务器只需要重新执行一遍AOF文件中的命令即可恢复服务器关闭之前的状态。步骤如下:

  • 创建一个不带网络连接的伪客户端;
  • 从AOF文件中分析并读取一条写命令;
  • 使用伪客户端执行被读出的写命令;

3、AOF重写:
由于AOF记录了数据库的所有写命令,所以随着服务器的运行,AOF文件中内容会越来越大。实际上,对于一个键值,由于多次的修改,会产生很多写命令;中间的一些写操作可以直接省去,直接把最终的键值信息记录到AOF文件中即可,从而减小AOF文件大小。

1)AOF重写:
为了解决AOF文件体积膨胀问题,redis服务器使用了AOF重写功能:创建一个新的AOF文件来代替现有的AOF文件,新旧两个AOF文件所保存的数据库状态相同,但新AOF文件不会包含任何浪费空间的冗余命令,所以新AOF文件体积会比旧AOF文件体积小很多。

2)原理:
AOF文件重写并不需要对现有AOF文件进行任何读取、分析或者写入操作,这个功能是通过读取服务器当前的数据库状态来实现的。首先,从数据库中读取键现在的值,然后使用一条命令去记录键值对,代替之前记录这个键值对的多条命令,这就是AOF重写的原理。

注:

为了避免在执行命令时造成客户端缓冲区溢出,重写程序在处理集合、链表等发现带有多个元素时,会检查元素数量是否超过redis默认的64,如果超过了,redis会使用多个命令代替这一条命令。

3)bgrewriteaof命令、AOF重写缓冲区:
由于redis是单进程的,为了不在进行重写时阻塞服务,redis使用了子进程的方式进行AOF重写。——bgrewriteaof

在使用子进程进程AOF重写时会产生另一个问题:子进程在AOF重写时,服务器主进程还在继续处理命令请求,而新的命令可能会对现有数据库状态进行修改,而从使得服务器当前状态和重写后的AOF文件所保存的服务器状态不一致。为了解决这个问题,redis引入了AOF重写缓冲区。

AOF重写缓冲区是在服务器创建子进程之后开始使用:

  • 执行客户端命令;
  • 将执行后的写命令追加到aof_buf(AOF缓冲区);
  • 将执行后的写命令追加到AOF重写缓冲区;

aof_buf(AOF缓冲区)会定期被写入、同步到AOF文件;而在AOF重写期间新的命令会写到AOF重写缓冲区。当AOF重写完成后,会向父进程发送一个信号,父进程收到信号后会阻塞当前服务,进行如下操作:

  • 将AOF重写缓冲区中的写命令写入到新的AOF文件;(保证了新AOF文件数据库状态和当前数据库状态的一致)
  • 对新的AOF文件重命名、原子的覆盖旧的AOF;

注:

整个AOF重写中,只有信号处理是阻塞的;当信号处理完毕后父进程就可以接收命令请求了。

4、如果redis开始基于rdb进行的持久化,之后通过appendonly yes 打开了aof,这时重新启动redis后会根据aof进行载入,所以原来所有的数据无法加载到数据库中。

HaProxy日志详解

Log levels 日志级别
global 全局参数,如果实例上没设置参数,仅有log global那么每个实例都会使用该参数。

log global
log <address> [len <length>] <facility> [<level> [<minlevel>]]

address 日志发送的目的IP地址。

            - IPV4 默认UDP端口514  例如:127.0.0.1:514
            - IPV6 默认UDP端口514
            - 文件系统路径到scoket,保证文件是可写的。

length 日志输出行最大字符长度,值范围80-65535 默认值 1024
facility 必须是24个标准的syslog设施

             kern   user   mail   daemon auth   syslog lpr    news
             uucp   cron   auth2  ftp    ntp    audit  alert  cron2
             local0 local1 local2 local3 local4 local5 local6 local7

level 日志级别,可以设置一个最低日志级别,可以发送最低日志级别以上级别的日志信息

            emerg  alert  crit   err    warning notice info  debug
Example :

log global

log 127.0.0.1:514 local0 notice         

log 127.0.0.1:514 local0 notice notice

log ${LOCAL_SYSLOG}:514 local0 notice 

Log Formats
Haproxy 支持5种日志格式
1、默认格式:这是非常基本的,很少使用。它只提供关于传入连接的非常基本的信息 源IP:端口、目的IP:端口,和前端的名字。

Example :
    listen www
        mode http
        log global
        server srv1 127.0.0.1:8000

>&gt;&gt; Feb  6 12:12:09 localhost \
      haproxy[14385]: Connect from 10.0.1.2:33312 to 10.0.3.31:8012 \
      (www/HTTP)

    Field   Format                                Extract from the example above
      1   process_name '[' pid ']:'                            haproxy[14385]:
      2   'Connect from'                                          Connect from
      3   source_ip ':' source_port                             10.0.1.2:33312
      4   'to'                                                              to
      5   destination_ip ':' destination_port                   10.0.3.31:8012
      6   '(' frontend_name '/' mode ')'                            (www/HTTP)

2、TCP协议格式:通过“option tcplog” 启用该格式,这种格式提供了更丰富的信息,如定时器,连接数,队列大小等。这格式推荐纯TCP代理。

TCP协议格式字段定义串:
log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq


Example :
    frontend fnt
        mode tcp
        option tcplog
        log global
        default_backend bck

    backend bck
        server srv1 127.0.0.1:8000

>&gt;&gt; Feb  6 12:12:56 localhost \
      haproxy[14387]: 10.0.1.2:33313 [06/Feb/2009:12:12:51.443] fnt \
      bck/srv1 0/0/5007 212 -- 0/0/0/0/3 0/0
  Field   Format                                Extract from the example above
      1   process_name '[' pid ']:'                            haproxy[14387]:
      2   client_ip ':' client_port                             10.0.1.2:33313
      3   '[' accept_date ']'                       [06/Feb/2009:12:12:51.443]
      4   frontend_name                                                    fnt
      5   backend_name '/' server_name                                bck/srv1
      6   Tw '/' Tc '/' Tt*                                           0/0/5007
      7   bytes_read*                                                      212
      8   termination_state                                                 --
      9   actconn '/' feconn '/' beconn '/' srv_conn '/' retries*    0/0/0/0/3
     10   srv_queue '/' backend_queue                                      0/0

字段含义

Detailed fields description :
  - "client_ip" is the IP address of the client which initiated the TCP
    connection to haproxy. If the connection was accepted on a UNIX socket
    instead, the IP address would be replaced with the word "unix". Note that
    when the connection is accepted on a socket configured with "accept-proxy"
    and the PROXY protocol is correctly used, then the logs will reflect the
    forwarded connection's information.

  - "client_port" is the TCP port of the client which initiated the connection.
    If the connection was accepted on a UNIX socket instead, the port would be
    replaced with the ID of the accepting socket, which is also reported in the
    stats interface.

  - "accept_date" is the exact date when the connection was received by haproxy
    (which might be very slightly different from the date observed on the
    network if there was some queuing in the system's backlog). This is usually
    the same date which may appear in any upstream firewall's log.

  - "frontend_name" is the name of the frontend (or listener) which received
    and processed the connection.

  - "backend_name" is the name of the backend (or listener) which was selected
    to manage the connection to the server. This will be the same as the
    frontend if no switching rule has been applied, which is common for TCP
    applications.

  - "server_name" is the name of the last server to which the connection was
    sent, which might differ from the first one if there were connection errors
    and a redispatch occurred. Note that this server belongs to the backend
    which processed the request. If the connection was aborted before reaching
    a server, "<NOSRV>" is indicated instead of a server name.

  - "Tw" is the total time in milliseconds spent waiting in the various queues.
    It can be "-1" if the connection was aborted before reaching the queue.
    See "Timers" below for more details.

  - "Tc" is the total time in milliseconds spent waiting for the connection to
    establish to the final server, including retries. It can be "-1" if the
    connection was aborted before a connection could be established. See
    "Timers" below for more details.

  - "Tt" is the total time in milliseconds elapsed between the accept and the
    last close. It covers all possible processing. There is one exception, if
    "option logasap" was specified, then the time counting stops at the moment
    the log is emitted. In this case, a '+' sign is prepended before the value,
    indicating that the final one will be larger. See "Timers" below for more
    details.

  - "bytes_read" is the total number of bytes transmitted from the server to
    the client when the log is emitted. If "option logasap" is specified, the
    this value will be prefixed with a '+' sign indicating that the final one
    may be larger. Please note that this value is a 64-bit counter, so log
    analysis tools must be able to handle it without overflowing.

  - "termination_state" is the condition the session was in when the session
    ended. This indicates the session state, which side caused the end of
    session to happen, and for what reason (timeout, error, ...). The normal
    flags should be "--", indicating the session was closed by either end with
    no data remaining in buffers. See below "Session state at disconnection"
    for more details.

  - "actconn" is the total number of concurrent connections on the process when
    the session was logged. It is useful to detect when some per-process system
    limits have been reached. For instance, if actconn is close to 512 when
    multiple connection errors occur, chances are high that the system limits
    the process to use a maximum of 1024 file descriptors and that all of them
    are used. See section 3 "Global parameters" to find how to tune the system.

  - "feconn" is the total number of concurrent connections on the frontend when
    the session was logged. It is useful to estimate the amount of resource
    required to sustain high loads, and to detect when the frontend's "maxconn"
    has been reached. Most often when this value increases by huge jumps, it is
    because there is congestion on the backend servers, but sometimes it can be
    caused by a denial of service attack.

  - "beconn" is the total number of concurrent connections handled by the
    backend when the session was logged. It includes the total number of
    concurrent connections active on servers as well as the number of
    connections pending in queues. It is useful to estimate the amount of
    additional servers needed to support high loads for a given application.
    Most often when this value increases by huge jumps, it is because there is
    congestion on the backend servers, but sometimes it can be caused by a
    denial of service attack.

  - "srv_conn" is the total number of concurrent connections still active on
    the server when the session was logged. It can never exceed the server's
    configured "maxconn" parameter. If this value is very often close or equal
    to the server's "maxconn", it means that traffic regulation is involved a
    lot, meaning that either the server's maxconn value is too low, or that
    there aren't enough servers to process the load with an optimal response
    time. When only one of the server's "srv_conn" is high, it usually means
    that this server has some trouble causing the connections to take longer to
    be processed than on other servers.

  - "retries" is the number of connection retries experienced by this session
    when trying to connect to the server. It must normally be zero, unless a
    server is being stopped at the same moment the connection was attempted.
    Frequent retries generally indicate either a network problem between
    haproxy and the server, or a misconfigured system backlog on the server
    preventing new connections from being queued. This field may optionally be
    prefixed with a '+' sign, indicating that the session has experienced a
    redispatch after the maximal retry count has been reached on the initial
    server. In this case, the server name appearing in the log is the one the
    connection was redispatched to, and not the first one, though both may
    sometimes be the same in case of hashing for instance. So as a general rule
    of thumb, when a '+' is present in front of the retry count, this count
    should not be attributed to the logged server.

  - "srv_queue" is the total number of requests which were processed before
    this one in the server queue. It is zero when the request has not gone
    through the server queue. It makes it possible to estimate the approximate
    server's response time by dividing the time spent in queue by the number of
    requests in the queue. It is worth noting that if a session experiences a
    redispatch and passes through two server queues, their positions will be
    cumulated. A request should not pass through both the server queue and the
    backend queue unless a redispatch occurs.

  - "backend_queue" is the total number of requests which were processed before
    this one in the backend's global queue. It is zero when the request has not
    gone through the global queue. It makes it possible to estimate the average
    queue length, which easily translates into a number of missing servers when
    divided by a server's "maxconn" parameter. It is worth noting that if a
    session experiences a redispatch, it may pass twice in the backend's queue,
    and then both positions will be cumulated. A request should not pass
    through both the server queue and the backend queue unless a redispatch
    occurs.

3、HTTP协议格式:通过“option httplog” 启用该格式,http代理推荐使用该格式。

HTTP协议格式字段定义串:
    log-format %ci:%cp\ [%t]\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ \
               %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %hs\ %{+Q}r

Example :
    frontend http-in
        mode http
        option httplog
        log global
        default_backend bck

    backend static
        server srv1 127.0.0.1:8000

>&gt;&gt; Feb  6 12:14:14 localhost \
      haproxy[14389]: 10.0.1.2:33317 [06/Feb/2009:12:14:14.655] http-in \
      static/srv1 10/0/30/69/109 200 2750 - - ---- 1/1/1/1/0 0/0 {1wt.eu} \
      {} &quot;GET /index.html HTTP/1.1&quot;
  Field   Format                                Extract from the example above
      1   process_name '[' pid ']:'                            haproxy[14389]:
      2   client_ip ':' client_port                             10.0.1.2:33317
      3   '[' accept_date ']'                       [06/Feb/2009:12:14:14.655]
      4   frontend_name                                                http-in
      5   backend_name '/' server_name                             static/srv1
      6   Tq '/' Tw '/' Tc '/' Tr '/' Tt*                       10/0/30/69/109
      7   status_code                                                      200
      8   bytes_read*                                                     2750
      9   captured_request_cookie                                            -
     10   captured_response_cookie                                           -
     11   termination_state                                               ----
     12   actconn '/' feconn '/' beconn '/' srv_conn '/' retries*    1/1/1/1/0
     13   srv_queue '/' backend_queue                                      0/0
     14   '{' captured_request_headers* '}'                   {haproxy.1wt.eu}
     15   '{' captured_response_headers* '}'                                {}
     16   '&quot;' http_request '&quot;'                      &quot;GET /index.html HTTP/1.1&quot;

字段含义

Detailed fields description :
  - "client_ip" is the IP address of the client which initiated the TCP
    connection to haproxy. If the connection was accepted on a UNIX socket
    instead, the IP address would be replaced with the word "unix". Note that
    when the connection is accepted on a socket configured with "accept-proxy"
    and the PROXY protocol is correctly used, then the logs will reflect the
    forwarded connection's information.

  - "client_port" is the TCP port of the client which initiated the connection.
    If the connection was accepted on a UNIX socket instead, the port would be
    replaced with the ID of the accepting socket, which is also reported in the
    stats interface.

  - "accept_date" is the exact date when the TCP connection was received by
    haproxy (which might be very slightly different from the date observed on
    the network if there was some queuing in the system's backlog). This is
    usually the same date which may appear in any upstream firewall's log. This
    does not depend on the fact that the client has sent the request or not.

  - "frontend_name" is the name of the frontend (or listener) which received
    and processed the connection.

  - "backend_name" is the name of the backend (or listener) which was selected
    to manage the connection to the server. This will be the same as the
    frontend if no switching rule has been applied.

  - "server_name" is the name of the last server to which the connection was
    sent, which might differ from the first one if there were connection errors
    and a redispatch occurred. Note that this server belongs to the backend
    which processed the request. If the request was aborted before reaching a
    server, "<NOSRV>" is indicated instead of a server name. If the request was
    intercepted by the stats subsystem, "<STATS>" is indicated instead.

  - "Tq" is the total time in milliseconds spent waiting for the client to send
    a full HTTP request, not counting data. It can be "-1" if the connection
    was aborted before a complete request could be received. It should always
    be very small because a request generally fits in one single packet. Large
    times here generally indicate network trouble between the client and
    haproxy. See "Timers" below for more details.

  - "Tw" is the total time in milliseconds spent waiting in the various queues.
    It can be "-1" if the connection was aborted before reaching the queue.
    See "Timers" below for more details.

  - "Tc" is the total time in milliseconds spent waiting for the connection to
    establish to the final server, including retries. It can be "-1" if the
    request was aborted before a connection could be established. See "Timers"
    below for more details.

  - "Tr" is the total time in milliseconds spent waiting for the server to send
    a full HTTP response, not counting data. It can be "-1" if the request was
    aborted before a complete response could be received. It generally matches
    the server's processing time for the request, though it may be altered by
    the amount of data sent by the client to the server. Large times here on
    "GET" requests generally indicate an overloaded server. See "Timers" below
    for more details.

  - "Tt" is the total time in milliseconds elapsed between the accept and the
    last close. It covers all possible processing. There is one exception, if
    "option logasap" was specified, then the time counting stops at the moment
    the log is emitted. In this case, a '+' sign is prepended before the value,
    indicating that the final one will be larger. See "Timers" below for more
    details.

  - "status_code" is the HTTP status code returned to the client. This status
    is generally set by the server, but it might also be set by haproxy when
    the server cannot be reached or when its response is blocked by haproxy.

  - "bytes_read" is the total number of bytes transmitted to the client when
    the log is emitted. This does include HTTP headers. If "option logasap" is
    specified, the this value will be prefixed with a '+' sign indicating that
    the final one may be larger. Please note that this value is a 64-bit
    counter, so log analysis tools must be able to handle it without
    overflowing.

  - "captured_request_cookie" is an optional "name=value" entry indicating that
    the client had this cookie in the request. The cookie name and its maximum
    length are defined by the "capture cookie" statement in the frontend
    configuration. The field is a single dash ('-') when the option is not
    set. Only one cookie may be captured, it is generally used to track session
    ID exchanges between a client and a server to detect session crossing
    between clients due to application bugs. For more details, please consult
    the section "Capturing HTTP headers and cookies" below.

  - "captured_response_cookie" is an optional "name=value" entry indicating
    that the server has returned a cookie with its response. The cookie name
    and its maximum length are defined by the "capture cookie" statement in the
    frontend configuration. The field is a single dash ('-') when the option is
    not set. Only one cookie may be captured, it is generally used to track
    session ID exchanges between a client and a server to detect session
    crossing between clients due to application bugs. For more details, please
    consult the section "Capturing HTTP headers and cookies" below.

  - "termination_state" is the condition the session was in when the session
    ended. This indicates the session state, which side caused the end of
    session to happen, for what reason (timeout, error, ...), just like in TCP
    logs, and information about persistence operations on cookies in the last
    two characters. The normal flags should begin with "--", indicating the
    session was closed by either end with no data remaining in buffers. See
    below "Session state at disconnection" for more details.

  - "actconn" is the total number of concurrent connections on the process when
    the session was logged. It is useful to detect when some per-process system
    limits have been reached. For instance, if actconn is close to 512 or 1024
    when multiple connection errors occur, chances are high that the system
    limits the process to use a maximum of 1024 file descriptors and that all
    of them are used. See section 3 "Global parameters" to find how to tune the
    system.

  - "feconn" is the total number of concurrent connections on the frontend when
    the session was logged. It is useful to estimate the amount of resource
    required to sustain high loads, and to detect when the frontend's "maxconn"
    has been reached. Most often when this value increases by huge jumps, it is
    because there is congestion on the backend servers, but sometimes it can be
    caused by a denial of service attack.

  - "beconn" is the total number of concurrent connections handled by the
    backend when the session was logged. It includes the total number of
    concurrent connections active on servers as well as the number of
    connections pending in queues. It is useful to estimate the amount of
    additional servers needed to support high loads for a given application.
    Most often when this value increases by huge jumps, it is because there is
    congestion on the backend servers, but sometimes it can be caused by a
    denial of service attack.

  - "srv_conn" is the total number of concurrent connections still active on
    the server when the session was logged. It can never exceed the server's
    configured "maxconn" parameter. If this value is very often close or equal
    to the server's "maxconn", it means that traffic regulation is involved a
    lot, meaning that either the server's maxconn value is too low, or that
    there aren't enough servers to process the load with an optimal response
    time. When only one of the server's "srv_conn" is high, it usually means
    that this server has some trouble causing the requests to take longer to be
    processed than on other servers.

  - "retries" is the number of connection retries experienced by this session
    when trying to connect to the server. It must normally be zero, unless a
    server is being stopped at the same moment the connection was attempted.
    Frequent retries generally indicate either a network problem between
    haproxy and the server, or a misconfigured system backlog on the server
    preventing new connections from being queued. This field may optionally be
    prefixed with a '+' sign, indicating that the session has experienced a
    redispatch after the maximal retry count has been reached on the initial
    server. In this case, the server name appearing in the log is the one the
    connection was redispatched to, and not the first one, though both may
    sometimes be the same in case of hashing for instance. So as a general rule
    of thumb, when a '+' is present in front of the retry count, this count
    should not be attributed to the logged server.

  - "srv_queue" is the total number of requests which were processed before
    this one in the server queue. It is zero when the request has not gone
    through the server queue. It makes it possible to estimate the approximate
    server's response time by dividing the time spent in queue by the number of
    requests in the queue. It is worth noting that if a session experiences a
    redispatch and passes through two server queues, their positions will be
    cumulated. A request should not pass through both the server queue and the
    backend queue unless a redispatch occurs.

  - "backend_queue" is the total number of requests which were processed before
    this one in the backend's global queue. It is zero when the request has not
    gone through the global queue. It makes it possible to estimate the average
    queue length, which easily translates into a number of missing servers when
    divided by a server's "maxconn" parameter. It is worth noting that if a
    session experiences a redispatch, it may pass twice in the backend's queue,
    and then both positions will be cumulated. A request should not pass
    through both the server queue and the backend queue unless a redispatch
    occurs.

  - "captured_request_headers" is a list of headers captured in the request due
    to the presence of the "capture request header" statement in the frontend.
    Multiple headers can be captured, they will be delimited by a vertical bar
    ('|'). When no capture is enabled, the braces do not appear, causing a
    shift of remaining fields. It is important to note that this field may
    contain spaces, and that using it requires a smarter log parser than when
    it's not used. Please consult the section "Capturing HTTP headers and
    cookies" below for more details.

  - "captured_response_headers" is a list of headers captured in the response
    due to the presence of the "capture response header" statement in the
    frontend. Multiple headers can be captured, they will be delimited by a
    vertical bar ('|'). When no capture is enabled, the braces do not appear,
    causing a shift of remaining fields. It is important to note that this
    field may contain spaces, and that using it requires a smarter log parser
    than when it's not used. Please consult the section "Capturing HTTP headers
    and cookies" below for more details.

  - "http_request" is the complete HTTP request line, including the method,
    request and HTTP version string. Non-printable characters are encoded (see
    below the section "Non-printable characters"). This is always the last
    field, and it is always delimited by quotes and is the only one which can
    contain quotes. If new fields are added to the log format, they will be
    added before this field. This field might be truncated if the request is
    huge and does not fit in the standard syslog buffer (1024 characters). This
    is the reason why this field must always remain the last one.

4、CLF HTTP协议格式:相当于http协议格式。

CLF HTTP协议格式字段定义串:
    log-format %{+Q}o\ %{-Q}ci\ -\ -\ [%T]\ %r\ %ST\ %B\ \"\"\ \"\"\ %cp\ \
               %ms\ %ft\ %b\ %s\ \%Tq\ %Tw\ %Tc\ %Tr\ %Tt\ %tsc\ %ac\ %fc\ \
               %bc\ %sc\ %rc\ %sq\ %bq\ %CC\ %CS\ \%hrl\ %hsl

5、自定义格式

R	var	field name (8.2.2 and 8.2.3 for description)	type
%o	special variable, apply flags on all next var	
%B	bytes_read (from server to client)	numeric
H	%CC	captured_request_cookie	string
H	%CS	captured_response_cookie	string
%H	hostname	string
H	%HM	HTTP method (ex: POST)	string
H	%HP	HTTP request URI without query string (path)	string
H	%HQ	HTTP request URI query string (ex: ?bar=baz)	string
H	%HU	HTTP request URI (ex: /foo?bar=baz)	string
H	%HV	HTTP version (ex: HTTP/1.0)	string
%ID	unique-id	string
%ST	status_code	numeric
%T	gmt_date_time	date
%Tc	Tc	numeric
%Tl	local_date_time	date
H	%Tq	Tq	numeric
H	%Tr	Tr	numeric
%Ts	timestamp	numeric
%Tt	Tt	numeric
%Tw	Tw	numeric
%U	bytes_uploaded (from client to server)	numeric
%ac	actconn	numeric
%b	backend_name	string
%bc	beconn (backend concurrent connections)	numeric
%bi	backend_source_ip (connecting address)	IP
%bp	backend_source_port (connecting address)	numeric
%bq	backend_queue	numeric
%ci	client_ip (accepted address)	IP
%cp	client_port (accepted address)	numeric
%f	frontend_name	string
%fc	feconn (frontend concurrent connections)	numeric
%fi	frontend_ip (accepting address)	IP
%fp	frontend_port (accepting address)	numeric
%ft	frontend_name_transport (‘~’ suffix for SSL)	string
%lc	frontend_log_counter	numeric
%hr	captured_request_headers default style	string
%hrl	captured_request_headers CLF style	string list
%hs	captured_response_headers default style	string
%hsl	captured_response_headers CLF style	string list
%ms	accept date milliseconds (left-padded with 0)	numeric
%pid	PID	numeric
H	%r	http_request	string
%rc	retries	numeric
%rt	request_counter (HTTP req or TCP session)	numeric
%s	server_name	string
%sc	srv_conn (server concurrent connections)	numeric
%si	server_IP (target address)	IP
%sp	server_port (target address)	numeric
%sq	srv_queue	numeric
S	%sslc	ssl_ciphers (ex: AES-SHA)	string
S	%sslv	ssl_version (ex: TLSv1)	string
%t	date_time (with millisecond resolution)	date
%ts	termination_state	string
H	%tsc	termination_state with cookie status	string
R = Restrictions : H = mode http only ; S = SSL only

Example :

global
        maxconn 65535
        chroot /usr/local/haproxy
        uid 99
        gid 99
        daemon
        nbproc 1
        description haproxy
        pidfile /var/run/haproxy.pid
defaults
        log global
        mode http
        balance roundrobin
        option forceclose
        option dontlognull
        option redispatch
        option abortonclose
        log-format %ci:%cp\ [%t]\ %U\ %HM\ %HU\ %HV\ %ST\ %si:%sp

>&gt;&gt; Sep 12 10:17:52 localhost haproxy[22909]: 10.1.250.98:53300 [12/Sep/2016:10:17:52.532] 496 GET / HTTP/1.1 200 10.1.1.20:9090

错误日志
error日志格式:

 >>> Dec  3 18:27:14 localhost \
          haproxy[6103]: 127.0.0.1:56059 [03/Dec/2012:17:35:10.380] frt/f1: \
          Connection error during SSL handshake

  Field   Format                                Extract from the example above
      1   process_name '[' pid ']:'                             haproxy[6103]:
      2   client_ip ':' client_port                            127.0.0.1:56059
      3   '[' accept_date ']'                       [03/Dec/2012:17:35:10.380]
      4   frontend_name "/" bind_name ":"                              frt/f1:
      5   message                        Connection error during SSL handshake

捕捉 HTTP Headers

Example:
capture request header Host len 15
capture request header X-Forwarded-For len 15
capture request header Referer len 15

自定义headers
frontend webapp
        bind *:80
        capture request header test len 20
        capture request header test2 len 20

HAProxy配置文档

HAProxy的配置过程包含3个主要的参数来源:

  • 来自命令行的参数,始终是优先的
  • 配置文件的“global”部分,设置进程全局范围有效的参数
  • 配置文件的代理部分,以“defaults”、“listen”、“frontend”和“backend”的形式出现

配置文件的语法是:一行一行的配置信息,每行以本手册涉及的关键词开始;跟随一个或多个可选参数,以空格分隔。如果空格必须出现在字符串中,则空格必须前置一个反斜杠(\)进行转义。反斜杠自己如果要出现在字符串中,也需要写两遍来转义。

时间格式

某些参数值表示时间,如超时时间。这些值通常以毫秒为单位表示(除非显式声明为其他单位),也可以在数值后添加单位后缀来声明使用其他单位。谨记这一点非常重要,我们后面不会对关键词的时间参数重复解释。支持的单位有:

  • us:微秒。1微秒 = 1/1000000秒
  • ms:毫秒。1毫秒 = 1/1000秒,默认单位
  • s:秒。1秒 = 1000毫秒
  • m:分钟。1分钟 = 60秒 = 60000毫秒
  • h:小时。1小时 = 60分钟 = 3600秒 = 3600000毫秒
  • d:天。1天 = 24小时 = 1440分钟 = 86400秒 = 86400000毫秒

示例

# 简单配置一个HTTP代理监听80端口,将请求转发到一个后端“servers”,
# 该后端仅有一个服务器“server1”监听着127.0.0.1:8000
global
    daemon
    maxconn 256

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend http-in
    bind *:80
    default_backend servers

backend servers
    server server1 127.0.0.1:8000 maxconn 32

# 以一个listen块来定义与上例相同的配置。更简短但
# 表达能力弱一些,特别是在HTTP模式下
global
    daemon
    maxconn 256

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

listen http-in
    bind *:80
    server server1 127.0.0.1:8000 maxconn 32

假设haproxy命令在$PATH定义的查找路径中,可以这样在shell中测试上述配置:

$ sudo haproxy -f configuration.conf -c

全局参数

“global”部分的参数是进程全局范围有效的,通常也和操作系统相关。这些参数通常设置一次就一劳永逸了。其中某些参数具有等价的命令行选项。

“global”部分支持以下关键词:

  • 进程管理与安全
    • chroot
    • daemon
    • gid
    • group
    • log
    • log-send-hostname
    • nbproc
    • pidfile
    • uid
    • ulimit-n
    • user
    • stats
    • node
    • description
  • 性能调优
    • maxconn
    • maxpipes
    • noepoll
    • nokqueue
    • nopoll
    • nosepoll
    • nosplice
    • spread-checks
    • tune.bufsize
    • tune.chksize
    • tune.maxaccept
    • tune.maxpollevents
    • tune.maxrewrite
    • tune.rcvbuf.client
    • tune.rcvbuf.server
    • tune.sndbuf.client
    • tune.sndbuf.server
  • 调试
    • debug
    • quiet

进程管理与安全

chroot

将当前目录修改为 <jail dir> ,并在失去权限之前在该目录下执行一次chroot()。这能够在某些不明漏洞被攻击的情况下提高安全级别,因为这会使得骇客很难攻击系统。但这仅在进程以超级用户权限启动时才有效。一定要确保目录为空且任何人都不可写。

daemon

使进程fork为守护进程。生产运营环境(operation)下的推荐模式,等价于命令行的“-D”选项。也可以通过命令行“-db”选项来禁用。

log <address> <facility> [max level [min level]]

添加一个全局syslog服务器。可以定义多达两个的全局服务器。它们会收到进程启动和退出时的日志,还有配置有“log global”的代理的所有日志。

<address>可以以下两者之一:

  • 一个IPv4地址,可选跟随一个冒号和一个UDP端口。如果未指定端口,则默认使用514(标准syslog端口)。
  • 一个UNIX域套接字(socket)的文件系统路径,留意chroot(确保在chroot中可以访问该路径)和uid/gid(确保该路径相应地可写)。

<facility>必须是24个标准syslog设备(facilities)之一:

kern    user    mail    daemon  auth    syslog  lpr     news
uucp    cron    auth2   ftp     ntp     audit   alert   cron2
local0  local1  local2  local3  local4  local5  local6  local7

可以指定一个可选的级别来过滤输出的信息。默认情况下会发送所有信息。如果指定了一个最大级别,那么仅有严重性至少同于该级别的信息才会被发送。也可以指定一个可选的最小级别,如果设置了,则以比该级别更严重的级别发送的日志会被覆盖到该级别。这样能避免一些默认syslog配置将“emerg”信息发送到所有终端。八个级别如下所示:

emerg   alert   crit    err     warning     notice  info    debug

log-send-hostname [<string>]

设置syslog头部的主机名称(hostname)字段。如果设置了可选的“string”参数,那么头部主机名称字段会被设置为string内容,否则使用系统的主机名。通常用于不通过一个中间syslog服务器来转发日志信息之时,或仅仅是为了定制日志中打印的主机名称。

log-tag <string>

将syslog头部的tag字段设置为string。默认为从命令行启动的程序名称,一般为“haproxy”。有时为了区分同一主机上的多个进程会比较有用。

nbproc <number>

当以守护进程方式运行时创建个进程。该指令要求“daemon”模式。默认情况下,仅会创建一个进程,这是生产运营环境下的推荐模式。对于每个进程可持有的文件描述符较少的系统,也许需要fork多个守护进程。但使用多个进程调试起来会更困难,所以非常不推荐使用。

pidfile <pidfile>

将所有守护进程的pid写到文件中。该选项等价于命令行参数“-p”。该文件对于启动进程的用户必须是可访问的。

ulimit-n <number>

将单个进程可持有文件描述符的最大数目设置为。默认情况下会自动计算出该值,因此建议不使用该选项。

注:linux下有个命令:ulimit,进一步阅读: 通过 ulimit 改善系统性能

node <name>

仅允许使用字母、数字、连字符和下划线,就像DNS名称。

在HA配置中当有两个或更多进程或服务器共享相同的IP地址时,该语句比较有用。通过为所有节点设置一个不同的节点名称,非常易于即时找到哪台服务器正在处理流量。

description <text>

添加一段文本来描述当前实例。

注意:要求对某些字符(例如 # )进行转移,并且该文本会被插入到一个html页面中,所以应避免使用“<”和“>”字符。

性能调优

maxconn <number>

将单个进程的最大并发连接数目设置为。该选项等价于命令行参数“-n”。当达到该限制时,代理会停止接受连接。“ulimit-n”参数的值会根据该值自动调整。

maxpipes <number>

将单个进程可持有管道的最大数目设置为。目前,管道仅用于基于内核的TCP粘合(tcp splicing,参考阅读:TCP Splicing )。由于每个管道包含两个文件描述符,“ulimit-n”的值也会相应地增大。默认值为maxconn/4,对于大多数任务繁重的使用情况似乎已远远够用了。粘合的代码会动态分配和释放管道,也可以回退为标准拷贝,因此将该值设置得过低应该仅会影响性能。

spread-checks <0..50, in percent>

有时会期望不要以精确的时间间隔发送服务器健康检测数据包,例如当多个逻辑服务器位于同一物理服务器上。借助该参数,在检测时间间隔上添加一点随机性是可能的。2到5之间的一个数值看起来效果不错。默认值为0。

tune.bufsize <number>

将缓冲区大小设置为(以字节为单位)。更低的值允许在相同大小的内存中共存更多会话,而更高的值允许一些带有大量cookie的应用能够正常工作。默认值为16384,在编译构建之时可以修改。强烈建议不要修改默认值,因为非常低的值会破坏某些服务,如统计,而大于默认大小的值会增大内存使用量,可能会导致系统耗尽内存。随着这个值的增大,全局的maxconn参数至少应该减小相同的比率。

tune.chksize <number>

将检测缓冲区大小设置为该值(单位为字节)。更高的值也许有助于在海量页面中找到字符串或正则模式,虽然这样做也许意味着更多的内存和CPU使用量。默认值为16384,可以在编译构建时修改。不推荐改变该值,但是可能的话还是让检测更优吧(啥检测?目前还没搞懂)

tune.maxaccept <number>

设置一个进程在一次唤醒中也许会处理的连续接收的请求的最大数目(the maximum number of consecutive accepts that a process may perform on a single wake up)。更高的值给予高请求连接率更高优先级,更低的值则给予已建立的连接更高的(处理)优先级。在单进程模式下,该值默认限制为100。然而,在多进程模式中(nbproc > 1),该值默认为8,这样当一个进程唤醒时,它不会自己接受所有进入的连接,而是将一部分请求留给其他进程。将该值设置为-1会完全禁用这个限制。通常不需要变动该值。

tune.maxpollevents <number>

设置在对轮询系统(polling system)的一次调用中能同时(at once)处理的事件的最大数目。默认值不同的操作系统有所不同。需要注意的是将该值减小到200以下往往会以网络带宽为代价稍微减小延迟,而增大该值至200以上则是以延迟换取稍稍提升的网络带宽。

tune.maxrewrite <number>

将保留的缓冲空间设置为该值,以字节为单位。该保留空间是用于头部重写或附加信息的。The first reads on sockets will never fill more than bufsize-maxrewrite(啥意思?)。一直以来该值都默认为bufsize的一半,但这样做并没有多大的意义,因为很少需要添加大量头部。该值设置得过高会影响大量请求或响应的处理。设置得过低又会阻碍为大量请求增加新头部信息或影响POST请求。将该值设置为1024左右通常是比较明智的。如果比那还大,它会自动重新调整为bufsize的一半。这意味着修改bufsize时你并不需要担心该值。

tune.rcvbuf.client <number>

tune.rcvbuf.server <number>

将客户端或服务器端的内核套接字接收缓冲大小强制设置为指定大小(单位为字节)。该值应用于所有TCP/HTTP前端和后端。一般不应该设置,默认值(0)让内核根据内存可用量自动优化该值。然而有时为了通过阻止缓冲过多接收到的数据来节省内核内存,将该参数设置为一个非常低的值(如:4096)会有助于此。但是更低的值会显著地增大CPU使用量。

tune.sndbuf.client <number>

tune.sndbuf.server <number>

调试

debug

启用debug模式,将所有来往信息输出到标准输出,并进程无法在后台运行(即无法成为守护进程?)。该指令等价于命令行参数“-d”。在生产环境配置中绝对不应该使用,因为它会阻碍完整的系统启动(为什么?什么意思?)。

代理

代理配置可以位于以下部分中:

  • defaults
  • frontend
  • backend
  • listen

一个“defaults”部分是为该部分之后的所有其他部分设置默认参数。那些默认参数可以在下一个“defaults”部分中被重置。name是可选的,但为了更好的可读性,鼓励使用。

一个“frontend”部分描述一组接受客户端连接的监听套接字。

一个“backend”部分则是描述一组服务器,代理会将进入的连接转发到这些服务器。

一个“listen”部分定义一个完整的代理,将前端后端部分合并入一个部分中。对于仅是TCP的流量通常比较有用(为什么?)。

所有代理的名字必须由大小写字母、数字、‘-’(连字符)、‘_’(下划线)、‘.’(点)和‘:’(冒号)中的字符组成。ACL名称是大小写敏感的,这意味着“www”和“WWW”是两个不同的代理。

当前,支持两种主要的代理模式:“tcp”,又名4层(协议)和“http”,又名7层(协议)。在4层模式中,HAProxy简单地在两方之间转发双向流量。在7层模式中,HAProxy会分析协议,并且可以基于任意标准来允许、阻止、转换、增加、修改或删除请求或响应中的任意内容来与协议交互。

代理关键词参考手册

(仅对部分关键词进行说明)

acl <aclname> <criterion> [flags] [operator] <value> ...

可用于:frontend、listen、backend

“ACL”:Access Control List

声明或添加一个访问控制列表。其他地方根据调用该acl进行条件判断做出不同的行为。


appsession <cookie> len <length> timeout <holdtime> [request-learn] [prefix] [mode <path-parameters|query-string>]

可用于:listen、backend

参考文章: haproxy 解决集群session 共享问题方法


balance <algorithm> [ <arguments> ]

balance url_param <param> [check_post [<max_wait>]]

可用于:defaults、listen、backend

定义用于后端的负载均衡算法。

参考文章: HAProxy 配置手册 4.2 balance 相关


block { if | unless } <condition>

可用于:frontend、listen、backend

如果/除非匹配到某条件,则阻止一个7层协议的请求。

如果/除非匹配到,HTTP请求在7层处理中很早就会被阻止掉。如果请求被阻止,则返回一个403错误。条件参考ACL。这一般用于拒绝对某些敏感资源的访问,如果符合/不符合某些条件。每个实例中“block”语句的数量并没有一个固定的限制。

示例:

acl invalid_src src         0.0.0.0/7 224.0.0.0/3
acl invalid_src src_port    0:1023
acl local_dst   hdr(host) -i localhost
block if invalid_src | local_dst

capture cookie <name> len <length>

可用于:frontend、listen

捕获并记录请求或相应中的cookie。

参数

<name> 是所要捕获cookie的名称的开始部分。若想精确匹配名称,只需为名称添加一个等于号的后缀即可。完整的名称会出现在日志中。

<length> 是能在日志中记录的字符最大长度,包括cookie名称,等于号和值,以“名称=值”的形式出现。如果字符串长度超过了<length>,则会截掉右边部分。

示例

capture cookie ASPSESSION len 32

capture request header <name> len <length>

可用于:frontend、listen

捕获记录指定请求头首次出现的值。

示例 :

capture request header Host len 15
capture request header X-Forwarded-For len 15
capture request header Referrer len 15

capture response header <name> len <length>

可用于:frontend、listen

捕获记录指定响应头首次出现的值。

示例

capture response header Content-length len 9
capture response header Location len 15

cookie <name> [ rewrite | insert | prefix ] [ indirect ] [ nocache ] [ postonly ] [ preserve ] [ httponly ] [ secure ] [ domain <domain> ]\* [ maxidle <idle> ] [ maxlife <life> ]

可用于:defaults、listen、backend

在后端(backend)启用基于cookie的持久性。

HAProxy在WEB服务端发送给客户端的cookie中插入(或添加加前缀)haproxy定义的后端的服务器COOKIE ID。


default-server [param*]

可用于:defaults、listen、backend

修改后端服务器的默认参数。

参数 :

<param*> 提供给服务器的一个参数列表。

示例 :

default-server inter 1000 weight 13

default_backend <backend>

可用于:defaults、frontend、listen

为没有匹配到“use_backend”规则的情况指定使用的后端。

在使用关键词“use_backend”实现前端后端之间内容转换时,指明未匹配到规则时使用的后端通常比较有用。一般该后端是动态后端,用于接收所有不确定的请求。

示例 :

use_backend     dynamic if  url_dyn
use_backend     static  if  url_css url_img extension_img
default_backend dynamic

errorfile <code> <file>

可用于:defaults、frontend、listen、backend

以一个文件的内容替换HAProxy生成的错误信息,返回给请求客户端。

参数

<code> HTTP状态码。目前,HAProxy支持产生200、400、403、408、500、502、503以及504的状态信息。

<file> 指定一个包含完整HTTP响应的文件。建议在文件名后加“.http”后缀,这样别人就不会将该响应内容与HTML错误页面相混淆。也建议使用绝对路径,因为是在执行任何chroot之前读取文件的。

需要理解的是该关键词的功能并不是改写服务器返回的错误信息,而是检测错误信息并由HAProxy返回响应。这也就是为什么仅支持一小部分的错误状态码。

Code 200 is emitted in response to requests matching a “monitor-uri” rule.

The files are read at the same time as the configuration and kept in memory. For this reason, the errors continue to be returned even when the process is chrooted, and no file change is considered while the process is running. A simple method for developing those files consists in associating them to the
403 status code and interrogating a blocked URL.

示例 :

::

errorfile 400 /etc/haproxy/errorfiles/400badreq.http
errorfile 403 /etc/haproxy/errorfiles/403forbid.http
errorfile 503 /etc/haproxy/errorfiles/503sorry.http

errorloc303 <code> <url>

可用于:defaults、frontend、listen、backend

返回一个HTTP重定向到一个URL,而不是返回HAProxy生成的错误信息。

参数

<code> HTTP状态码。目前,HAProxy支持状态码400、403、408、500、502、503和504。

<url> 响应头字段“Location”的确切内容。可能包含一个到相同站点上错误信息页面的相对URI,或一个到另一站点上错误信息页面的绝对URI。特别要注意的是使用相对URI要避免该URI自己也产生相同的错误(如:500)从而造成重定向成环。

hash-type <method>

可用于:defaults、listen、backend

指定一种方法用于将哈希值映射到服务器。


http-request { allow | deny | auth [realm <realm>] } [ { if | unless } <condition> ]

可用于:frontend、listen、backend

7层请求的访问控制。

这些选项允许精细地控制对一个frontend/listen/backend的访问。每个选项都可能跟随一个if/unless和acl。如果一个选项的条件被匹配到,则不再继续往后匹配。

每个实例的http-request语句的数量并没有固定的限制。

示例 :

acl nagios src 192.168.129.3
acl local_net src 192.168.0.0/16
acl auth_ok http_auth(L1)

http-request allow if nagios
http-request allow if local_net auth_ok
http-request auth realm Gimme if local_net auth_ok
http-request deny

示例

acl auth_ok http_auth_group(L1) G1

http-request auth unless auth_ok

ignore-persist { if | unless } <condition>

可用于:frontend、listen、backend

声明一个忽略持久性的条件。

默认情况下,当启用cookie持久性,每个包含该cookie的请求都是无条件地持久的(假设目标服务器是启动运行的)。

“ignore-persist”语句允许我们声明各种基于ACL的条件,当条件匹配时,会导致一个请求忽略持久性。对于静态文件请求的负载均衡(通常不要求持久性)有时是有用的。也经常用于针对某个特定User-Agent完全禁用持久性(例如,一些网络抓取机器人)。

与“appsession”结合使用,也有助于减少HAProxy内存使用,因为如果持久性被忽略,appsession表就不会增大。


mode { tcp|http|health }

可用于:defaults、frontend、listen、backend

设置HAProxy实例的运行模式或协议。

参数 :

tcp 实例将以纯TCP模式工作。在客户端与服务器之间会建立一个全双工的连接,也不会执行7层的检查。默认模式,应用于SSL、SSH、SMTP、...

http 实例将以HTTP模式工作。在连接到任何服务器之前,客户端请求会经过深度的分析。任何非RFC兼容的请求都会被拒绝。7层过滤、处理和转换都可能发生。该模式也是HAProxy的最大价值所在。

health 实例将以“health”模式工作。对进入的连接仅回复“OK”并关闭连接。不会记录任何东西。该模式用于回复外部组件的健康监测。该模式已过时,不应该再使用,因为结合TCP或HTTP模式与“monitor”关键词可以实现同样的功能甚至做得更好。

monitor fail { if | unless } <condition>

可用于:frontend、listen

添加一个条件用于向一个监控HTTP请求报告一个失败。

示例 :

frontend www:
    mode http
    acl site_dead nbsrv(dynamic) lt 2
    acl site_dead nbsrv(static) lt 2
    monitor-uri /site_alive
    monitor fail if site_dead

monitor-net <source>

可用于:defaults、frontend、listen

声明一个仅可发送监控请求的源网络地址段。

参数

<source> 源IPv4地址或网络地址段,来自该地址或网络的请求仅能得到监控响应。可以是一个IPv4地址、主机名、或一个地址跟随一个斜杠('/')以及一个掩码。

Monitor requests are processed very early. It is not possible to block nor divert them using ACLs. They cannot be logged either, and it is the intended purpose. They are only used to report HAProxy’s health to an upper component, nothing more. Right now, it is not possible to set failure conditions on requests caught by “monitor-net”.

please note that only one “monitor-net” statement can be specified in a frontend. If more than one is found, only the last one will be considered.

示例

# address .252 and .253 are just probing us.
frontend www
    monitor-net 192.168.0.252/31

monitor-uri <uri>

可用于:defaults、frontend、listen

拦截一个外部组件监控请求的URI。

参数

<uri> 我们想要拦截的确切的URI,向其返回HAProxy的监控状态而不是转发该请求。

示例

# Use /haproxy_test to report haproxy's status
frontend www
    mode http
    monitor-uri /haproxy_test

option allbackups

no option allbackups

可用于:defaults、listen、backend

一次性使用所有备份服务器或仅使用第一个备份服务器。

默认情况下,当常规服务器都宕掉后,第一台可使用的备份服务器会接收所有流量。有时,可能一次性使用多台备份机器会更好,因为一台机器不够用啊。启用了“option allbackups”之后,当所有常规服务器都不可用时,就会在所有备份服务器上进行负载均衡。使用相同的负载均衡算法并考虑服务器的权重。因此,在备份服务器之间不再有任何优先顺序。

该选项多数时候用于静态服务器群,当一个应用完全下线时返回一个“抱歉”页面。

如果该选项在一个“defaults”部分启用了,则可以在某个具体实例中通过前置一个“no”关键词来禁用它。


option checkcache

no option checkcache

可用于:defaults、listen、backend

分析服务的所有响应并阻塞带有可缓存cookie的请求。


option clitcpka

no option clitcpka

可用于:defaults、frontend、listen

启用或禁用客户端TCP keepalive数据包的发送。


option contstats

可用于:defaults、frontend、listen

启用持续性的流量统计更新。

默认情况下,用于统计计算的计数器仅当一个会话结束时才增加。当服务小数据对象时,这样的工作效果相当不错,当对于大的数据对象(例如:大的图片或归档文件)或A/V流,haproxy计数器生成的统计图形看起来就像一个刺猬。启用该选项后,在整个会话期间计数器都会持续地增加。


option dontlog-normal

no option dontlog-normal

可用于:defaults、frontend、listen

启用或禁用对正常、成功连接的日志记录。

有些大型网站每秒要处理几千个连接,日志记录这些连接压力是很大的。其中有些网站被迫关闭日志记录,也就无法调试生产环境中的问题。设置该选项则haproxy不再日志记录正常的连接,即那些没有错误、没有超时、没有重试也没有重新分发的连接。这样硬盘空间不会出现异常情况。在HTTP模式中,会检测响应状态码,返回5xx状态码的连接还是会被日志记录的。

大多数时间强烈不推荐使用该选项,复杂问题的突破口往往是在常规的日志里,如果启用该选项,这些关键信息就不会被记录了。如果需要分离日志,可替代之使用“log-separate-errors”选项。


option dontlognull

no option dontlognull

可用于:defaults、frontend、listen

启用或禁用对空(null)连接的日志记录。


option forwardfor [ except <network> ] [ header <name> ] [ if-none ]

可用于:defaults、frontend、listen、backend

启用向发往服务器的请求中插入X-Forwarded-For请求头字段。

参数

<network> 可选参数用于为匹配<network>的来源禁用该选项。

<name> 可选参数指定一个不同的“X-Forwarded-For”头名称。

由于HAProxy工作在反向代理模式,服务器将它的IP地址当做请求的客户端地址。当期望服务器日志中有客户端IP地址时有时会令人烦恼。为了解决该问题,HAProxy可以向发往服务器的所有请求中添加众所周知的HTTP头部字段“X-Forwarded-For”。该头部字段的值代表客户端的IP地址。

示例

# Public HTTP address also used by stunnel on the same machine
frontend www
    mode http
    option forwardfor except 127.0.0.1  # stunnel already adds the header

# Those servers want the IP Address in X-Client
backend www
    mode http
    option forwardfor header X-Client

option httpchk

option httpchk <uri>

option httpchk <method> <uri>

option httpchk <method> <uri> <version>

可用于:defaults、listen、backend

启用HTTP检测服务器的健康状态。

参数 :

<method> 可选的HTTP方法用于请求。若未设置,则使用“OPTIONS”方法,因为它通常只需要简单的服务器处理,也易于从日志中过滤掉。可以使用任意方法,但不推荐发明使用非标准方法。

<uri> HTTP请求中引用的URI。默认为“/”,因为几乎任何服务器默认都可以访问,但也可以修改为任何其他URI,查询字符串(Query string)也是允许的。

<version> 可选的HTTP版本字符串。默认为“HTTP/1.0”,但某些服务器对于HTTP 1.0的行为可能不正确,所以切换成HTTP/1.1有时可能会有帮助。注意Host头部字段在HTTP/1.1中是必须的。

默认情况下,服务器健康检测仅是建立一个TCP连接。当指定了“option httpchk”,一旦建立了TCP连接,就会发送一个完整的HTTP请求,并认为2xx和3xx响应是有效的,而其他的响应状态码都表示服务器挂了,包括没有任何响应。

This option does not necessarily require an HTTP backend, it also works with plain TCP backends. This is particularly useful to check simple scripts bound to some dedicated ports using the inetd daemon.

示例

# Relay HTTPS traffic to Apache instance and check service availability
# Using HTTP request "OPTIONS * HTTP/1.1" on port 80
backend https_relay
    mode tcp
    option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www
    server apache1 192.168.1.1:443 check port 80

option httpclose

no option httpclose

可用于:defaults、frontend、listen、backend

启用或禁用被动HTTP连接关闭。


option httplog [ clf ]

可用于:defaults、frontend、listen、backend

启用日志记录HTTP请求、会话状态以及定时器(timers)。

参数

clf 若添加了“clf”参数,则输出格式为CLF格式而不是HAProxy的默认HTTP格式。当你需要将HAProxy的日志用于一个特定的日志分析器,而该分析器仅支持CLF格式且不可扩展,那就用这种格式。

默认情况下,日志输出格式非常简单,仅包含源地址和目的地址,以及实例名称。通过指定“option httplog”,每行日志就会转换为一种更加丰富的格式,包括但不限于:HTTP请求,连接定时器(connection timers),会话状态,连接数,捕获的数据包头部以及cookie,frontend、backend和服务器名称,当然还包括源地址和端口。

该选项可以在frontend或backend部分设置。

如果该选项在“defaults”部分启用,则可以在某个具体的实例中通过前置一个“no”关键词来禁用它。


option ldap-check

可用于:defaults、listen、backend

使用LDAPv3健康检测来测试服务器。

The server is considered valid only when the LDAP response contains success resultCode (http://tools.ietf.org/html/rfc4511#section-4.1.9).

示例

option ldap-check

option mysql-check [ user <username> ]

可用于:defaults、listen、backend

使用MySQL健康检测来测试服务器。

参数

<username> 用户名,用于连接到MySQL服务器。

option nolinger

no option nolinger

可用于:defaults、frontend、listen、backend

启用或禁用连接关闭后立即清空会话资源。


option redispatch

no option redispatch

可用于:defaults、listen、backend

启用或禁用连接失败之时重新分发会话。


option smtpchk

option smtpchk <hello> <domain>

可用于:defaults、listen、backend

使用SMTP健康检测来测试服务器。

参数 :

<hello> 可选参数。即使用“hello”命令。可以是“HELO”(SMTP)或“EHLO”(ESTMP)。任何其他的值都会被转换成默认命令(“HELO”)。

<domain> 发送给服务器的域名。若指定了hello命令才需指定(也必须指定)。默认使用“localhost”。

option splice-auto

no option splice-auto

可用于:defaults、frontend、listen、backend

启用或禁用对于套接字双向地自动内核加速。

在frontend或backend中启用该选项后,haproxy会自动评估使用内核TCP拼接在客户端与服务器端间任一方向转发数据的可能性。

注意:基于内核的TCP拼接是一个Linux特有的特性,首次出现在内核2.6.25中。为套接字之间的数据传输提供基于内核的加速,无需将这些数据拷贝到用户空间,因此性能得到显著提升,节省了大量CPU时钟周期。

示例 :

option splice-auto

option splice-request

no option splice-request

可用于:defaults、frontend、listen、backend

对请求启用或禁用套接字上的自动内核加速。


option splice-response

no option splice-response

可用于:defaults、frontend、listen、backend

对于响应启用或禁用套接字上的自动内核加速。


option srvtcpka

no option srvtcpka

可用于:defaults、listen、backend

启用或禁用服务器端发送TCP keepalive数据包。

当客户端与服务器之间有防火墙或其他会话相关的组件,当协议包含时间非常长的会话,并有较长的空闲期(如:远程桌面),那么中间组件可能会将长时间空闲的会话做过期处理。

启用套接字级别的TCP保活(keep-alives)使得系统定期发送数据包到连接的另一端,让其保持活跃。keep-alive探针(数据包)之间的延迟/间隔仅由系统控制,依赖于操作系统和调优参数。

关键要理解保活数据包不是在应用层面收发的。仅网络协议栈可以看到它们。因此,即使代理的一方已经使用保活数据包来维持连接的活跃,那些保活数据包不会被转发到代理的另一方。

注意这与HTTP keep-alive无关。

“srvtcpka”选项启用连接的服务器端发送TCP保活探针数据包,当HAProxy与服务器之间对于session过期比较敏感,该选项能提供帮助。


option ssl-hello-chk

可用于:defaults、listen、backend

使用SSLv3客户端hello健康检测来测试服务器。


option tcpka

可用于:defaults、frontend、listen、backend

启用或禁用在连接的两端发送TCP保活数据包的发送。


option tcplog

可用于:defaults、frontend、listen、backend

启用高级的日志记录TCP连接,带有会话状态和定时器。


option transparent

no option transparent

可用于:defaults、listen、backend

启用客户端透明代理。

为了向3层协议负载均衡器提供7层协议持久性而引入该选项。想法是利用操作系统的能力将一个刚进入的到远程地址的连接重定向到本地进程(这里是HAProxy),并让该进程知道原本请求的是什么地址。

When this option is used, sessions without cookies will be forwarded to the original destination IP address of the incoming request (which should match that of another equipment), while requests with cookies will still be forwarded to the appropriate server.


reqadd <string> [{if | unless} <cond>]

可用于:frontend、listen、backend

在HTTP请求的尾部添加一个头部字段。

参数 :

<string> 需要添加的完整的头部字段行。任何空格或常用定界符都必须使用反斜杠(‘\’)进行转义。

<cond> 可选,ACL构建的匹配条件。

Header transformations only apply to traffic which passes through HAProxy, and not to traffic generated by HAProxy, such as health-checks or error responses.

示例

# Add "X-Proto:SSL" to requests coming via port 81
acl is-ssl  dst_port    81
reqadd  X-Proto:\ SSL   if is-ssl

retries <value>

可用于:defaults、listen、backend

设置对某个服务器连接失败后重试的次数。

When “option redispatch” is set, the last retry may be performed on another server even if a cookie references a different server.


stats admin { if | unless } <cond>

可用于:listen、backend

启用统计管理级别,如果/除非达到某个条件。

管理级别允许通过web界面启用/禁用后端服务器。默认情况下,统计页面考虑到安全问题是只读的。

Currently, the POST request is limited to the buffer size minus the reserved buffer space, which means that if the list of servers is too long, the request won’t be processed. It is recommended to alter few servers at a time.

示例

# statistics admin level only for localhost
backend stats_localhost
    stats enable
    stats admin if LOCALHOST
# statistics admin level always enabled because of the authentication
backend stats_auth
    stats enable
    stats auth admin:AdMiN123
    stats admin if TRUE
# statistics admin level depends on the authenticated user
userlist stats-auth
    group admin users admin
    user admin insecure-password AdMiN123
    group readonly users haproxy
    user haproxy insecure-password haproxy

backend stats_auth
    stats enable
    acl AUTH        http_auth(stats-auth)
    acl AUTH_ADMIN  http_auth_group(stats-auth) admin
    stats http-request auth unless AUTH
    stats admin if AUTH_ADMIN

stats auth <user>:<passwd>

可用于:defaults、listen、backend

启用带身份认证的统计信息页面,并赋予某账户访问权限。

示例

# public access (limited to this backend only)
backend public_www
    server srv1 192.168.0.1:80
    stats enable
    stats hide-version
    stats scope   .
    stats uri     /admin?stats
    stats realm   Haproxy\ Statistics
    stats auth    admin1:AdMiN123
    stats auth    admin2:AdMiN321

# internal monitoring access (unlimited)
backend private_monitoring
    stats enable
    stats uri     /admin?stats
    stats refresh 5s

stats enable

可用于:defaults、listen、backend

以默认设置启用统计报告。

This statement enables statistics reporting with default settings defined
at build time. Unless stated otherwise, these settings are used :

  • stats uri : /haproxy?stats
  • stats realm : “HAProxy Statistics”
  • stats auth : no authentication
  • stats scope : no restriction

stats hide-version

可用于:defaults、listen、backend

启用统计并隐藏HAProxy版本报告。


stats http-request { allow | deny | auth [realm <realm>] } [ { if | unless } <condition> ]

可用于:listen、backend

统计(页面)访问控制。

As “http-request”, these set of options allow to fine control access to statistics. Each option may be followed by if/unless and acl. First option with matched condition (or option without condition) is final. For “deny” a 403 error will be returned, for “allow” normal processing is performed, for “auth” a 401/407 error code is returned so the client should be asked to enter a username and password.

There is no fixed limit to the number of http-request statements per instance.


stats scope { <name> | "." }

可用于:defaults、listen、backend

启用统计并限制访问范围。

参数

<name> 被报告的一个listen、frontend或backend部分的名称。特殊名称“.”指派该语句出现的部分。

When this statement is specified, only the sections enumerated with this statement will appear in the report. All other ones will be hidden. This statement may appear as many times as needed if multiple sections need to be reported. Please note that the name checking is performed as simple string comparisons, and that it is never checked that a give section name really exists.


stats show-desc [ <description> ]

可用于:defaults、listen、backend

在统计页面上显示一段描述语句。


stats show-legends

Enable reporting additional informations on the statistics page :

  • cap: capabilities (proxy)
  • mode: one of tcp, http or health (proxy)
  • id: SNMP ID (proxy, socket, server)
  • IP (socket, server)
  • cookie (backend, server)

tcp-request content accept [{if | unless} <condition>]

可用于:frontend、listen

如果/除非满足一个内容检查条件,则接受一个连接。

Note that the “if/unless” condition is optional. If no condition is set on the action, it is simply performed unconditionally.

If no “tcp-request content” rules are matched, the default action already is “accept”. Thus, this statement alone does not bring anything without another “reject” statement.


tcp-request content reject [{if | unless} <condition>]


timeout http-keep-alive <timeout>

可用于:defaults、frontend、listen、backend

设置等待出现一个新的HTTP请求的最大允许时间。

By default, the time to wait for a new request in case of keep-alive is set by “timeout http-request”. However this is not always convenient because some people want very short keep-alive timeouts in order to release connections
faster, and others prefer to have larger ones but still have short timeouts once the request has started to present itself.

The “http-keep-alive” timeout covers these needs. It will define how long to wait for a new HTTP request to start coming after a response was sent. Once the first byte of request has been seen, the “http-request” timeout is used
to wait for the complete request to come. Note that empty lines prior to a new request do not refresh the timeout and are not counted as a new request.

There is also another difference between the two timeouts : when a connection expires during timeout http-keep-alive, no error is returned, the connection just closes. If the connection expires in “http-request” while waiting for a
connection to complete, a HTTP 408 error is returned.

In general it is optimal to set this value to a few tens to hundreds of milliseconds, to allow users to fetch all objects of a page at once but without waiting for further clicks. Also, if set to a very small value (eg: 1 millisecond) it will probably only accept pipelined requests but not the non-pipelined ones. It may be a nice trade-off for very large sites running with tens to hundreds of thousands of clients.


timeout http-request <timeout>

可用于:defaults、frontend、listen、backend

设置等待一个完整的HTTP请求的最大允许时间。


use_backend <backend> if <condition>

use_backend <backend> unless <condition>

可用于:frontend、listen

若满足某个基于ACL的条件,则切换到某个特定的backend。

5. server和default-server选项

server <name> <address>[:port] [settings ...]

default-server [settings ...]

backup

当“backup”出现在server行中,则该服务器在负载均衡中仅当所有其他非备份服务器都不可用时才会被使用。


weight <weight>

“weight”参数用于调整服务器相对于其他服务器的权重。所有服务器根据它们的权重比上所有权重之和得到一个负载比例,因此权重越高,负载越高。默认权重是1,权重最大值是256。零值意味着该服务器将不参与负载均衡,但仍接受持久连接。如果该参数用于根据服务器的能力分布负载,推荐开始设置为一个可以伸缩的值,例如10到100之间的某个值,为之后向上向下调整留出足够的空间。

使用访问控制列表(ACL)和模式抽取

访问控制列表(ACL)的使用为执行内容转换以及基于从请求、响应或任何环境状态中抽取的内容作决策提供了一种灵活的解决方案。

原理很简单:

  • 使用一些值来定义测试条件

  • 仅当一组测试通过才执行动作

动作通常包括阻塞请求,或选择一个backend。

为了定义一个测试,需使用“acl”关键词。语法为:

acl <aclname> <criterion> [flags] [operator] <value> ...

示例

# Match any negative Content-Length header
acl negative-length hdr_val(content-length) lt 0
# Match SSL versions between 3.0 and 3.1(inclusive)
acl sslv3 req_ssl_ver 3:3.1
# To select a different backend for requests to static contents 
# on the "www" site and to every request on the "img", "video",
# "download" and "ftp" hosts
acl url_static  path_beg             /static /imags/ /img /css
acl url_static  path_end             .gif .png .jpg .css .js
acl host_www    hdr_beg(host) -i www
acl host_static hdr_beg(host) -i img. video. download. ftp.

# now use backend "static" for all static-only hosts, and for static urls
# of host "www". Use backend "www" for the rest.
use_backend     static  if host_static or host_www url_static
use_backend     www     if host_www

Redis用于数据类型 及 应用场景

字符串(String)

应用场景

String是最常用的一种数据类型,普通的key/ value 存储都可以归为此类.即可以完全实现目前 Memcached 的功能,并且效率更高。还可以享受Redis的定时持久化,操作日志及 Replication等功能。除了提供与 Memcached 一样的get、set、incr、decr 等操作外,Redis还提供了下面一些操作:

  • 获取字符串长度
  • 往字符串append内容
  • 设置和获取字符串的某一段内容
  • 设置及获取字符串的某一位(bit)
  • 批量设置一系列字符串的内容

应用举例
统计网站访问数量、当前在线人数、微博数、粉丝数等,incr命令(++操作)

实现方式

String在redis内部存储默认就是一个字符串,被redisObject所引用,当遇到incr,decr等操作时会转成数值型进行计算,此时redisObject的encoding字段为int。

常用接口

接口 介绍
SET key value 设置指定 key 的值
GET key 获取指定 key 的值。
GETRANGE key start end 返回 key 中字符串值的子字符
GETSET key value 将给定 key 的值设为 value ,并返回 key 的旧值(old value)。
GETBIT key offset 对 key 所储存的字符串值,获取指定偏移量上的位(bit)。
MGET key1 [key2..] 获取所有(一个或多个)给定 key 的值。
SETBIT key offset value 对 key 所储存的字符串值,设置或清除指定偏移量上的位(bit)。
SETEX key seconds value 将值 value 关联到 key ,并将 key 的过期时间设为 seconds (以秒为单位)。
SETNX key value 只有在 key 不存在时设置 key 的值。
SETRANGE key offset value 用 value 参数覆写给定 key 所储存的字符串值,从偏移量 offset 开始。
STRLEN key 返回 key 所储存的字符串值的长度。
MSET key value [key value …] 同时设置一个或多个 key-value 对。
MSETNX key value [key value …] 同时设置一个或多个 key-value 对,当且仅当所有给定 key 都不存在。
PSETEX key milliseconds value 这个命令和 SETEX 命令相似,但它以毫秒为单位设置 key 的生存时间,而不是像 SETEX 命令那样,以秒为单位。
INCR key 将 key 中储存的数字值增一。
INCRBY key increment 将 key 所储存的值加上给定的增量值(increment) 。
INCRBYFLOAT key increment 将 key 所储存的值加上给定的浮点增量值(increment) 。
DECR key 将 key 中储存的数字值减一。
DECRBY key decrement key 所储存的值减去给定的减量值(decrement) 。
APPEND key value 如果 key 已经存在并且是一个字符串, APPEND 命令将 value 追加到 key 原来的值的末尾。

列表(List)

应用场景

Lists 就是链表,可以通过push和pop操作从列表的头部或者尾部添加或者删除元素,这样List即可以作为栈,也可以作为队列。

使用List结构,可以轻松地实现:
1、最新消息排行榜。
2、消息队列,以完成多程序之间的消息交换。可以利用List的PUSH将任务存在List中(生产者),然后工作线程再用POP将任务取出(消费者)。
3、Redis还提供了操作List中某一段的api,可以直接查询,删除List中某一段的元素。

应用举例
1.微博关注列表、粉丝列表。
2.获取最新n条微博(缓存主页或第一个评论页,不用去DB查询)

实现方式

Redis list的实现为一个双向链表,Redis内部的很多实现,包括发送缓冲队列等也都是用的这个数据结构。

常用接口

接口 介绍
BLPOP key1 [key2 ] timeout 移出并获取列表的第一个元素, 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止。
BRPOP key1 [key2 ] timeout 移出并获取列表的最后一个元素, 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止。
BRPOPLPUSH source destination timeout 从列表中弹出一个值,将弹出的元素插入到另外一个列表中并返回它; 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止。
LINDEX key index 通过索引获取列表中的元素
LINSERT key BEFORE AFTER pivot value
LLEN key 获取列表长度
LPOP key 移出并获取列表的第一个元素
LPUSH key value1 [value2] 将一个或多个值插入到列表头部
LPUSHX key value 将一个或多个值插入到已存在的列表头部
LRANGE key start stop 获取列表指定范围内的元素
LREM key count value 移除列表元素
LSET key index value 通过索引设置列表元素的值
LTRIM key start stop 对一个列表进行修剪(trim),就是说,让列表只保留指定区间内的元素,不在指定区间之内的元素都将被删除。
RPOP key 移除并获取列表最后一个元素
RPOPLPUSH source destination 移除列表的最后一个元素,并将该元素添加到另一个列表并返回
RPUSH key value1 [value2] 在列表中添加一个或多个值
RPUSHX key value 为已存在的列表添加值

哈希(Hash)

应用场景

hash特别适合存储对象。

  • 1、相对于将对象的每个字段存成单个string 类型。一个对象存储在hash类型中会占用更少的内存,并且可以更方便的存取整个对象。
  • 2、相对于采用String类型的存储对象,需要将对象进行序列化。增加了序列化/反序列化的开销,并且在需要修改其中一项信息时,需要把整个对象取回。而Redis的Hash提供了直接存取这个Map成员的接口。

应用举例
例如存储、读取、修改用户属性(name,age,pwd等)

实现方式

redis的Hash数据类型的value内部是一个HashMap,如果该Map的成员比较少,则会采用一维数组的方式来紧凑存储该Map,省去了大量指针的内存开销,这个参数在redis.conf配置文件中下面2项。

Hash-max-zipmap-entries 64
Hash-max-zipmap-value 512

Hash-max-zipmap-entries含义:是当value这个Map内部不超过64个成员时会采用线性紧凑格式存储(默认是64),即value内部有64个以下的成员就是使用线性紧凑存储,超过该值自动转成真正的HashMap。
Hash-max-zipmap-value含义:是当value这个Map内部的每个成员值长度不超过多少字节就会采用线性紧凑存储来节省空间。

以上两个条件任意一个条件超过设置值都会转成真正的HashMap,也就不会再节省内存了,这个值设置多少需要权衡,HashMap的优势就是查找和操作效率高。

常用接口

接口 介绍
HDEL key field2 [field2] 删除一个或多个哈希表字段
HEXISTS key field 查看哈希表 key 中,指定的字段是否存在。
HGET key field 获取存储在哈希表中指定字段的值。
HGETALL key 获取在哈希表中指定 key 的所有字段和值。
HINCRBY key field increment 为哈希表 key 中的指定字段的整数值加上增量 increment 。
HINCRBYFLOAT key field increment 为哈希表 key 中的指定字段的浮点数值加上增量 increment 。
HKEYS key 获取所有哈希表中的字段
HLEN key 获取哈希表中字段的数量
HMGET key field1 [field2] 获取所有给定字段的值
HMSET key field1 value1 [field2 value2 ] 同时将多个 field-value (域-值)对设置到哈希表 key 中。
HSET key field value 将哈希表 key 中的字段 field 的值设为 value 。
HSETNX key field value 只有在字段 field 不存在时,设置哈希表字段的值。
HVALS key 获取哈希表中所有值
HSCAN key cursor [MATCH pattern] [COUNT count] 迭代哈希表中的键值对。

注意:Redis提供了接口(HGETALL )可以直接取到全部的属性数据,但是如果内部Map的成员很多,那么涉及到遍历整个内部Map的操作,由于Redis单线程模型的缘故,这个遍历操作可能会比较耗时,而导致其它客户端的请求完全不响应。

集合(Set)

应用场景

Redis Set对外提供的功能与List类似是一个列表的功能,特殊之处在于Set是自动去重的,Set中的元素是没有顺序的。Set提供了判断某个成员是否在一个Set集合内的接口,这个也是List没有的。

Set集合的概念就是一堆不重复值的组合。比如在微博应用中,可以将一个用户所有的关注人存在一个集合中,将其所有粉丝存在一个集合。Redis还为集合提供了求交集、并集、差集等操作,可以非常方便的实现如共同关注、共同喜好、二度好友等功能,对上面的所有集合操作,你还可以使用不同的命令选择将结果返回给客户端还是存集到一个新的集合中。

应用举例
1.利用交集求共同好友。
2.利用唯一性,可以统计访问网站的所有独立IP。
3.好友推荐的时候根据tag求交集,大于某个threshold(临界值的)就可以推荐。

实现方式

set 的内部实现是一个value永远为null的HashMap,实际就是通过计算hash的方式来快速去重的,这也是set能提供判断一个成员是否在集合内的原因。

常用接口

接口 介绍
SADD key member1 [member2] 向集合添加一个或多个成员
SCARD key 获取集合的成员数
SDIFF key1 [key2] 返回给定所有集合的差集
SDIFFSTORE destination key1 [key2] 返回给定所有集合的差集并存储在 destination 中
SINTER key1 [key2] 返回给定所有集合的交集
SINTERSTORE destination key1 [key2] 返回给定所有集合的交集并存储在 destination 中
SISMEMBER key member 判断 member 元素是否是集合 key 的成员
SMEMBERS key 返回集合中的所有成员
SMOVE source destination member 将 member 元素从 source 集合移动到 destination 集合
SPOP key 移除并返回集合中的一个随机元素
SRANDMEMBER key [count] 返回集合中一个或多个随机数
SREM key member1 [member2] 移除集合中一个或多个成员
SUNION key1 [key2] 返回所有给定集合的并集
SUNIONSTORE destination key1 [key2] 所有给定集合的并集存储在 destination 集合中
SSCAN key cursor [MATCH pattern] [COUNT count] 迭代集合中的元素

有序集合(Sorted Set)

使用场景

Redis Sorted Set的使用场景与set类似,区别是Set是无序的,而Sorted Set可以通过用户额外提供一个优先级(score)的参数来为成员排序,并且是插入有序的,即自动排序。当你需要一个有序的并且不重复的集合列表,那么可以选择Sorted Set数据结构,比如微博的timeline(时间线)可以以发表时间作为score来存储,这样获取时就是自动按时间排好序的。

应用举例
1.timeline(时间线);
2.可以用于一个大型在线游戏的积分排行榜,每当玩家的分数发生变化时,可以执行zadd更新玩家分数(score),此后在通过zrange获取几分top ten的用户信息;
3.用Sorted Set来做带权重的队列,比如普通消息的score为1,重要消息的score为2,然后工作线程可以选择按score的倒序来获取工作任务。让重要的任务优先执行。

实现方式

Redis Sorted Set的内部使用HashMap和跳跃表(SkipList)来保证数据的存储和有序,HashMap里放的是成员到score的映射,而跳跃表里存放的是所有的成员,排序依据是HashMap里存的score,使用跳跃表的结构可以获得比较高的查找效率,并且在实现上比较简单。

常用接口

接口 说明
ZADD key score1 member1 [score2 member2] 向有序集合添加一个或多个成员,或者更新已存在成员的分数
ZCARD key 获取有序集合的成员数
ZCOUNT key min max 计算在有序集合中指定区间分数的成员数
ZINCRBY key increment member 有序集合中对指定成员的分数加上增量 increment
ZINTERSTORE destination numkeys key [key …] 计算给定的一个或多个有序集的交集并将结果集存储在新的有序集合 key 中
ZLEXCOUNT key min max 在有序集合中计算指定字典区间内成员数量
ZRANGE key start stop [WITHSCORES] 通过索引区间返回有序集合成指定区间内的成员
ZRANGEBYLEX key min max [LIMIT offset count] 通过字典区间返回有序集合的成员
ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT] 通过分数返回有序集合指定区间内的成员
ZRANK key member 返回有序集合中指定成员的索引
ZREM key member [member …] 移除有序集合中的一个或多个成员
ZREMRANGEBYLEX key min max 移除有序集合中给定的字典区间的所有成员
ZREMRANGEBYRANK key start stop 移除有序集合中给定的排名区间的所有成员
ZREMRANGEBYSCORE key min max 移除有序集合中给定的分数区间的所有成员
ZREVRANGE key start stop [WITHSCORES] 返回有序集中指定区间内的成员,通过索引,分数从高到底
ZREVRANGEBYSCORE key max min [WITHSCORES] 返回有序集中指定分数区间内的成员,分数从高到低排序
ZREVRANK key member 返回有序集合中指定成员的排名,有序集成员按分数值递减(从大到小)排序
ZSCORE key member 返回有序集中,成员的分数值
ZUNIONSTORE destination numkeys key [key …] 计算给定的一个或多个有序集的并集,并存储在新的 key 中
ZSCAN key cursor [MATCH pattern] [COUNT count] 迭代有序集合中的元素(包括元素成员和元素分值)