nginx 配置实践

创建:xiaozi · 最后修改:xiaozi 2019-07-27 15:28 ·

背景介绍

tool.lu 上线后的这6年多,对nginx这块的配置一直都在跟进和改进;本文将会介绍在实践中遇到的问题和解决办法。

编译参数

依赖的第三方模块

模块 描述 下载地址

nginx-http-concat

阿里出品的css,js合并插件

https://github.com/alibaba/nginx-http-concat

lua-nginx-module

openresty的核心模块

https://github.com/openresty/lua-nginx-module

ngx_devel_kit

openresty需要依赖这个模块

https://github.com/simplresty/ngx_devel_kit

ngx_brotli

Google出品的brotli压缩算法

https://github.com/google/ngx_brotli

编译命令

./configure \
--prefix=/usr/local/nginx \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/lock/nginx.lock \
--user=nobody \
--group=nobody \
--error-log-path=/data/log/nginx/error.log \
--http-log-path=/data/log/nginx/access.log \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_realip_module \
--with-http_gzip_static_module \
--with-http_stub_status_module \
--with-pcre \
--with-file-aio \
--add-module=/usr/local/src/nginx-http-concat \
--add-module=/usr/local/src/lua-nginx-module \
--add-module=/usr/local/src/ngx_devel_kit \
--add-module=/usr/local/src/ngx_brotli \
--with-http_image_filter_module \
--with-openssl=/usr/local/src/openssl

基本配置

http到https的跳转

server {
    listen 80;
    server_name www.tool.lu tool.lu;

    access_log off;
    error_log /dev/null crit;

    return 301 https://tool.lu$request_uri;
}

www到根域名的跳转

server {
    listen 443 ssl http2;
    server_name www.tool.lu;

    ssl_certificate vhosts/***.crt;
    ssl_certificate_key vhosts/***.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4:!DH:!DHE;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:50m;
    ssl_session_timeout 1d;

    access_log off;
    error_log /dev/null crit;

    return 301 https://tool.lu$request_uri;
}

更高效压缩算法的配置

http {
    # ...
    brotli on;
    brotli_comp_level 6;
    brotli_buffers 16 8k;
    brotli_min_length 20;
    brotli_types *;
    # ...
}

server主体配置

upstream php_sever {
    server 127.0.0.1:9000 weight=10;
}

server {
    listen 443 ssl http2 reuseport; # fastopen=3
    server_name tool.lu;

    ssl_certificate ***.crt;
    ssl_certificate_key ***.key;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4:!DH:!DHE;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:50m;
    ssl_session_timeout 1d;

    root /path/to/public/directory;
    index index.html index.php;

    fastcgi_intercept_errors on;
    error_page 404 /404.html;
    error_page 429 /429.html;
    # error_page 500 502 503 504 /500.html;

    access_log /data/log/nginx/tool.lu.access.log;
    error_log /data/log/nginx/tool.lu.error.log;

    add_header Content-Security-Policy upgrade-insecure-requests;
    add_header Content-Security-Policy-Report-Only "default-src *; script-src 'self' 'unsafe-inline' 'unsafe-eval' *.tool.lu *.baidu.com; object-src 'none'; style-src 'self' 'unsafe-inline' 'unsafe-eval' *.tool.lu *.baidu.com; img-src 'self' data: *.tool.lu *.href.lu *.baidu.com; media-src 'none'; child-src 'self' *.tool.lu; font-src *.tool.lu *.alicdn.com; connect-src 'self' *.tool.lu *.baidu.com *.alicdn.com; report-uri //analytics.tool.lu/csp";

    # add_header X-Frame-Options SAMEORIGIN; # coderunner embed cannot
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    # HSTS
    add_header Strict-Transport-Security max-age=15768000;

    add_header X-UA-Compatible "IE=Edge";
    add_header X-Request-Id $request_id;
    add_header X-Request-Start $msec;
    add_header X-Backend-Server $hostname;

    if ($request_method !~ ^(GET|HEAD|POST|OPTIONS)$ ) {
        return 405;
    }

    access_by_lua_file script/detect_blacklist.lua;

    location / {
        try_files $uri $uri/ /index.php$is_args$query_string;
    }

    location ~ \.php$ {
        try_files $uri =404;

        include fastcgi_params;
        fastcgi_pass php_sever;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        fastcgi_param DOCUMENT_ROOT $realpath_root;
        # fastcgi_param HTTP_HOST $host
    }

    location ~ /\.(ht|git|svn) {
        deny all;
    }

    location ~ ^/status$ {
        access_log off;
        allow 127.0.0.0/24;
        deny all;
        include fastcgi_params;
        fastcgi_pass php_sever;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        fastcgi_param DOCUMENT_ROOT $realpath_root;
    }

    location ~ ^/nginx_status$ {
        stub_status on;
    }

    # ...

}

高级配置

nginx进程模型

nginx: master process
 \_ nginx: worker process
 \_ nginx: worker process
 \_ nginx: worker process
 \_ nginx: worker process

从网上找了一张open resty生命周期的图临摹了一下

173620RLhqG3c0b8RdBYt8

IP黑名单

实现

1809212tOXZDKPAYXGXqKr

1、进程启动的时候由一个进程load redis中的黑名单数据到nginx的共享变量中

2、redis中的数据可以由web控制台修改,挑选worker 0作为定时任务的执行者,定时将redis中的最新数据同步到nginx的共享变量中

3、nginx每次接收到访问请求的时候,判断是否攻击请求,如果命中将ip写入nginx的共享变量和redis中

local redis_host = "127.0.0.1"
local redis_port = 6379

local redis_connection_timeout = 100
local redis_key = "security:ip:blacklist"

local ip_blacklist = ngx.shared.ip_blacklist

local redis = require "resty.redis"
local delay = 1 * 3600
local handler

handler = function()
	ngx.log(ngx.ERR, "run handler...")
	local red = redis:new()
	red:set_timeout(redis_connection_timeout)

	local ok, err = red:connect(redis_host, redis_port)
	if not ok then
		ngx.log(ngx.ERR, "Redis connection error" .. err)
	else
		-- local new_ip_blacklist, err = red:smembers(redis_key)
		local batch = 100
		local lastCursor = "0"
		repeat
			local res, err = red:sscan(redis_key, lastCursor, "count", batch)
			if err then
				ngx.log(ngx.ERR, "Redis read error" .. err)
				break
			end
			local cursor, values, err = unpack(res)
			for index, banned_ip in ipairs(values) do
				ip_blacklist:set(banned_ip, true)
			end
			lastCursor = cursor
			ngx.log(ngx.ERR, "sscan cursor:", cursor)
		until cursor == "0"
	end
	red:close()
end

if 0 == ngx.worker.id() then
	local ok, err = ngx.timer.at(0, handler)
	if not ok then
		 ngx.log(ngx.ERR, "failed to create the timer: ", err)
	end

	local ok, err = ngx.timer.every(delay, handler)
	if not ok then
		 ngx.log(ngx.DEBUG, "failed to create the timer: ", err)
	end
end
问题和解决办法

1、redis set 慢慢变大,达到9w+, nginx lua一开始使用了smembers,引起了redis慢查询; 使用 sscan 解决了 redis 慢查询的问题

2、原来nginx每个进程都会起一个timer同步redis中的数据,这样就会比较浪费; 使用 ngx.worker.id() 获取 worker 的id 进行绑定(进程kill之后,master会再起一个进程,这个进程会继承原进程的id)


浏览 25549 次

首页
Copyright © 2011-2021 iteam. Current version is 2.78.0. UTC+08:00, 2021-10-19 20:15
浙ICP备14020137号-1 $访客地图$