Nginx+Memcached+Tomcat 搭载负责均衡实现session共享

一环境：

windows 7-x64

tomcat版本：apache-tomcat-6.0.36-windows-x64

 

jar包下载地址：http://code.google.com/p/memcached-session-manager/downloads/list

本人使用：javolution-5.4.3.1；memcached-session-manager-1.6.1；memcached-session-manager-tc6-1.6.1；msm-javolution-serializer-1.6.1；spymemcached-2.7.3

 

项目路径：

nginx，memcached，tomcat0，tomcat1（两份tomcat）都放在：E:\nginx+memcached+tomcat目录下，

测试项目：E:\nginx+memcached+tomcat\test0 和 E:\nginx+memcached+tomcat\test1

 

二 tomcat配置：

 

copy两份tomcat，分别放到tomcat0,、tomcat1目录下，分别命名tomcat_0、tomcat_1:

 

配置tomcat0：

配置第一个$TOMCAT/conf下的server.xml文件

修改以下4处配置项：

1.

 2.

 3.


 4.

 

上图中<Context docBase="E:\nginx+memcached+tomcat\test0" path=""></Context>的docBase=“”是tomcat部署测试项目路径

以下是本人tomcat0的server.xml的配置后的文件

<?xml version='1.0' encoding='utf-8'?>
<!--
  Licensed to the Apache Software Foundation (ASF) under one or more
  contributor license agreements.  See the NOTICE file distributed with
  this work for additional information regarding copyright ownership.
  The ASF licenses this file to You under the Apache License, Version 2.0
  (the "License"); you may not use this file except in compliance with
  the License.  You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<!-- Note:  A "Server" is not itself a "Container", so you may not
     define subcomponents such as "Valves" at this level.
     Documentation at /docs/config/server.html
 -->
<Server port="8006" shutdown="SHUTDOWN">

  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
  <!--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -->
  <Listener className="org.apache.catalina.core.JasperListener" />
  <!-- Prevent memory leaks due to use of particular java/javax APIs-->
  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" />
  <!-- JMX Support for the Tomcat server. Documentation at /docs/non-existent.html -->
  <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" />

  <!-- Global JNDI resources
       Documentation at /docs/jndi-resources-howto.html
  -->
  <GlobalNamingResources>
    <!-- Editable user database that can also be used by
         UserDatabaseRealm to authenticate users
    -->
    <Resource name="UserDatabase" auth="Container"
              type="org.apache.catalina.UserDatabase"
              description="User database that can be updated and saved"
              factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
              pathname="conf/tomcat-users.xml" />
  </GlobalNamingResources>

  <!-- A "Service" is a collection of one or more "Connectors" that share
       a single "Container" Note:  A "Service" is not itself a "Container", 
       so you may not define subcomponents such as "Valves" at this level.
       Documentation at /docs/config/service.html
   -->
  <Service name="Catalina">
  
    <!--The connectors can use a shared executor, you can define one or more named thread pools-->
    <!--
    <Executor name="tomcatThreadPool" namePrefix="catalina-exec-" 
        maxThreads="150" minSpareThreads="4"/>
    -->
    
    
    <!-- A "Connector" represents an endpoint by which requests are received
         and responses are returned. Documentation at :
         Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
         Java AJP  Connector: /docs/config/ajp.html
         APR (HTTP/AJP) Connector: /docs/apr.html
         Define a non-SSL HTTP/1.1 Connector on port 8080
    -->
    <Connector port="8088" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    <!-- A "Connector" using the shared thread pool-->
    <!--
    <Connector executor="tomcatThreadPool"
               port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />
    -->           
    <!-- Define a SSL HTTP/1.1 Connector on port 8443
         This connector uses the JSSE configuration, when using APR, the 
         connector should be using the OpenSSL style configuration
         described in the APR documentation -->
    <!--
    <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
               maxThreads="150" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS" />
    -->

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8010" protocol="AJP/1.3" redirectPort="8443" />


    <!-- An Engine represents the entry point (within Catalina) that processes
         every request.  The Engine implementation for Tomcat stand alone
         analyzes the HTTP headers included with the request, and passes them
         on to the appropriate Host (virtual host).
         Documentation at /docs/config/engine.html -->

    <!-- You should set jvmRoute to support load-balancing via AJP ie :
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">         
    --> 
    <Engine name="Catalina" jvmRoute="tomcat0" defaultHost="localhost">

      <!--For clustering, please take a look at documentation at:
          /docs/cluster-howto.html  (simple how to)
          /docs/config/cluster.html (reference documentation) -->
      <!--
      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
      -->        

      <!-- The request dumper valve dumps useful debugging information about
           the request and response data received and sent by Tomcat.
           Documentation at: /docs/config/valve.html -->
      <!--
      <Valve className="org.apache.catalina.valves.RequestDumperValve"/>
      -->

      <!-- This Realm uses the UserDatabase configured in the global JNDI
           resources under the key "UserDatabase".  Any edits
           that are performed against this UserDatabase are immediately
           available for use by the Realm.  -->
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase"/>

      <!-- Define the default virtual host
           Note: XML Schema validation will not work with Xerces 2.2.
       -->
      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true"
            xmlValidation="false" xmlNamespaceAware="false">
            <Context docBase="E:\nginx+memcached+tomcat\test0" path="">
			</Context>
        <!-- SingleSignOn valve, share authentication between web applications
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
        -->

        <!-- Access log processes all example.
             Documentation at: /docs/config/valve.html -->
        <!--
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
               prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
        -->

      </Host>
     
    </Engine>
  </Service>
</Server>

 

 配置tomcat1  

修改$TOMCAT/conf下的server.xml文件和$TOMCAT/conf下的context.xml文件。

同第一个配置方法相同，只是将图中圈出的部分修改为不同的值即可，注意：tomcat1中的修改路径，即：<Context docBase="E:\nginx+memcached+tomcat\test1" path=""></Context>

三.安装&配置memcached

安装：

将memcached-win61.zip解压到到E:\nginx+memcached+tomcat\下面，

以管理员身份启动终端，进入到E:\nginx+memcached+tomcat\\memcached-win64

执行安装命令：memcached.exe –d install

启动命令：memcached -m 512 -d start，默认端口号为11211

配置：

1.添加jar包

将javolution-5.4.3.1；memcached-session-manager-1.6.1；memcached-session-manager-tc6-1.6.1；msm-javolution-serializer-1.6.1；spymemcached-2.7.3 这些jar包分别复制到tomcat0、tomcat1下的$TOMCAT/lib目录下。

2.配置文件

然后修改$TOMCAT/conf下的context.xml

在<Context>标签下添加如下配置：注：如果有多个memcached，memcachedNodes=""的值用空格将多个节点分开。如：memcachedNodes="n1:localhost:11211 n2:localhost:11212 n3 localhost:11213"

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
		memcachedNodes="n1:localhost:11211"
		requestUriIgnorePattern=".*/.(png|gif|jpg|css|js)$" 
		sessionBackupAsync="false" 
		sessionBackupTimeout="100"
		transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory"
		copyCollectionsForSerialization="false">
</Manager>

四.安装&配置Nginx

安装：

将nginx-1.4.0.zip解压到E:\nginx+memcached+tomcat\下，

以管理员身份启动终端，进入到E:\nginx+memcached+tomcat\\nginx-1.4.0执行命令；

常用命令：

启动：start nginx.exe

停止：nginx.exe -s stop

检验错误：nginx.exe -t

重启：nginx.exe -s reload

配置：

修改$NGINX_HOME/conf/nginx.conf文件：

以下是本人修改后的文件：

#Nginx用户和组，windows下不指定
#user  nobody

#工作的子进程(通常等于CPU数量或者1倍于CPU)  
worker_processes  4;

#错误日志存放路径
#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#指定pid存放文件
#pid        logs/nginx.pid;

events {
    #允许最大连接数
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    #定义日志格式
    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                '$status $body_bytes_sent "$http_referer" '
    #                '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay on;

    #客户端上传文件大小控制  
    client_max_body_size 8m; 

    #开启gzip压缩
    #gzip  on;

    #负载均衡服务器列表
    upstream www.test.com {
	server 127.0.0.1:8080;#可以加权重-->weight=1
	server 127.0.0.1:8088;

	#同一机器在多网情况下，路由切换，ip可能不同
	#ip_hash;
    }

    server {
        listen       80;
	#主机名称
	server_name 127.0.0.1;

	#字符集设置
        #charset koi8-r;

	#访问日志存放路径
        #access_log  logs/host.access.log  main;

        location / {
	    #html文件存放路径
            root   html;
            index  index.jsp index.html index.htm;

	    #此处的 http://www.test.com与upstream www.test.com对应
	    proxy_pass http://www.test.com;

	    proxy_redirect              off;
            proxy_set_header            Host $host;
            proxy_set_header            X-Real-IP $remote_addr;
            proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;
            client_max_body_size        10m;
            client_body_buffer_size     128k;
            proxy_connect_timeout       90;
            proxy_send_timeout          90;
            proxy_read_timeout          90;
            proxy_buffer_size           4k;
            proxy_buffers               4 32k;
            proxy_busy_buffers_size     64k;
            proxy_temp_file_write_size  64k; 

        }

        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}
	location ~ .*.jsp$ {
	    index index.jsp
            proxy_pass   http://127.0.0.1:8080;
        }
	location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$ {
            expires      30d; 
        }
        location ~ .*\.(js|css)?$ {
            expires      1h; 
        }

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           html;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443;
    #    server_name  localhost;

    #    ssl                  on;
    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_timeout  5m;

    #    ssl_protocols  SSLv2 SSLv3 TLSv1;
    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers   on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

}

 

 

五.测试 以上配置都完成 然后在E:\nginx+memcached+tomcat\test0目录下添加测试文件：index.jsp

<%@ page language="java" contentType="text/html; charset=UTF-8"
    pageEncoding="UTF-8"%>

<%
	String path = request.getContextPath();
	String basePath = request.getScheme() + "://" + request.getServerName() + ":" + request.getServerPort() + path + "/admin_form/";
	out.println("the server address: " + basePath);
	out.println("<br/><br/>");

	String tmp = (String)session.getAttribute("tmp");
	if(tmp == null || tmp == "") {
		session.setAttribute("tmp", "i am created in tomcat 0 . my id is : " + session.getId());
		tmp = (String)session.getAttribute("tmp");
		out.println(" null now to created value !");
		out.println("<br/>");
		out.println(tmp);
		System.out.println(tmp);
	} else {
		out.println(tmp);
		System.out.println(tmp);
	}
%>

  在E:\nginx+memcached+tomcat\test1目录下也添加测试文件：index.jsp，文件内容和test0一样，只是为了区别，将session.setAttribute("tmp", "i am created in tomcat 0 . my id is : " + session.getId());改成tomcat 1   最后： 1.启动memcached 2.分别启动两个tomcat 3.启动nginx 在浏览器输入： http://127.0.0.1:80测试 浏览器显示信息： the server address: http://127.0.0.1:80/admin_form/ 
i am created in tomcat 0 . my id is : 4AF931E5DDBE816C49B83F5A63E9FEB6-n1.tomcat0 说明达到session共享。   如果将memcached关闭： 会发现：浏览器信息中tomcat0和tomcat1交替出现。