网站Logo linux从入门到入土
首页

企业级集群架构容器化实战：从传统部署到云原生转型

admin
13
2023-12-12

企业级集群架构容器化实战：从传统部署到云原生转型

项目概述

业务背景

随着业务规模扩大和微服务架构普及，传统部署方式面临诸多挑战。本项目通过容器化改造，将传统LNMP架构升级为云原生架构，实现快速部署、弹性伸缩和高效运维。

架构目标

构建具备以下特性的现代化应用架构：

  • 容器化部署：所有服务组件容器化运行
  • 自动化运维：Ansible实现环境一键部署
  • 高可用负载：多层级负载均衡保障服务连续性
  • 弹性扩展：支持水平扩展应对流量波动

系统架构设计

整体架构图

用户访问 → [前端负载均衡: LB01/LB02] → [前端容器: Web01/Web02] → [后端负载均衡] → [后端容器: Web03/Web04] → [数据库容器: DB01]

服务器规划

角色 主机名 IP地址 容器化组件
前端负载 doc-lb01 172.16.1.15 Nginx (宿主机)
前端负载 doc-lb02 172.16.1.16 Nginx (宿主机)
前端应用 doc-web01 172.16.1.17 Nginx容器
前端应用 doc-web02 172.16.1.18 Nginx容器
后端应用 doc-web03 172.16.1.19 Java应用容器
后端应用 doc-web04 172.16.1.20 Java应用容器
数据库 doc-db01 172.16.1.53 MySQL容器

第一阶段：基础设施自动化

1.1 Ansible自动化部署Docker环境

主机清单配置：

# /server/ans/playbooks/hosts
[docker_lb]
172.16.1.15  hostname=doc-lb01.your.cn
172.16.1.16  hostname=doc-lb02.your.cn

[docker_web]
172.16.1.17  hostname=doc-web01.your.cn
172.16.1.18  hostname=doc-web02.your.cn
172.16.1.19  hostname=doc-web03.your.cn
172.16.1.20  hostname=doc-web04.your.cn

[docker_db]
172.16.1.53  hostname=doc-db01.your.cn

[docker:children]
docker_lb
docker_web
docker_db

Docker环境部署Playbook：

# docker_deploy.yml
---
- name: 部署Docker运行环境
  hosts: docker
  become: yes
  vars:
    docker_version: "27.0.3"
  
  tasks:
    - name: 清理现有runc组件
      file:
        path: /usr/local/bin/runc
        state: absent
      
    - name: 创建临时解压目录
      tempfile:
        state: directory
        suffix: _docker_unpack
      register: temp_dir
    
    - name: 解压Docker安装包
      unarchive:
        src: "{{ playbook_dir }}/files/docker-{{ docker_version }}.tgz"
        dest: "{{ temp_dir.path }}"
        remote_src: no
      
    - name: 安装Docker二进制文件
      copy:
        src: "{{ temp_dir.path }}/docker/"
        dest: /usr/bin/
        mode: '0755'
        remote_src: yes
      
    - name: 部署Docker服务文件
      copy:
        src: "{{ playbook_dir }}/files/docker.service"
        dest: /usr/lib/systemd/system/docker.service
        mode: '0644'
      
    - name: 配置Docker镜像加速
      copy:
        dest: /etc/docker/daemon.json
        content: |
          {
            "registry-mirrors": [
              "https://mirror.aliyuncs.com",
              "https://docker.mirrors.ustc.edu.cn",
              "https://docker.nju.edu.cn"
            ],
            "insecure-registries": [
              "registry.your.cn:5000"
            ]
          }
        mode: '0644'
      
    - name: 启动Docker服务
      systemd:
        name: docker
        state: started
        enabled: yes
        daemon_reload: yes
      
    - name: 验证Docker安装
      command: docker --version
      register: docker_result
      changed_when: false
    
    - name: 显示安装结果
      debug:
        msg: "Docker {{ docker_result.stdout }} 安装成功"
      
    - name: 清理临时目录
      file:
        path: "{{ temp_dir.path }}"
        state: absent

1.2 Docker Compose环境部署

Docker Compose部署Playbook：

# docker_compose_deploy.yml
---
- name: 部署Docker Compose环境
  hosts: docker
  become: yes
  vars:
    compose_version: "2.30.3"
  
  tasks:
    - name: 安装Docker Compose
      copy:
        src: "{{ playbook_dir }}/files/docker-compose-linux-x86_64"
        dest: /usr/local/bin/docker-compose
        mode: '0755'
      
    - name: 创建符号链接
      file:
        src: /usr/local/bin/docker-compose
        dest: /usr/bin/docker-compose
        state: link
      
    - name: 验证安装
      command: docker-compose --version
      register: compose_result
      changed_when: false
    
    - name: 显示安装结果
      debug:
        msg: "Docker Compose {{ compose_result.stdout }} 安装成功"

1.3 主机名统一配置

主机名配置Playbook：

# hostname_config.yml
---
- name: 集群主机名统一配置
  hosts: all
  gather_facts: false
  
  tasks:
    - name: 设置主机名
      hostname:
        name: "{{ hostname }}"
      
    - name: 分发统一的hosts文件
      copy:
        src: "{{ playbook_dir }}/files/hosts"
        dest: /etc/hosts
        backup: yes
      
    - name: 验证主机名配置
      command: hostname
      register: hostname_result
      changed_when: false
    
    - name: 显示配置结果
      debug:
        msg: "主机名已设置为: {{ hostname_result.stdout }}"

第二阶段：数据库容器化

2.1 MySQL数据库容器部署

Docker Compose配置：

# /server/docker-compose/mysql/docker-compose.yml
version: '3.8'

services:
  mysql:
    image: mysql:8.0-debian
    container_name: exam_mysql
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: "YourSecureRootPassword123!"
      MYSQL_DATABASE: exam
      MYSQL_USER: exam
      MYSQL_PASSWORD: "ExamUserPassword123!"
      TZ: Asia/Shanghai
    volumes:
      - mysql_data:/var/lib/mysql
      - ./conf.d:/etc/mysql/conf.d:ro
    ports:
      - "3306:3306"
    networks:
      - exam_network
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
      timeout: 10s
      retries: 3

volumes:
  mysql_data:

networks:
  exam_network:
    driver: bridge

数据库初始化脚本：

#!/bin/bash
# init_database.sh

# 等待MySQL服务启动
while ! docker exec exam_mysql mysqladmin ping -h localhost --silent; do
    echo "等待MySQL启动..."
    sleep 5
done

# 导入数据库结构
docker cp xzs-mysql.sql exam_mysql:/tmp/
docker exec exam_mysql bash -c "mysql -uexam -pExamUserPassword123! exam < /tmp/xzs-mysql.sql"

# 验证数据导入
docker exec exam_mysql mysql -uexam -pExamUserPassword123! -e "SHOW TABLES FROM exam;"

echo "数据库初始化完成"

第三阶段：后端应用容器化

3.1 Java应用容器配置

项目目录结构：

/app/code/exam/backend/
├── docker-compose.yml
├── Dockerfile
├── entry.sh
├── application-prod.yml
└── xzs-3.9.0.jar

Dockerfile配置：

# Dockerfile
FROM openjdk:8-jre-slim

LABEL maintainer="devops@your.cn"
LABEL version="3.9.0"
LABEL description="在线考试系统后端服务"

# 环境变量配置
ENV APP_DIR /app
ENV PROFILE prod
ENV JAR_FILE xzs-3.9.0.jar
ENV TZ Asia/Shanghai

# 创建应用目录
RUN mkdir -p ${APP_DIR} && \
    apt-get update && \
    apt-get install -y curl && \
    rm -rf /var/lib/apt/lists/* && \
    ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

WORKDIR ${APP_DIR}

# 复制应用文件
COPY ${JAR_FILE} application-${PROFILE}.yml ./
COPY entry.sh /

# 健康检查
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD curl -f http://localhost:8000/actuator/health || exit 1

EXPOSE 8000

ENTRYPOINT ["/entry.sh"]

启动脚本：

#!/bin/bash
# entry.sh

echo "启动在线考试系统后端服务..."
echo "环境: ${PROFILE}"
echo "JAR文件: ${JAR_FILE}"

exec java -Duser.timezone=Asia/Shanghai \
          -Dspring.profiles.active=${PROFILE} \
          -jar ${JAR_FILE}

应用配置文件：

# application-prod.yml
server:
  port: 8000
  servlet:
    context-path: /
  undertow:
    io-threads: 16
    worker-threads: 400
    buffer-size: 1024
    direct-buffers: true
  compression:
    enabled: true
    min-response-size: 1024

logging:
  level:
    com.your.exam: INFO
  file:
    name: ${APP_DIR}/logs/application.log
  pattern:
    file: "%d{yyyy-MM-dd HH:mm:ss} - %msg%n"

spring:
  datasource:
    url: jdbc:mysql://doc-db01.your.cn:3306/exam?useSSL=false&useUnicode=true&serverTimezone=Asia/Shanghai&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&allowPublicKeyRetrieval=true&allowMultiQueries=true
    username: exam
    password: ExamUserPassword123!
    driver-class-name: com.mysql.cj.jdbc.Driver
    hikari:
      maximum-pool-size: 20
      minimum-idle: 5
      connection-timeout: 30000

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics

Docker Compose配置：

# docker-compose.yml
version: '3.8'

services:
  exam_backend:
    build:
      context: .
      dockerfile: Dockerfile
    image: registry.your.cn/exam/backend:3.9.0
    container_name: exam_backend
    restart: unless-stopped
    ports:
      - "8000:8000"
    environment:
      - PROFILE=prod
    networks:
      - exam_network
    depends_on:
      mysql:
        condition: service_healthy

networks:
  exam_network:
    external: true

第四阶段：后端负载均衡配置

4.1 四层负载均衡（Nginx Stream）

Nginx Stream配置：

# /etc/nginx/nginx.conf 添加stream模块配置
stream {
    upstream exam_backend_pool {
        # 一致性哈希，保证同一客户端IP访问同一后端
        hash $remote_addr consistent;
        server doc-web03.your.cn:8000 max_fails=3 fail_timeout=30s;
        server doc-web04.your.cn:8000 max_fails=3 fail_timeout=30s;
    }

    # 日志格式定义
    log_format stream_log '$remote_addr [$time_local] '
                         '$protocol $status $bytes_sent $bytes_received '
                         '$session_time $upstream_addr';

    # 访问日志
    access_log /var/log/nginx/stream-access.log stream_log;

    server {
        listen 8000;
        proxy_pass exam_backend_pool;
        proxy_connect_timeout 5s;
        proxy_timeout 30s;
      
        # 健康检查
        proxy_next_upstream on;
        proxy_next_upstream_timeout 0;
        proxy_next_upstream_tries 0;
    }
}

第五阶段：前端应用容器化

5.1 前端应用配置

项目目录结构：

/app/code/exam/frontend/
├── docker-compose.yml
├── Dockerfile
├── exam.conf
├── admin/
└── student/

前端Nginx配置：

# exam.conf
# 管理端配置
server {
    listen 80;
    server_name admin.your.cn;
  
    root /app/code/exam/frontend/admin;
    index index.html;
  
    # Gzip压缩
    gzip on;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
  
    # 静态资源缓存
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg)$ {
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
  
    # API代理
    location /api/ {
        proxy_pass http://doc-lb01.your.cn:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
      
        # 超时设置
        proxy_connect_timeout 30s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }
  
    # 前端路由支持
    location / {
        try_files $uri $uri/ /index.html;
    }
}

# 学生端配置
server {
    listen 80;
    server_name stu.your.cn;
  
    root /app/code/exam/frontend/student;
    index index.html;
  
    # 配置同上...
    location /api/ {
        proxy_pass http://doc-lb01.your.cn:8000;
        # ... 其他配置
    }
  
    location / {
        try_files $uri $uri/ /index.html;
    }
}

Dockerfile配置：

FROM nginx:1.24-alpine

LABEL maintainer="frontend@your.cn"
LABEL version="1.0.0"

# 设置时区
RUN apk add --no-cache tzdata && \
    ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \
    echo "Asia/Shanghai" > /etc/timezone

# 创建应用目录
ENV APP_DIR /app/code/exam/frontend
RUN mkdir -p ${APP_DIR}

WORKDIR ${APP_DIR}

# 复制前端文件（已提前打包为tar.gz）
COPY exam-frontend.tar.gz .
RUN tar -xzf exam-frontend.tar.gz && \
    rm -f exam-frontend.tar.gz && \
    chown -R nginx:nginx ${APP_DIR}

# 复制Nginx配置
COPY exam.conf /etc/nginx/conf.d/

# 健康检查
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD curl -f http://localhost/ || exit 1

EXPOSE 80

CMD ["nginx", "-g", "daemon off;"]

前端Docker Compose配置：

# docker-compose.yml
version: '3.8'

services:
  exam_frontend:
    build:
      context: .
      dockerfile: Dockerfile
    image: registry.your.cn/exam/frontend:1.0.0
    container_name: exam_frontend
    restart: unless-stopped
    ports:
      - "80:80"
    networks:
      - exam_network

networks:
  exam_network:
    external: true

第六阶段：前端负载均衡与高可用

6.1 七层负载均衡配置

Nginx负载均衡配置：

# /etc/nginx/conf.d/exam.conf
upstream frontend_pool {
    least_conn;
    server doc-web01.your.cn:80 weight=3 max_fails=3 fail_timeout=30s;
    server doc-web02.your.cn:80 weight=3 max_fails=3 fail_timeout=30s;
}

server {
    listen 80;
    server_name admin.your.cn stu.your.cn;
  
    access_log /var/log/nginx/exam-access.log main;
    error_log /var/log/nginx/exam-error.log notice;
  
    location / {
        proxy_pass http://frontend_pool;
      
        # 透传头部信息
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
      
        # 超时配置
        proxy_connect_timeout 30s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
      
        # 缓冲区优化
        proxy_buffering on;
        proxy_buffer_size 4k;
        proxy_buffers 8 4k;
    }
  
    # 负载均衡状态监控
    location /nginx_status {
        stub_status on;
        access_log off;
        allow 127.0.0.1;
        allow 172.16.1.0/24;
        deny all;
    }
}

6.2 Keepalived高可用配置

主负载均衡器配置：

# /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id doc_lb01
    script_user root
    enable_script_security
}

vrrp_script chk_nginx {
    script "/usr/bin/systemctl is-active nginx"
    interval 2
    weight -5
    fall 2
    rise 1
}

vrrp_instance VI_EXAM {
    state MASTER
    interface ens33
    virtual_router_id 88
    priority 100
    advert_int 1
  
    authentication {
        auth_type PASS
        auth_pass YourKeepalivedPass123
    }
  
    virtual_ipaddress {
        172.16.1.100/24 dev ens33
    }
  
    track_script {
        chk_nginx
    }
  
    notify_master "/etc/keepalived/notify_master.sh"
    notify_backup "/etc/keepalived/notify_backup.sh"
}

部署与验证

一键部署脚本

#!/bin/bash
# deploy_cluster.sh

echo "开始部署在线考试系统集群..."

# 1. 基础设施部署
echo "步骤1: 部署Docker环境"
ansible-playbook -i hosts docker_deploy.yml

echo "步骤2: 部署Docker Compose"
ansible-playbook -i hosts docker_compose_deploy.yml

# 2. 数据库部署
echo "步骤3: 部署MySQL数据库"
cd /server/docker-compose/mysql
docker-compose up -d
./init_database.sh

# 3. 后端服务部署
echo "步骤4: 部署后端服务"
ansible-playbook -i hosts backend_deploy.yml

# 4. 前端服务部署
echo "步骤5: 部署前端服务"
ansible-playbook -i hosts frontend_deploy.yml

# 5. 负载均衡配置
echo "步骤6: 配置负载均衡"
ansible-playbook -i hosts loadbalancer_deploy.yml

echo "集群部署完成!"
echo "访问地址:"
echo "管理端: http://admin.your.cn"
echo "学生端: http://stu.your.cn"

健康检查脚本

#!/bin/bash
# health_check.sh

echo "集群健康检查..."

services=(
    "doc-db01.your.cn:3306"
    "doc-web03.your.cn:8000"
    "doc-web04.your.cn:8000"
    "doc-web01.your.cn:80"
    "doc-web02.your.cn:80"
    "doc-lb01.your.cn:80"
)

for service in "${services[@]}"; do
    host=${service%:*}
    port=${service#*:}
  
    if nc -z $host $port; then
        echo "✓ $service 服务正常"
    else
        echo "✗ $service 服务异常"
    fi
done

# 检查容器状态
echo "容器状态检查:"
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

监控与维护

日志收集配置

# 配置日志轮转
cat > /etc/logrotate.d/docker-exam << EOF
/var/lib/docker/containers/*/*.log {
    daily
    rotate 7
    compress
    delaycompress
    copytruncate
    missingok
}
EOF

性能监控

# 监控脚本
#!/bin/bash
# monitor_resources.sh

echo "系统资源监控:"
echo "CPU使用率: $(top -bn1 | grep "Cpu(s)" | awk '{print $2}')%"
echo "内存使用: $(free -h | grep Mem | awk '{print $3"/"$2}')"
echo "磁盘使用: $(df -h / | awk 'NR==2 {print $3"/"$2}')"

echo "容器资源使用:"
docker stats --no-stream --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}"

总结

通过本项目的实施，我们成功实现了：

技术成果

  1. 全栈容器化：所有服务组件完成容器化改造
  2. 自动化部署：Ansible实现环境一键部署
  3. 高可用架构：多层级负载均衡保障服务连续性
  4. 标准化运维：统一的配置管理和监控体系

业务价值

  1. 部署效率：部署时间从小时级降至分钟级
  2. 资源利用率：容器化带来更高的资源密度
  3. 故障恢复：快速的服务重启和故障转移
  4. 扩展能力：支持快速的业务规模扩展
动物装饰