1. 为什么需要多服务日志集中管理在微服务架构中一个典型的生产环境往往包含多个后端服务和前端应用。以我最近部署的一个电商平台为例整套系统包含3个Node.js后端服务用户服务/订单服务/商品服务2个Vue前端应用主站/管理后台1个Nginx网关Redis和MySQL各1个实例当所有服务都通过Docker Compose部署后最头疼的问题就是日志分散在各个容器中。上周排查一个支付失败问题时我不得不同时docker logs order-service查看订单服务日志docker logs payment-service查看支付服务日志docker logs nginx查看网关日志再到Kibana中筛选前端错误日志这种碎片化的排查方式效率极低特别是当多个服务间存在调用链时。更糟的是docker logs默认只保留最新日志历史数据随时可能丢失。2. 基础版方案直接挂载日志卷2.1 目录结构设计我采用的日志目录结构如下/project /services /backend - docker-compose.yml /frontend - docker-compose.yml /logs /backend /user-service /order-service /product-service /frontend /main-site /admin-console /infra /nginx /mysql2.2 关键Compose配置以订单服务为例的配置片段version: 3.8 services: order-service: image: registry.example.com/order:v1.2 volumes: - /project/logs/backend/order-service:/var/log/app logging: driver: json-file options: max-size: 10m max-file: 32.3 实际踩坑记录权限问题首次启动时出现Permission denied错误解决方法mkdir -p /project/logs/backend/order-service chmod -R 777 /project/logs # 生产环境应配置精确权限日志轮转失效发现日志文件超过10MB后未切割原因是必须同时设置max-size和max-file某些应用如Nginx需要在其配置中也开启日志轮转磁盘空间预警建议添加监控脚本#!/bin/bash LOG_DIR/project/logs THRESHOLD90 current_usage$(df -h $LOG_DIR | awk NR2 {print $5} | tr -d %) if [ $current_usage -gt $THRESHOLD ]; then echo WARNING: Log directory usage $current_usage% | mail -s Log Alert adminexample.com fi3. 进阶版方案EFK全栈日志系统3.1 架构设计graph TD A[Frontend] --|Log| B(Fluentd) C[Backend] --|Log| B D[Database] --|Log| B B -- E[Elasticsearch] E -- F[Kibana]3.2 关键组件配置3.2.1 Fluentd配置fluentd.conf核心部分source type forward port 24224 /source match docker.** type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix docker flush_interval 5s /match3.2.2 Elasticsearch调优docker-compose.efk.yml内存设置services: elasticsearch: image: elasticsearch:7.17.0 environment: - ES_JAVA_OPTS-Xms2g -Xmx2g # 根据服务器内存调整 - discovery.typesingle-node ulimits: memlock: soft: -1 hard: -13.3 性能优化技巧日志过滤在Fluentd中丢弃DEBUG级别日志filter docker.** type grep exclude key log pattern /DEBUG/ /exclude /filter索引策略每天自动创建新索引match docker.** type elasticsearch logstash_date_format %Y%m%d /match查询加速为常用字段添加索引PUT /docker-*/_mapping { properties: { container_name: { type: keyword }, service: { type: keyword } } }4. 混合部署实战案例4.1 场景需求开发环境使用基础版日志卷生产环境使用EFK全栈方案两种环境配置无缝切换4.2 条件化Compose配置x-logging: default-logging driver: ${LOG_DRIVER:-json-file} options: max-size: 10m max-file: 3 x-efk-logging: efk-logging driver: fluentd options: fluentd-address: fluentd:24224 tag: docker.{{.Name}} services: webapp: image: myapp:${TAG:-latest} logging: : *${LOG_CONFIG:-default-logging}启动命令示例# 开发环境 LOG_DRIVERjson-file docker-compose up # 生产环境 LOG_CONFIGefk-logging TAGprod docker-compose -f docker-compose.yml -f docker-compose.efk.yml up4.3 监控指标设计Grafana仪表盘建议包含日志量趋势按服务/级别错误日志TOP 10日志存储增长预测关键服务响应时间关联分析5. 异常处理手册5.1 Fluentd常见问题症状日志未出现在Kibana中 排查步骤检查Fluentd容器状态docker-compose logs fluentd | grep -i error验证Elasticsearch连接curl -XGET http://elasticsearch:9200/_cat/indices?v查看缓冲队列docker exec fluentd ls -lh /var/log/fluentd/buffer5.2 日志丢失预防启用本地备份双写策略match docker.** type copy store type elasticsearch ... /store store type file path /var/log/fluentd/backup /store /match设置磁盘水位警戒线docker run -it --rm --privileged --pidhost debian nsenter -t 1 -m -u -n -i sh -c echo 90 /proc/sys/vm/dirty_ratio6. 安全加固方案6.1 访问控制Kibana基础认证services: kibana: environment: - ELASTICSEARCH_USERNAMEadmin - ELASTICSEARCH_PASSWORD${ES_PASSWORD}Elasticsearch TLS加密# 生成证书 docker run --rm elasticsearch:7.17.0 \ bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass # 挂载到容器 volumes: - ./certs/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p126.2 日志脱敏Fluentd过滤器示例filter docker.** type record_transformer enable_ruby true record message ${record[message].gsub(/\b\d{4}-\d{4}-\d{4}-\d{4}\b/, [CREDIT_CARD])} /record /filter7. 成本优化实践7.1 冷热数据分离PUT _ilm/policy/logs_policy { policy: { phases: { hot: { actions: { rollover: { max_size: 50GB, max_age: 7d } } }, warm: { min_age: 7d, actions: { allocate: { number_of_replicas: 1 } } } } } }7.2 日志采样配置对DEBUG日志进行50%采样filter docker.** type sample rate 50 sample_key level sample_value DEBUG /filter 微服务架构下的日志集中管理实践与EFK方案
1. 为什么需要多服务日志集中管理在微服务架构中一个典型的生产环境往往包含多个后端服务和前端应用。以我最近部署的一个电商平台为例整套系统包含3个Node.js后端服务用户服务/订单服务/商品服务2个Vue前端应用主站/管理后台1个Nginx网关Redis和MySQL各1个实例当所有服务都通过Docker Compose部署后最头疼的问题就是日志分散在各个容器中。上周排查一个支付失败问题时我不得不同时docker logs order-service查看订单服务日志docker logs payment-service查看支付服务日志docker logs nginx查看网关日志再到Kibana中筛选前端错误日志这种碎片化的排查方式效率极低特别是当多个服务间存在调用链时。更糟的是docker logs默认只保留最新日志历史数据随时可能丢失。2. 基础版方案直接挂载日志卷2.1 目录结构设计我采用的日志目录结构如下/project /services /backend - docker-compose.yml /frontend - docker-compose.yml /logs /backend /user-service /order-service /product-service /frontend /main-site /admin-console /infra /nginx /mysql2.2 关键Compose配置以订单服务为例的配置片段version: 3.8 services: order-service: image: registry.example.com/order:v1.2 volumes: - /project/logs/backend/order-service:/var/log/app logging: driver: json-file options: max-size: 10m max-file: 32.3 实际踩坑记录权限问题首次启动时出现Permission denied错误解决方法mkdir -p /project/logs/backend/order-service chmod -R 777 /project/logs # 生产环境应配置精确权限日志轮转失效发现日志文件超过10MB后未切割原因是必须同时设置max-size和max-file某些应用如Nginx需要在其配置中也开启日志轮转磁盘空间预警建议添加监控脚本#!/bin/bash LOG_DIR/project/logs THRESHOLD90 current_usage$(df -h $LOG_DIR | awk NR2 {print $5} | tr -d %) if [ $current_usage -gt $THRESHOLD ]; then echo WARNING: Log directory usage $current_usage% | mail -s Log Alert adminexample.com fi3. 进阶版方案EFK全栈日志系统3.1 架构设计graph TD A[Frontend] --|Log| B(Fluentd) C[Backend] --|Log| B D[Database] --|Log| B B -- E[Elasticsearch] E -- F[Kibana]3.2 关键组件配置3.2.1 Fluentd配置fluentd.conf核心部分source type forward port 24224 /source match docker.** type elasticsearch host elasticsearch port 9200 logstash_format true logstash_prefix docker flush_interval 5s /match3.2.2 Elasticsearch调优docker-compose.efk.yml内存设置services: elasticsearch: image: elasticsearch:7.17.0 environment: - ES_JAVA_OPTS-Xms2g -Xmx2g # 根据服务器内存调整 - discovery.typesingle-node ulimits: memlock: soft: -1 hard: -13.3 性能优化技巧日志过滤在Fluentd中丢弃DEBUG级别日志filter docker.** type grep exclude key log pattern /DEBUG/ /exclude /filter索引策略每天自动创建新索引match docker.** type elasticsearch logstash_date_format %Y%m%d /match查询加速为常用字段添加索引PUT /docker-*/_mapping { properties: { container_name: { type: keyword }, service: { type: keyword } } }4. 混合部署实战案例4.1 场景需求开发环境使用基础版日志卷生产环境使用EFK全栈方案两种环境配置无缝切换4.2 条件化Compose配置x-logging: default-logging driver: ${LOG_DRIVER:-json-file} options: max-size: 10m max-file: 3 x-efk-logging: efk-logging driver: fluentd options: fluentd-address: fluentd:24224 tag: docker.{{.Name}} services: webapp: image: myapp:${TAG:-latest} logging: : *${LOG_CONFIG:-default-logging}启动命令示例# 开发环境 LOG_DRIVERjson-file docker-compose up # 生产环境 LOG_CONFIGefk-logging TAGprod docker-compose -f docker-compose.yml -f docker-compose.efk.yml up4.3 监控指标设计Grafana仪表盘建议包含日志量趋势按服务/级别错误日志TOP 10日志存储增长预测关键服务响应时间关联分析5. 异常处理手册5.1 Fluentd常见问题症状日志未出现在Kibana中 排查步骤检查Fluentd容器状态docker-compose logs fluentd | grep -i error验证Elasticsearch连接curl -XGET http://elasticsearch:9200/_cat/indices?v查看缓冲队列docker exec fluentd ls -lh /var/log/fluentd/buffer5.2 日志丢失预防启用本地备份双写策略match docker.** type copy store type elasticsearch ... /store store type file path /var/log/fluentd/backup /store /match设置磁盘水位警戒线docker run -it --rm --privileged --pidhost debian nsenter -t 1 -m -u -n -i sh -c echo 90 /proc/sys/vm/dirty_ratio6. 安全加固方案6.1 访问控制Kibana基础认证services: kibana: environment: - ELASTICSEARCH_USERNAMEadmin - ELASTICSEARCH_PASSWORD${ES_PASSWORD}Elasticsearch TLS加密# 生成证书 docker run --rm elasticsearch:7.17.0 \ bin/elasticsearch-certutil cert -out /tmp/elastic-certificates.p12 -pass # 挂载到容器 volumes: - ./certs/elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p126.2 日志脱敏Fluentd过滤器示例filter docker.** type record_transformer enable_ruby true record message ${record[message].gsub(/\b\d{4}-\d{4}-\d{4}-\d{4}\b/, [CREDIT_CARD])} /record /filter7. 成本优化实践7.1 冷热数据分离PUT _ilm/policy/logs_policy { policy: { phases: { hot: { actions: { rollover: { max_size: 50GB, max_age: 7d } } }, warm: { min_age: 7d, actions: { allocate: { number_of_replicas: 1 } } } } } }7.2 日志采样配置对DEBUG日志进行50%采样filter docker.** type sample rate 50 sample_key level sample_value DEBUG /filter