Docker搭建ELK笔记

ES version: 7.10.0

这里我们使用elk集成镜像,docker hub 地址:https://hub.docker.com/r/sebp/elk/

前提条件

  1. Docker至少得分配3GB的内存;
  2. Elasticsearch至少需要单独2G的内存;
  3. 防火墙开放相关端口; 9200,5404,56001
    如果配置太差了,可以指定使用的配置参数
    vm.max_map_count至少需要262144
1
2
3
echo "vm.max_map_count=262144" > /etc/sysctl.conf

sysctl -p

docker-compose

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
version: '2'
networks:
backend:
driver: bridge
services:
elk:
container_name: elk
image: sebp/elk:7.10.0
stdin_open: true
tty: true
restart: always
volumes:
- /opt/elk-data:/var/lib/elasticsearch
- /etc/localtime:/etc/localtime
ports:
- 5601:5601
- 9200:9200
- 5044:5044
environment:
- ES_MIN_MEM=128m
- ES_MAX_MEM=2048m
networks:
- backend

或者:

1
2
3
4
5
6
7
8
9
sudo docker run -dit --name elk \
-p 5601:5601 \
-p 9200:9200 \
-p 5044:5044 \
-e ES_MIN_MEM=128m \
-e ES_MAX_MEM=2048m \
-v /opt/elk-data:/var/lib/elasticsearch \
-v /etc/localtime:/etc/localtime \
sebp/elk:7.10.0

filebeat

https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html

filebeat的版本与es的版本一致

1
docker pull docker.elastic.co/beats/filebeat:7.10.0

Example configuration fileedit
Download this example configuration file as a starting point:

1
curl -L -O https://raw.githubusercontent.com/elastic/beats/7.10/deploy/docker/filebeat.docker.yml

filebeat.docker.yml 文件的内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false

filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true

processors:
- add_cloud_metadata: ~

output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
username: '${ELASTICSEARCH_USERNAME:}'
password: '${ELASTICSEARCH_PASSWORD:}'

docker-composer.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
version: '2'
networks:
backend:
driver: bridge
services:
filebeat:
container_name: filebeat
image: docker.elastic.co/beats/filebeat:7.10.0
stdin_open: true
tty: true
restart: always
volumes:
- $(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml
- /var/lib/docker/containers:/var/lib/docker/containers
- /var/run/docker.sock:/var/run/docker.sock
- /etc/localtime:/etc/localtime
- /www/wwwroot/PHP/thinkphp/runtime/log/:/logs/thinkphp/
networks:
- backend

filebeat.docker.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 日志输入配置
filebeat.inputs:
- type: log
enabled: true
paths:
- /logs/thinkphp/*/*.log
include_lines: ['error']
multiline:
pattern: '^\['
negate: true
match: after
output.elasticsearch:
hosts: ["elasticsearch:9200"]

#日志输出配置(采用 logstash 收集日志,5044为logstash端口)
#output.logstash:
# hosts: ['elasticsearch:5044']

只提取 error 日志


kibana 设置显示中文

修改 kibana 安装目录下的 config/kibana.yml, 在文件末尾添加:

1
i18n.locale: "zh-CN"

重启 kibana



ES 设置密码

ES 版本: 7.10.0

  1. 修改 elasticsearch 安装目录下的 config/elasticsearch.yml,在文件末尾添加,修改完之后,重启elasticsearch 服务
1
2
3
4
5
# discovery.type: single-node

xpack.security.enabled: true
xpack.license.self_generated.type: basic
xpack.security.transport.ssl.enabled: true
  1. 执行设置用户名和密码的命令,这里会为4个左右用户分别设置密码,elastic, kibana, logstash_system,beats_system 等
1
D:\devops\elasticsearch-7.8.0\bin> elasticsearch-setup-passwords interactive
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y

Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana_system]:
Reenter password for [kibana_system]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]
  1. 修改密码命令如下
1
curl -H "Content-Type:application/json" -XPOST -u elastic 'http://127.0.0.1:9200/_xpack/security/user/elastic/_password' -d '{ "password" : "123456" }'
  1. 修改 Kibana 配置文件 config/kibana.yml文件
1
2
elasticsearch.username: "elastic"
elasticsearch.password: "123456"

filebeat.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
filebeat.inputs:
- type: log
enabled: true
paths:
- D:/workspace/lamp/thinkphp/runtime/v1/log/*/*.log
# include_lines: ['error']
multiline:
pattern: '^\[\s\d{4}-\d{2}(-\d{2})?'
negate: true
match: after
tags: ["thinkphp-api"]

#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"]

# Protocol - either `http` (default) or `https`.
#protocol: "https"

# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# username: "elastic"
# password: "wakasann"
# indices:
# - index: "thinkphp-api-error"
# when.contains:
# tags: "thinkphp-api"


output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
index: "thinkphp-api-error"

filebeat 运行

1
filebeat.exe -c filebeat.yml -e

logstash

thinkphpapi.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
input {
beats {
port => 5044
}
}

output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "elastic"
password => "wakasann"
}
}

通过powershell 运行 logstash

1
.\bin\logstash.bat -f .\config\thinkphpapi.conf