elasticsearch学习笔记001

《Elasticsearch 核心技术与实战》Github代码 https://github.com/onebirdrocks/geektime-ELK

运行的环境: windows

安装了 PowerShell-7.0.0

  1. 下载 elasticsearch和Kibana 都可以在华为云 https://mirrors.huaweicloud.com/ 上下载到

我安装的 elasticsearch 7.4.0 和 kibana 7.4.0

安装是解压之后,就可以使用的,安装了jdk 1.8.221,在环境变量中,配置了 JAVA_HOME并加入到 path中去了。

启动 powershell,进入 elasticsearch解压的目录,启动 elasticsearch

1
./bin/elasticsearch

浏览器可通过http://localhost:9200 访问,查看elasticsearch的基本信息

查看安装的插件

1
./bin/elasticsearch-plugin list

安装一个 analysis-icu

1
./bin/elasticsearch-plugin install analysis-icu

可用 ./bin/elasticsearch-plugin list再次查看 安装的插件,然后启动 elasticsearch,

可通过http://localhost:9200/_cat/plugins 查看安装的列表

_cat/ 是 elasticsearch 提供的要给 api

  • plugins 查看安装的插件
  • nodes 查看运行的节点

如何在开发机上运行多个 Elasticsearch实例

1
2
3
./bin/elasticsearch -E node.name=node1 -E cluster.name=geektime -E path.data=node1_data -d 
./bin/elasticsearch -E node.name=node2 -E cluster.name=geektime -E path.data=node2_data -d
./bin/elasticsearch -E node.name=node3 -E cluster.name=geektime -E path.data=node3_data -d

node.name 节点的名称

cluster.name 一个相同的集群的名称

pat.data 为每个节点设置存放数据的地址

注: 在 windows下的 powershell 好像不需要最后的 -d 参数,我是打开 三个 powershell的窗口,单独运行上面的命令的

  • 删除进程 ps grep|elasticsearch / kill pid

kibana

启动

1
./bin/kibana

默认是 5601 端口

http://localhost:5601

在 首页 有一个 Add sample data 下面有一个 链接,点击进去,可以进行导入 kibana 提供的简单数据

kinana console

  • dev tools
  • Search Profiler
  • Help + 一些快捷键
    • cmd + / (查看API帮助文档)
    • cmd + option + l
    • cmd + option + 0
    • cmd + option + shift + 0

kibana plugins

1
2
3
bin/kibana-plugin install plugin_location
bin/kibana-plugin list
bin/kibana remove

Apps:

  • LogTrail
  • Own Home
  • Shard Allocation
  • Corweyor
  • Indices View
  • Analyze UI
  • Cleaner Setting index ttl
  • ElastAlert Kibana Plugin

Visualizations:

  • 3D Chars
  • 3D Graph
  • Bmap
  • C3JS Visualzations
  • Calendar Visualzation
  • Cohort analysis
  • Collored Metilc Visualzation
  • Dendrogram
  • Dotplot
  • Dropdown
  • Enhanced Table
  • Enhanced Themap
  • Extended Metric

学习在本机Docker环境中云高兴 ELK Stack

  • Docker-compose 相关命令
    • 运行 docker-compose up [-d]
    • docker-compose down
    • docker-compose down -v
    • docker stop / rm containerID

Demo

  • 运行 Docker-compose 本地构建 Elasticsearch 分布式特性
  • 集成 Cerobro,方便查看集群状态,默认运行在 9000端口,可通过http://localhost:9000

logstash

2.4-Logstash安装与导入数据

参考文档说明导入 movices.csv 数据,启动 logstash

1
bin/logstash -f path_logstash.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
input {
file {
path => "D:/dev/data/movies.csv"
start_position => "beginning"
sincedb_path => "D:/dev/logstash/mydata/moive.txt"
}
}
filter {
csv {
separator => ","
columns => ["id","content","genre"]
}

mutate {
split => { "genre" => "|" }
remove_field => ["path", "host","@timestamp","message"]
}

mutate {

split => ["content", "("]
add_field => { "title" => "%{[content][0]}"}
add_field => { "year" => "%{[content][1]}"}
}

mutate {
convert => {
"year" => "integer"
}
strip => ["title"]
remove_field => ["path", "host","@timestamp","message","content"]
}

}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "movies"
document_id => "%{id}"
}
stdout {}
}
1
[2019-10-23T09:58:45,519][ERROR][logstash.javapipeline    ][main] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<ArgumentError: The "sincedb_path" argument must point to a file, received a directory: "D:/dev/logstash/mydata/moive">,

sincedb_path 需要是一个文件,而不是一个目录


文檔CURD

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
//create document.自动生成 _id
POSt users/_doc
{
"user": "Mkie",
"post_date": "2019-10-23 00:01:22",
"message": "hello"
}

//create document,指定id,如果id存在,报错

PUT users/_doc/1?op_type=create
{
"user": "kangkang",
"post_date": "2019-10-23 02:01:22",
"message": "hello2"
}


//create document,指定id已经存在,就报错
PUT users/_create/1
{
"user": "lulu",
"post_date": "2019-10-23 03:01:22",
"message": "hello3"
}

//get document by id
GET users/_doc/1


PUT users/_doc/1
{
"user": "my mike"
}

//在原文档增加字段


POST users/_update/1
{
"doc":{
"post_date" : "2019-10-10 23:56:56",
"message": "just message"
}
}


DELETE usrs/_doc/1

bulk 操作

每个操作的失败不会影响其它操作

批量读取 - mget

1
2
3
4
5
6
7
8
9
10
11
12
13
GEt /_mget
{
"docs":[
{
"_index" : "user",
"_id": 1,
},
{
"_index": "comments",
"_id": 1
}
]
}

msearch _msearch


正排索引和倒排索引


analysis与analyzer

analyzer 分词器

使用 analyzer api

1
2
3
4
5
6
get /_analyze

POST books/_analyzer
{
"filed": "title"
}

Standard Analyzer

  • 默认分词器
  • 按词划分
  • 小写处理

Simple Analyzer

  • 按照非字母,非字母都被去除

Whitespace Analyzer

  • 按照空格划分

Stop Analyzer

  • 相比Simple Analyzer

language analyzer

icu analyzer

需要按照插件

1
elasticsearch-plugin install analysis-icu

提供了Unicode的支持,更好的支持亚洲语音

更多的中文分词器

  • IK
  • THULAC
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
//standard
GET _analyze
{
"analyzer": "standard",
"text": "He my polite be object oh change. Consider no mr am overcame yourself throwing sociable children. Hastily her totally conduct may. "
}

//simple
GET _analyze
{
"analyzer": "simple",
"text": "He my polite be object oh change. Consider no mr am overcame yourself throwing sociable children. Hastily her totally conduct may. "
}

//stop
GET _analyze
{
"analyzer": "stop",
"text": "Six started far placing saw respect females old. Civilly why how end viewing attempt related enquire visitor. Man particular insensible celebrated conviction stimulated principles day. "
}

//whitespace
GET _analyze
{
"analyzer": "whitespace",
"text": "Six started far placing saw respect females old. Civilly why how end viewing attempt related enquire visitor. Man particular insensible celebrated conviction stimulated principles day."
}

//keyword
GET _analyze
{
"analyzer": "keyword",
"text": "Six started far placing saw respect females old. Civilly why how end viewing attempt related enquire visitor. Man particular insensible celebrated conviction stimulated principles day."
}

//pattern,default \w
GET _analyze
{
"analyzer": "pattern",
"text": "2 Six started far-placing saw respect females old. Civilly why how end viewing attempt related enquire visitor. Man particular insensible celebrated conviction stimulated principles day."
}

GET _analyze
{
"analyzer": "english",
"text": "2 Six started far-placing saw respect females old. Civilly why how end viewing attempt related enquire visitor. Man particular insensible celebrated conviction stimulated principles day."
}

GET _analyze
{
"analyzer": "icu_analyzer",
"text": "他說的的確在理“"
}

GET _analyze
{
"analyzer": "standard",
"text": "他說的的確在理“"
}

search api

  • URL Search 使用 q
  • Request Body search
1
2
3
/_search

/index1/_search

URL Search

  • q
  • df 默认字段
  • sort
  • profile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
GET /movies/_search?q=2012&df=title
{
"profile":"true"
}

//泛查询
GET /movies/_search?q=2012
{
"profile":"true"
}

//指定查询
GET /movies/_search?q=title:2012
{
"profile":"true"
}

//使用引号
GET /movies/_search?q=title:"Beautiful Mind"
{
"profile":"true"
}

//查找美丽心灵,Mind 泛查询
GET /movies/_search?q=title:Beautiful Mind
{
"profile":"true"
}

//分组,bool查询
GET /movies/_search?q=title:(Beautiful Mind)
{
"profile":"true"
}


//查找美丽心灵
GET /movies/_search?q=title:(Beautiful AND Mind)
{
"profile":"true"
}


GET /movies/_search?q=title:(Beautiful NOT Mind)
{
"profile":"true"
}

//%2B 加号
GET /movies/_search?q=title:(Beautiful %2BMind)
{
"profile":"true"
}

GET /movies/_search?q=year:>=1990
{
"profile":"true"
}

GET /movies/_search?q=title:b*
{
"profile":"true"
}

GET /movies/_search?q=title:beautiful~1
{
"profile":"true"
}

GET /movies/_search?q=title:"Lord Rings"~2
{
"profile":"true"
}

Request body search

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
POST kibana_sample_data_ecommerce/_search
{
"_source": ["order_date"],
"sort": [
{
"order_date": {
"order": "desc"
}
}
],
"query": {
"match_all": {}
}
}

//脚本字段
GET kibana_sample_data_ecommerce/_search
{
"script_fields": {
"new_field": {
"script": {
"lang":"painless",
"source": "doc['order_date'].value+'_hello'"
}
}
},
"query": {
"match_all": {}
}
}

POST movies/_search
{
"query": {
"match": {
"title": "Last Christmas"
}
}
}

POST movies/_search
{
"query": {
"match": {
"title": {
"query": "Last Christmas",
"operator": "AND"
}

}
}
}

POST movies/_search
{
"query": {
"match_phrase": {
"title": {
"query": "one love"
}

}
}
}

POST movies/_search
{
"query": {
"match_phrase": {
"title": {
"query": "one love",
"slop": 1
}

}
}
}

query_string

simple query string query