文件目录
bin:存放脚本文件、包括启动elasticseaerch,安装插件。运行统计数据等。
config: 配置文件 - elasticsearch.yml,集群配置文件,user、role、based相关配置。
jdk:java运行环境
data:path.data,数据文件
lib:Java类库
logs:path.log,日志文件
modules:包含所有es模块
plugins:包含所有已安装插件
Xms和Xmx设置成一样
# Xms 表示总堆空间的初始大小
# Xmx represents the maximum size of total heap space 表示总堆空间的最大大小
-Xms2g
-Xmx2g
Xmx不要超过机器内存的50%
控制台提示
future versions of Elasticsearch will require Java 11; your Java version fro
你的java版本不是11,当前elasticsearch 需要使用11版本的。
在es的bin目录下elasticsearch-env文件中修改默认的JAVA_HOME(window是elasticsearch-env.bat文件,),在43行。最好使用elasticsearch自带的 jdk 就ok了。
当前默认的JAVA_HOME是不是jdk11。
window 解决办法:
在if的前一行 自定义设置JAVA_HOME的值,为 jdk11版本的。
rem compariing to empty string makes this equivalent to bash -v check on env var
rem and allows to effectively force use of the bundled jdk when launching ES
rem by setting JAVA_HOME=
set JAVA_HOME=G:\elasticesearch\elasticsearch-7.9.2\jdk
if "%JAVA_HOME%" == "" (
set JAVA="%ES_HOME%\jdk\bin\java.exe"
set JAVA_HOME="%ES_HOME%\jdk"
set JAVA_TYPE=bundled jdk
) else (
set JAVA="%JAVA_HOME%\bin\java.exe"
set JAVA_TYPE=JAVA_HOME
)
在bin目录下使用命令行模式
启动ES
elasticsearch.bat
浏览器访问: http://127.0.0.1:9200
{
"name": "NIANSHAO-PC",
"cluster_name": "elasticsearch",
"cluster_uuid": "vkfjE0SjSYWJJRN57ITZLg",
"version": {
"number": "7.9.2",
"build_flavor": "default",
"build_type": "zip",
"build_hash": "d34da0ea4a966c4e49417f2da2f244e3e97b4e6e",
"build_date": "2020-09-23T00:45:33.626720Z",
"build_snapshot": false,
"lucene_version": "8.6.2",
"minimum_wire_compatibility_version": "6.8.0",
"minimum_index_compatibility_version": "6.0.0-beta1"
},
"tagline": "You Know, for Search"
}
elasticsearch-plugin install analysis-icu
查看
G:\elasticesearch\elasticsearch-7.9.2\bin>elasticsearch-plugin list
analysis-icu
node.name : 节点名称
cluster.name: 集群的名字
path.data: 存放数据的配置
elasticsearch -E node.name=node1 -E cluster.name=geektime -E path.data=node1_data -d
elasticsearch -E node.name=node2 -E cluster.name=geektime -E path.data=node2_data -d
elasticsearch -E node.name=node3 -E cluster.name=geektime -E path.data=node3_data -d
查看集群节点
http://127.0.0.1:9200/_cat/nodes
下载地址:https://www.elastic.co/cn/downloads/kibana
kibana依赖于elasticsearch运行的
访问 http://127.0.0.1:5601
首页
修改Kibana为中文版的
vi config/kibana.yml
# 结尾加上
i18n.locale: "zh-CN"
kibana-plugin install ...
前提是已经安装好docker
-L:有的网址是自动跳转的,可以自动跳转到新的网址
-o:保存成文件
curl -L https://github.com/docker/compose/releases/download/1.27.4/docker-compose-Linux-x86_64 -O /usr/local/bin/docker-compose
将文件下载,保存到/usr/local/bin目录下,并命名为docker-compose
然后授予该文件 执行权限
chmod +x /usr/local/bin/docker-compose
就可以使用docker-compose命令了。
docker-compose --version
docker-compose -h # 查看帮助 docker-compose up # 创建并运行所有容器 docker-compose up -d # 创建并后台运行所有容器 docker-compose -f docker-compose.yml up -d # 指定模板 docker-compose down # 停止并删除容器、网络、卷、镜像。 docker-compose logs # 查看容器输出日志 docker-compose pull # 拉取依赖镜像 dokcer-compose config # 检查配置 dokcer-compose config -q # 检查配置,有问题才有输出 docker-compose restart # 重启服务 docker-compose start # 启动服务 docker-compose stop # 停止服务
创建docker-compose.yml文件
version: '2.2'
services:
cerebro:
image: lmenezes/cerebro:0.8.3
container_name: cerebro
ports:
- "9000:9000"
command:
- -Dhosts.0.host=http://elasticsearch:9200
networks:
- es72net
kibana:
image: kibana:7.2.0
container_name: kibana72
environment:
#- I18N_LOCALE=zh-CN
- XPACK_GRAPH_ENABLED=true
- TIMELION_ENABLED=true
- XPACK_MONITORING_COLLECTION_ENABLED="true"
ports:
- "5601:5601"
networks:
- es72net
elasticsearch:
image: elasticsearch:7.2.0
container_name: es72_01
environment:
- cluster.name=geektime
- node.name=es72_01
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- discovery.seed_hosts=es72_01,es72_02
# - network.publish_host=elasticsearch
- cluster.initial_master_nodes=es72_01,es72_02
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es72data1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- es72net
elasticsearch2:
image: elasticsearch:7.2.0
container_name: es72_02
environment:
- cluster.name=geektime
- node.name=es72_02
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- discovery.seed_hosts=es72_01,es72_02
# - network.publish_host=elasticsearch
- cluster.initial_master_nodes=es72_01,es72_02
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es72data2:/usr/share/elasticsearch/data
networks:
- es72net
volumes:
es72data1:
driver: local
es72data2:
driver: local
networks:
es72net:
driver: bridge
执行命令
docker-compose up -d
访问
这时候,一般elasticsearch 启动后,然后报错,停止运行。
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
elasticsearch用户拥有的内存权限太小,至少需要262144;
执行命令:
sysctl -w vm.max_map_count=262144
查看结果:
sysctl -a|grep vm.max_map_count
显示:
vm.max_map_count = 262144
上述方法修改之后,如果重启虚拟机将失效,所以:
解决办法:
在 /etc/sysctl.conf文件最后添加一行
vm.max_map_count=262144
即可永久修改
vim /etc/security/limits.conf
在最后面追加下面内容
* hard nofile 65536
* soft nofile 65536
修改成功之后,别忘了 重启虚拟机。
配置sincedb_path让logstash每次都从头读取文件
input {
file {
path => "G:/elasticesearch/logstash-7.9.2/bin/movies.csv"
start_position => "beginning"
sincedb_path => "G:/elasticesearch/logstash-7.9.2/bin/null"
}
}
filter {
csv {
separator => ","
columns => ["id","content","genre"]
}
mutate {
split => { "genre" => "|" }
remove_field => ["path", "host","@timestamp","message"]
}
mutate {
split => ["content", "("]
add_field => { "title" => "%{[content][0]}"}
add_field => { "year" => "%{[content][1]}"}
}
mutate {
convert => {
"year" => "integer"
}
strip => ["title"]
remove_field => ["path", "host","@timestamp","message","content"]
}
}
output {
elasticsearch {
hosts => "http://127.0.0.1:9200"
index => "movies"
document_id => "%{id}"
}
stdout {}
}
./logstatsh.bat -f ./logstatsg.conf
索引的不同语义:
抽象与类比:
关系型数据库 | Elasticsearch |
---|---|
Table | Index(Type) |
Row | Document |
Column | Field |
Schema(表定义) | Mapping |
SQL | DSL |
查看索引相关信息 类似于 表结构
GET movies
{ "movies" : { "aliases" : { }, "mappings" : { "properties" : { "@version" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "genre" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "id" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "title" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "year" : { "type" : "long" } } }, "settings" : { "index" : { "creation_date" : "1602338832368", "number_of_shards" : "1", "number_of_replicas" : "1", "uuid" : "q6TgPWZ3SRidhDUPkvAPeg", "version" : { "created" : "7020099" }, "provided_name" : "movies" } } } }
查看索引文档的总数 文档类似于 表中的 行
GET movies/_count
{ "count" : 9743, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 } }
查看前10条文档,了解文档格式
POST movies/_search
{ "took" : 9, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 9743, "relation" : "eq" }, "max_score" : 1.0, "hits" : [ { "_index" : "movies", "_type" : "_doc", "_id" : "579", "_score" : 1.0, "_source" : { "year" : 0, "genre" : [ "Crime", "Thriller" ], "title" : "Escort, The", "@version" : "1", "id" : "579" } }, { "_index" : "movies", "_type" : "_doc", "_id" : "580", "_score" : 1.0, "_source" : { "year" : 1994, "genre" : [ "Drama" ], "title" : "Princess Caraboo", "@version" : "1", "id" : "580" } }, { "_index" : "movies", "_type" : "_doc", "_id" : "581", "_score" : 1.0, "_source" : { "year" : 1995, "genre" : [ "Documentary" ], "title" : "Celluloid Closet, The", "@version" : "1", "id" : "581" } }, { "_index" : "movies", "_type" : "_doc", "_id" : "583", "_score" : 1.0, "_source" : { "year" : 0, "genre" : [ "Comedy", "Drama" ], "title" : "Dear Diary", "@version" : "1", "id" : "583" } }, { "_index" : "movies", "_type" : "_doc", "_id" : "585", "_score" : 1.0, "_source" : { "year" : 1995, "genre" : [ "Comedy" ], "title" : "Brady Bunch Movie, The", "@version" : "1", "id" : "585" } }, { "_index" : "movies", "_type" : "_doc", "_id" : "586", "_score" : 1.0, "_source" : { "year" : 1990, "genre" : [ "Children", "Comedy" ], "title" : "Home Alone", "@version" : "1", "id" : "586" } }, { "_index" : "movies", "_type" : "_doc", "_id" : "587", "_score" : 1.0, "_source" : { "year" : 1990, "genre" : [ "Comedy", "Drama", "Fantasy", "Romance", "Thriller" ], "title" : "Ghost", "@version" : "1", "id" : "587" } }, { "_index" : "movies", "_type" : "_doc", "_id" : "588", "_score" : 1.0, "_source" : { "year" : 1992, "genre" : [ "Adventure", "Animation", "Children", "Comedy", "Musical" ], "title" : "Aladdin", "@version" : "1", "id" : "588" } }, { "_index" : "movies", "_type" : "_doc", "_id" : "589", "_score" : 1.0, "_source" : { "year" : 1991, "genre" : [ "Action", "Sci-Fi" ], "title" : "Terminator 2: Judgment Day", "@version" : "1", "id" : "589" } }, { "_index" : "movies", "_type" : "_doc", "_id" : "590", "_score" : 1.0, "_source" : { "year" : 1990, "genre" : [ "Adventure", "Drama", "Western" ], "title" : "Dances with Wolves", "@version" : "1", "id" : "590" } } ] } }
cat api
indices : 是索引(index)的复数形式。
查看indices 对索引的名字 进行通配符的查询
GET /_cat/indices/kibana*?&s=index
green open kibana_sample_data_flights -AAiAITRS-q9CiwxglY5cQ 1 1 13059 0 12.7mb 6.3mb
查看状态为绿的索引
GET /_cat/indices?v&health=green
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open .kibana_task_manager Phv_k8FHRuu4Rw-pu9zn1g 1 1 2 0 92.6kb 46.3kb green open kibana_sample_data_flights -AAiAITRS-q9CiwxglY5cQ 1 1 13059 0 12.7mb 6.3mb green open movies q6TgPWZ3SRidhDUPkvAPeg 1 1 9743 0 2.7mb 1.3mb green open .kibana_1 EndCRzm-TKOSvWmUt1nVFA 1 1 70 0 198.4kb 103.8kb
按照文档 的个数排序 (按照文档的个数,降序排列)
GET /_cat/indices?v&s=docs.count:desc
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open kibana_sample_data_flights -AAiAITRS-q9CiwxglY5cQ 1 1 13059 0 12.7mb 6.3mb green open movies q6TgPWZ3SRidhDUPkvAPeg 1 1 9743 0 2.7mb 1.3mb green open .kibana_1 EndCRzm-TKOSvWmUt1nVFA 1 1 70 0 198.4kb 103.8kb green open .kibana_task_manager Phv_k8FHRuu4Rw-pu9zn1g 1 1 2 0 92.6kb 46.3kb
查看具体的字段
GET /_cat/indices/kibana*?pri&v&h=health,index,pri,rep,docs,count,mt
health index pri rep mt pri.mt green kibana_sample_data_flights 1 1 0 0
eligible:合格的;具备条件的;合适的;有资格的。
http://localhost:9200/_cluster/health
http://localhost:9200/_cat/nodes
查看健康状态
GET _cluster/health
状态是 绿色,2个节点{ "cluster_name" : "geektime", "status" : "green", "timed_out" : false, "number_of_nodes" : 2, // 有两个节点 "number_of_data_nodes" : 2, // 2个节点 承担的 data node节点 "active_primary_shards" : 4, // 4个主分片 "active_shards" : 8, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }
查看节点信息
GET _cat/nodes
172.19.0.4 27 96 8 0.08 0.13 0.20 mdi * es72_01 172.19.0.5 22 96 8 0.08 0.13 0.20 mdi - es72_02
查看分片信息
GET _cat/shards
.kibana_task_manager 0 p STARTED 2 46.3kb 172.19.0.4 es72_01 .kibana_task_manager 0 r STARTED 2 46.3kb 172.19.0.5 es72_02 kibana_sample_data_flights 0 p STARTED 13059 6.3mb 172.19.0.4 es72_01 kibana_sample_data_flights 0 r STARTED 13059 6.3mb 172.19.0.5 es72_02 movies 0 p STARTED 9743 1.3mb 172.19.0.4 es72_01 movies 0 r STARTED 9743 1.3mb 172.19.0.5 es72_02 .kibana_1 0 p STARTED 70 103.8kb 172.19.0.4 es72_01 .kibana_1 0 r STARTED 70 94.5kb 172.19.0.5 es72_02