Elasticsearch篇之Search API介绍
1 SearchAPI概览
实现对es中存储的数据进行查询分析,endpoint为_search,如下所示:
查询主要有两种形式
2 URISearch详解与演示
通过url query参数来实现搜索,常用参数如下:
- q指定查询的语句,语法为Query String Syntax
- df q中不指定字段时默认查询的字段,如果不指定,es会查询所有字段
- sort 排序
- timeout 指定超时时间,默认不超时
- from、size 用于分页
- term 与 phrase
- alfred way 等效于 alfred or way
- “alfred way”词语查询,要求先后排序
- 泛查询
- alfred 等效于在所有字段中去匹配该term
- 指定字段
- name:aflred
- Group分组设定,使用括号指定匹配的规则
- (quick OR brown)AND fox
- status:(active OR pending) title:(full text search)
创建索引,生成测试文档
PUT my_index_search
{
"settings":
{
"number_of_shards": "5",
"number_of_replicas": "0"
}
}
POST my_index_search/doc/_bulk
{"index":{"_id": "1"}}
{"username": "alfred way","job": "java engineer","age": 18,"birth": "1990-01-02","isMarried":false}
{"index":{"_id": "2"}}
{"username": "alfred","job": "java senior and java specialist","age": 28,"birth": "1980-05-07","isMarried":true}
{"index":{"_id": "3"}}
{"username": "lee","job": "java and ruby engineer","age": 22,"birth": "1985-08-07","isMarried":false}
{"index":{"_id": "4"}}
{"username": "alfred junior way","job": "ruby engineer","age": 23,"birth": "1989-08-02","isMarried":false}
# 查询所有字段中有alfred的文档
GET my_index_search/_search?q=alfred
# 设置profile可以看具体的查询语句
GET my_index_search/_search?q=alfred
{
"profile": true
}
GET my_index_search/_search?q=username:alfred
GET my_index_search/_search?q=username:alfred
{
"profile": true
}
# username:alfred和way是OR的关系
GET my_index_search/_search?q=username:alfred way
{
"profile": true
}
# PhraseQuery词语的查询
GET my_index_search/_search?q=username:"alfred way"
{
"profile": true
}
# "description": "username:alfred username:way" 下面描述
GET my_index_search/_search?q=username:(alfred way)
{
"profile": true
}
- 布尔操作符
- AND(&&) OR(||) NOT(!)
- name:(tom NOT lee)
- 注意大写,不能小写
- + - 分别对应must和must_not
- name:(tom +lee -alfred) 或者 name:((lee && !alfred)||(tome && lee && !alfred))
- + 在url中会被解析为空格,要使用encode后才可以,为%2B
GET my_index_search/_search?q=username:alfred AND way
{
"profile": true
}
GET my_index_search/_search?q=username:(alfred AND way)
{
"profile": true
}
GET my_index_search/_search?q=username:(alfred NOT way)
{
"profile": true
}
GET my_index_search/_search?q=username:(alfred +way)
{
"profile": true
}
GET my_index_search/_search?q=username:(alfred %2Bway)
{
"profile": true
}
- 范围查询,支持数值和日期
- 区间写法,闭区间用[],开区间{}
- age:[1 TO 10]意为 1<=age<=10
- age:[1 TO 10}意为 1<=age<10
- age:[1 TO]意为 age>=1
- age:{* TO 10]意为age<=10
- 算术符号写法
- age:>=1
- age:(>=1 && <=10)或者 age:(+>=1 +<=10)
- 区间写法,闭区间用[],开区间{}
GET my_index_search/_search?q=username:alfred age:>20
GET my_index_search/_search?q=username:alfred AND age:>20
GET my_index_search/_search?q=birth:(>1980 AND <1990)
- 通配符查询
- ? 代表1个字符, * 代表0或多个字符
- name:t?m
- name:tom*
- name:t*m
- 通配符匹配执行效率低,且占用较多内存,不建议使用
- 如无特殊需求,不要将?/ * 放在最前面
GET my_index_search/_search?q=username:alf*
- 正则表达式匹配
GET my_index_search/_search?q=username:/[a]?l.*/
- 模糊匹配fuzzy query
- name:roam~1
- 匹配roam差1个character的词,比如foam、roams等
- 近似度查询proximity search
- “fox quick”~5
- 以term为单位进行差异比较,比如“quick fox” “quick brown fox”都会被匹配
GET my_index_search/_search?q=username:alfed~1
GET my_index_search/_search?q=job:"java engineer"~2
3 QueryDSL简介
将查询语句通过http request body发送到es,主包含如下参数:
- query符合Query DSL 语法的查询语句
- from、size
- timeout
- sort
- …
- 基于JSON定义的查询语言,主要包含如下两种类型:
- 字段类查询
- 如term、math、range等,只针对某一个字段进行查询
- 符合查询
- 如bool查询等、包含一个或多个字段类型查询或者复合查询语句
- 字段类查询
4 字段类查询简介及match-query
-
字段类查询主要包括以下两类:
-
全文匹配
- 针对text类型的字段进行全文检索,会对查询语句先进行分词处理,如match、match_phrase等query类型
-
单词匹配
- 不会对查询语句做分词处理,直接去匹配字段的倒排索引,如term、terms、range等query类型
-
对字段作全文检索,最基本和常用的查询类型,API实例如下:
GET my_index_search/_search
{
"query": {
"match": {
"username": "alfred way"
}
}
}
# 查看查询语句
GET my_index_search/_search
{
"profile": true,
"query": {
"match": {
"username": "alfred way"
}
}
}
- 通过operator参数可以控制单词间的匹配关系,可选项为or和and
GET my_index_search/_search
{
"profile": true,
"query": {
"match": {
"username": {
"query": "alfred way",
"operator": "and"
}
}
}
}
- 通过minmun_should_match参数可以控制需要匹配的单词数
GET my_index_search/_search
{
"profile": true,
"query": {
"match": {
"job": {
"query": "java ruby engineer",
"minimum_should_match": 2
}
}
}
}
5 相关性算分
- 相关性算分是指文档与查询语句间的相关度,英文为relevance
- 通过倒排索引可以获取与查询语句相匹配的文档列表
- 本质是一个排序问题,排序的依据是相关性算分
- 相关性算分的几个重要概念如下:
- Term Frequency(TF)词频,即单词在该文档中出现的次数。词频越高,相关度越高
- Document Frequency(DF)文档频率,即单词出现的文档树
- Inverse Document Frequency(IDF)逆向文档频率,与文档频率相反,简单理解为1/DF。即单词出现的文档数越少,相关度越高
- Field-length Norm文档越短,相关度越高
- ES目前主要有两个相关性算分模型,例如:
- TF/IDF模型
- BM25模型 5.x之后的默认模型
- TF/IDF模型
6 match-phrase-query
- 对字段检索,有顺序要求,API示例如下
GET my_index_search/_search
{
"profile": true,
"query": {
"match_phrase": {
"job": {
"query": "java engineer"
}
}
}
}
GET my_index_search/_search
{
"profile": true,
"query": {
"match_phrase": {
"job": {
"query": "engineer java"
}
}
}
}
GET my_index_search/_search
{
"profile": true,
"query": {
"match_phrase": {
"job": {
"query": "java engineer",
"slop": 1
}
}
}
}
GET my_index_search/_search
{
"profile": true,
"query": {
"match_phrase": {
"job": {
"query": "java engineer",
"slop": 2
}
}
}
}
7 query-string-query
GET my_index_search/_search
{
"query": {
"query_string": {
"default_field": "username",
"query": "alfred AND way"
}
}
}
GET my_index_search/_search
{
"profile": true,
"query": {
"query_string": {
"fields": [
"username",
"job"
],
"query": "alfred OR (java AND ruby)"
}
}
}
8 simple-query-string-query
- 类似Query String,但是会忽略错误的查询语法,并且仅支持部分查询语法
- 其常用的逻辑符号如下、不能使用AND、OR、NOT等关键词:
- + 代指AND
- | 代指OR
- - 代指NOT
# 必须包含away,可以包含alfred
GET my_index_search/_search
{
"profile": true,
"query": {
"simple_query_string": {
"query": "alfred +way",
"fields": ["username"]
}
}
}
GET my_index_search/_search
{
"profile": true,
"query": {
"simple_query_string": {
"query": "alfred +way AND java",
"fields": ["username"]
}
}
}
query_string和simple_query_string的对比
扫描二维码关注公众号,回复:
8583856 查看本文章
GET my_index_search/_search
{
"profile": true,
"query": {
"query_string": {
"fields": ["username"],
"query": "alfred OR (\"java AND ruby)"
}
}
}
GET my_index_search/_search
{
"profile": true,
"query": {
"simple_query_string": {
"query": "alfred +way AND \"java",
"fields": ["username"]
}
}
}
9 term/terms-query
- term-query将查询语句作为整个单词进行查询,即不对查询语句做分词处理,如下所示:
- terms-query一次传入多个单词进行查询,如下所示:
# term query
GET my_index_search/_search
{
"profile": true,
"query": {
"term": {
"username": "alfred"
}
}
}
GET my_index_search/_search
{
"profile": true,
"query": {
"term": {
"username": "alfred way"
}
}
}
# terms query
GET my_index_search/_search
{
"profile": true,
"query": {
"terms": {
"username": [
"alfred",
"way"
]
}
}
}
10 range-query
GET my_index_search/_search
{
"query": {
"range": {
"age": {
"gte": 10,
"lte": 20
}
}
}
}
GET my_index_search/_search
{
"query": {
"range": {
"birth": {
"gte": "1980-01-01"
}
}
}
}
GET my_index_search/_search
{
"query": {
"range": {
"birth": {
"gte": "now-35y"
}
}
}
}