Spring Boot整合Elasticsearch
Elasticsearch版本7.10.2
Spring Boot版本2.4.2
使用Spring Boot操作Elasticsearch有两种方式，一种是Spring Data Repositories的方式，一种是使用ElasticsearchRestTemplate方式。
Spring Data Repositories方式
先放Spring Boot文档https://docs.spring.io/spring-data/elasticsearch/docs/4.1.3/reference/html/#repositories.core-concepts。
编写实体类
编写实体类主要会用到的注解
-
@Document（必写）
属性名 说明 indexName 索引名，支持SpEl type deprecated since Elasticsearch 4.0 shards 分片 replicas 每个分区备份数 refreshIntervall 刷新间隔，默认1s indexStoreType 索引文件存储类型，默认fs versionType 配置版本管理，默认EXTERNAL @Id（必写）
-
@Field
属性名 说明 name 将在Elasticsearch文档中表示的字段名称，如果未设置，则使用Java字段名称 type 属性类型 store 标记是否原始字段值应存储在Elasticsearch中，默认值为false。
demo
@Data
@Document(indexName = "stu", shards = 3, replicas = 0)
public class StuEntity {
@Id
private Long stuId;
@Field(store = true)
private String name;
@Field(store = true)
private Integer age;
@Field(store = true, type = FieldType.Keyword)
private String sign;
@Field(store = true)
private String description;
}
编写Repository
编写一个借口继承ElasticsearchRepository<T, ID>
public interface StuEntityRepository extends ElasticsearchRepository<Book, String> {
List<Book> findAllByName(String name);
}
注意：ElasticsearchRepository接口中的方式都是@Deprecated
编写curd
编写curd有两种：关键字拼接查询条件、@Query注解查询
关键字拼接查询条件
和关系型数据库一样直接使用就好，Elasticsearch有特殊的关键字，用到了自己查询官方文档。
| Keyword | Sample | Elasticsearch Query String |
|---|---|---|
And |
findByNameAndPrice |
{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } }, { "query_string" : { "query" : "?", "fields" : [ "price" ] } } ] } }} |
Or |
findByNameOrPrice |
{ "query" : { "bool" : { "should" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } }, { "query_string" : { "query" : "?", "fields" : [ "price" ] } } ] } }} |
Is |
findByName |
{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } } ] } }} |
Not |
findByNameNot |
{ "query" : { "bool" : { "must_not" : [ { "query_string" : { "query" : "?", "fields" : [ "name" ] } } ] } }} |
Between |
findByPriceBetween |
{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] } }} |
LessThan |
findByPriceLessThan |
{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : false } } } ] } }} |
LessThanEqual |
findByPriceLessThanEqual |
{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] } }} |
GreaterThan |
findByPriceGreaterThan |
{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : false, "include_upper" : true } } } ] } }} |
GreaterThanEqual |
findByPriceGreaterThan |
{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : true, "include_upper" : true } } } ] } }} |
Before |
findByPriceBefore |
{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : null, "to" : ?, "include_lower" : true, "include_upper" : true } } } ] } }} |
After |
findByPriceAfter |
{ "query" : { "bool" : { "must" : [ {"range" : {"price" : {"from" : ?, "to" : null, "include_lower" : true, "include_upper" : true } } } ] } }} |
Like |
findByNameLike |
{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }} |
StartingWith |
findByNameStartingWith |
{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }} |
EndingWith |
findByNameEndingWith |
{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "*?", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }} |
Contains/Containing |
findByNameContaining |
{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "*?*", "fields" : [ "name" ] }, "analyze_wildcard": true } ] } }} |
In (when annotated as FieldType.Keyword) |
findByNameIn(Collection<String>names) |
{ "query" : { "bool" : { "must" : [ {"bool" : {"must" : [ {"terms" : {"name" : ["?","?"]}} ] } } ] } }} |
In |
findByNameIn(Collection<String>names) |
{ "query": {"bool": {"must": [{"query_string":{"query": "\"?\" \"?\"", "fields": ["name"]}}]}}} |
NotIn (when annotated as FieldType.Keyword) |
findByNameNotIn(Collection<String>names) |
{ "query" : { "bool" : { "must" : [ {"bool" : {"must_not" : [ {"terms" : {"name" : ["?","?"]}} ] } } ] } }} |
NotIn |
findByNameNotIn(Collection<String>names) |
{"query": {"bool": {"must": [{"query_string": {"query": "NOT(\"?\" \"?\")", "fields": ["name"]}}]}}} |
Near |
findByStoreNear |
Not Supported Yet ! |
True |
findByAvailableTrue |
{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "true", "fields" : [ "available" ] } } ] } }} |
False |
findByAvailableFalse |
{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "false", "fields" : [ "available" ] } } ] } }} |
OrderBy |
findByAvailableTrueOrderByNameDesc |
{ "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "true", "fields" : [ "available" ] } } ] } }, "sort":[{"name":{"order":"desc"}}] } |
@Query注解查询
public interface StuEntityRepository extends ElasticsearchRepository<StuEntity, Long> {
List<StuEntity> findAllByName(String name);
@Query("{\"match\": {\"name\": {\"query\": \"?0\"}}}")
Page<StuEntity> findByName(String name, Pageable pageable);
}
ElasticsearchRestTemplate
ElasticsearchRestTemplate是在spring data 中操作Elasticsearch的模板类，其中实现了对Elasticsearch 操作的各类操作方法。例如创建索引、创建别名、创建映射，以及数据的查询和其他操作。
放官方文档：https://docs.spring.io/spring-data/elasticsearch/docs/4.1.3/reference/html/#elasticsearch.operations
主要有以下操作
-
IndexOperations定义索引级别的操作，如创建或删除索引。 -
DocumentOperations定义基于实体 ID 存储、更新和检索实体的操作。 -
SearchOperations定义使用查询搜索多个实体的操作
image-20210218172418556.png
CURD
下面demo中的StuEntity为我们之前创建的那个类
-
新增
@Autowired private ElasticsearchRestTemplate elasticsearchRestTemplate; @Test public void add() { StuEntity stu = new StuEntity(); stu.setStuId(1005L); stu.setName("iron man"); stu.setAge(54); stu.setSign("I am iron man"); stu.setDescription("I have a iron army"); elasticsearchRestTemplate.save(stu); }直接调用
elasticsearchRestTemplate中DocumentOperations的save方法就可以了，save方法有几个重载方法，详情看api或源码 -
修改
@Test public void update() { StuEntity stu = new StuEntity(); stu.setStuId(1005L); stu.setName("iron manss"); stu.setAge(100); stu.setSign("I am iron man"); stu.setDescription("I have a iron army"); System.out.println(JSON.toJSONString(stu)); // 创建Document对象 // 第一种方式 Document document = Document.create(); // 将修改的内容塞进去 document.putAll(JSON.parseObject(JSON.toJSONString(stu), Map.class)); // 第二种方式 Document document1 = Document.parse(JSON.toJSONString(stu)); // 第三种方式 Document document2 = Document.from(JSON.parseObject(JSON.toJSONString(stu), Map.class)); // 构造updateQuery UpdateQuery updateQuery = UpdateQuery.builder("1") // 如果不存在就新增，默认为false .withDocAsUpsert(true) .withDocument(Document.parse(JSON.toJSONString(stu))) .build(); elasticsearchRestTemplate.update(updateQuery, IndexCoordinates.of("stu")); }直接调用
elasticsearchRestTemplate中DocumentOperations的update方法，详情看api或者源码 -
删除
@Test public void delete() { StuEntity stu = new StuEntity(); stu.setStuId(1005L); stu.setName("iron man"); stu.setAge(54); stu.setSign("I am iron man"); stu.setDescription("I have a iron army"); elasticsearchRestTemplate.delete(stu); }直接调用
elasticsearchRestTemplate中DocumentOperations的delete方法，delete有好几个重载方法，具体使用哪个详情看api或者源码 -
查询
查询就调用
elasticsearchRestTemplate中SearchOperations`的search方法。在search的各种方法中都需要传入
Query。Spring Data Elasticsearch中Query的实现类CriteriaQuery,StringQueryandNativeSearchQuery
CriteriaQuery基于查询的查询允许创建查询来搜索数据，而无需了解 Elasticsearch 查询的语法或基础知识。它们允许用户通过简单地链接和组合指定搜索文档必须满足的条件的对象来生成查询StringQuery使用json字符串来构建查询条件。就和Repository中@Query注解中的那个json字符串一样。-
NativeSearchQuery用于复杂查询。使用
CriteriaQuery来构建查询@Test public void search1() { Criteria criteria = new Criteria("name").is("iron man"); Query query = new CriteriaQuery(criteria); SearchHits searchHits = elasticsearchRestTemplate.search(query, StuEntity.class); System.out.println(searchHits.getSearchHits()); }使用
StringQuery构建查询@Test public void search2() { Query query = new StringQuery("{\n" + " \"match\": { \n" + " \"age\": { \"query\": \"54\" } \n" + " } \n" + " }"); SearchHits<StuEntity> searchHits = elasticsearchRestTemplate.search(query, StuEntity.class); System.out.println(searchHits.getSearchHits()); }NativeSearchQuery查询网上的例子就很多了，以后再写篇文章写。
另类查询方式
使用elasticsearch-sql插件像写sql一样查询Elasticsearch。
jdbc方式
-
添加依赖
<dependency> <groupId>org.nlpcn</groupId> <artifactId>elasticsearch-sql</artifactId> <version>7.8.0.1</version> </dependency> <!-- https://mvnrepository.com/artifact/org.elasticsearch.client/x-pack-transport --> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>x-pack-transport</artifactId> <version>7.10.2</version> <exclusions> <exclusion> <artifactId>elasticsearch-core</artifactId> <groupId>org.elasticsearch</groupId> </exclusion> <exclusion> <artifactId>elasticsearch-ssl-config</artifactId> <groupId>org.elasticsearch</groupId> </exclusion> </exclusions> </dependency> <!-- https://mvnrepository.com/artifact/org.elasticsearch.plugin/x-pack-core --> <dependency> <groupId>org.elasticsearch.plugin</groupId> <artifactId>x-pack-core</artifactId> <version>7.10.2</version> </dependency> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>7.10.2</version> </dependency> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>transport</artifactId> <version>7.10.2</version> </dependency> <dependency> <groupId>org.elasticsearch.plugin</groupId> <artifactId>transport-netty4-client</artifactId> <version>7.10.2</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>druid</artifactId> <version>1.1.16</version> </dependency>Elasticsearch是7.10.2。现在elasticsearch-sql还没有7.10.2所以就用能用到的最新的7.8.0.1。druid是1.1.16。
@Test public void search4() throws Exception { Properties properties = new Properties(); properties.put("url", "jdbc:elasticsearch://192.168.1.123:9300/"); DruidDataSource dds = (DruidDataSource) ElasticSearchDruidDataSourceFactory.createDataSource(properties); Connection connection = dds.getConnection(); PreparedStatement ps = connection.prepareStatement("SELECT * from stu"); ResultSet resultSet = ps.executeQuery(); ps.close(); connection.close(); dds.close(); }当然为了查询方便，封装成NamedParameterJdbcTemplate。
api方式
这个方式是我在一次调试的时候发现的，官方文档中并没有介绍。仅供参考。
controller
@RestController
@RequestMapping("/elasticsearchSqlApiController")
public class ElasticsearchSqlApiController {
@Autowired
private ElasticsearchApiDao elasticsearchApiDao;
/**
* 使用es-sql插件的api方式查询sql
* @param sql 要查询的sql 例如：select * from student where name='小憨'
* @return
*/
@GetMapping("/search")
public ActionResult search(String sql) {
return ActionResult.success(elasticsearchApiDao.search(sql));
}
/**
* 将sql解析为 DSL语句
* @param sql
* @return
*/
@GetMapping("/explain")
public ActionResult explain(String sql) {
// String sql = "select * from a_icd_person where PERNAME='王晓光'";
return ActionResult.success(elasticsearchApiDao.explain(sql));
}
}
service
@Slf4j
@Service
public class ElasticsearchApiDaoImpl implements ElasticsearchApiDao {
@Autowired
private TransportClient transportClient;
@Override
public EsSearchResultDTO search(String sql) {
EsSearchResultDTO resultDTO = new EsSearchResultDTO();
try {
long before = System.currentTimeMillis();
SearchDao searchDao = new SearchDao(transportClient);
QueryAction queryAction = searchDao.explain(sql);
Object execution = QueryActionElasticExecutor.executeAnyAction(searchDao.getClient(), queryAction);
ObjectResult result = getObjectResult(execution, true, false, false, true, false, queryAction);
resultDTO.setResultColumns(Sets.newHashSet(result.getHeaders()));
List<IndexRowData> indexRowDatas = new ArrayList<>();
for (List<Object> line : result.getLines()) {
IndexRowData indexRowData = new IndexRowData();
for (int i = 0; i < result.getHeaders().size(); i++) {
indexRowData.build(result.getHeaders().get(i), line.get(i));
}
indexRowDatas.add(indexRowData);
}
resultDTO.setResultSize(indexRowDatas.size());
if (execution instanceof SearchHits) {
resultDTO.setTotal(((SearchHits) execution).getTotalHits());
} else {
resultDTO.setTotal(indexRowDatas.size());
}
resultDTO.setResult(indexRowDatas);
resultDTO.setTime((System.currentTimeMillis() - before) / 1000);
log.info("查询数据结果集: {}", JSONObject.toJSONString(resultDTO));
} catch (Exception e) {
throw new ElasticsearchException("根据ES-SQL查询数据异常: {}", e, e.getMessage());
}
return resultDTO;
}
/**
* 解析sql
* @param sql
* @return
*/
@Override
public String explain(String sql) {
SearchDao searchDao = new SearchDao(transportClient);
QueryAction queryAction = null;
try {
queryAction = searchDao.explain(sql);
return queryAction.explain().explain();
} catch (SqlParseException e) {
throw new RuntimeException(e);
} catch (SQLFeatureNotSupportedException e) {
throw new RuntimeException(e);
}
}
private ObjectResult getObjectResult(Object execution, boolean flat, boolean includeScore, boolean includeType, boolean includeId, boolean incluedScrollId, QueryAction queryAction) throws Exception {
return (new ObjectResultsExtractor(includeScore, includeType, includeId, incluedScrollId, queryAction)).extractResults(execution, flat);
}
}
配置类
@Slf4j
@Configuration
@PropertySource("classpath:elasticsearch.properties")
public class ElasticsearchConfig {
@Value("${elasticSearch.host}")
private String[] ipAddress;
@Value("${elasticSearch.maxRetryTimeout}")
private Integer maxRetryTimeout;
@Value("${elasticSearch.sql.host}")
private String[] esSqlAddress;
@Bean
public TransportClient transportClient() {
Settings settings = Settings.builder()
// 不允许自动刷新地址列表
.put("client.transport.sniff", false)
.put("client.transport.ignore_cluster_name", true)
.build();
// 初始化地址
TransportAddress[] transportAddresses = new TransportAddress[esSqlAddress.length];
for (int i = 0; i < esSqlAddress.length; i++) {
String[] addressItems = esSqlAddress[i].split(":");
try {
transportAddresses[i] = new TransportAddress(InetAddress.getByName(addressItems[0]),
Integer.valueOf(addressItems[1]));
} catch (UnknownHostException e) {
log.error(e.toString());
}
}
PreBuiltTransportClient preBuiltTransportClient = new PreBuiltTransportClient(settings);
TransportClient client = preBuiltTransportClient
.addTransportAddresses(transportAddreses);
return client;
}
}
最后：
sql的方式虽然不是Elasticsearch官方推荐的方式，但是上手快，会写sql就能写查询。Elasticsearch官方在6.3.0之后就开始支持sql方式了，但不是免费的，所以当时有个项目就用的这个elasticsearch-sql这个jar包。总体来说还行，但是有缺陷，基本的查询没有问题，但是高级操作就不如用官方推荐的了。这个仓库的star数从1k多到现在6.2k了。
