Basic usage of springboot springdataelasticsearch
introduce
Spring Data Elasticsearch is a sub module under the Spring Data project.
Check the official website of Spring Data: http://projects.spring.io/spring-data/
The mission of Spring Data is to provide a unified programming interface for various data access, whether it is a relational database (such as MySQL), a non relational database (such as Redis), or an index database such as Elasticsearch. So as to simplify the developer's code and improve the development efficiency.
Create project
Project structure
POM.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.4.RELEASE</version>
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.75</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
Configure the application.yaml file
spring:
data:
elasticsearch:
cluster-name: elasticsearch
cluster-nodes: 192.168.0.22:9300
Entity class and annotation
public class Item {
Long id;
String title; //title
String category;// classification
String brand; // brand
Double price; // Price
String images; // Picture address
}
Spring Data declares the mapping attributes of fields through annotations. There are three annotations:
-
@Document
It acts on the class and marks the entity class as a document object. It generally has four attributes
- indexName: the name of the corresponding index library (such as database)
- Type: corresponds to the type in the index library (such as database table)
- shards: the number of shards. The default value is 5
- Replicas: number of replicas. The default is 1
-
@id acts on the member variable and marks a field as the id primary key
-
@Field
Act on the member variable, mark it as the field of the document, and specify the field mapping attribute:
- Type: field type. The value is enumeration: FieldType
- Index: whether to index. Boolean type. The default is true
- Store: whether to store. Boolean type. The default is false
- analyzer: word breaker Name: ik_max_word
Example:
package cn.es.pojo;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;
@Document(indexName = "item",type = "docs")
@Data //Must have get set
@AllArgsConstructor //Must have full parameters
@NoArgsConstructor //There must be no tragedy
public class Item {
@Id
private Long id;
@Field(type = FieldType.Text, analyzer = "ik_max_word")
private String title; //title
@Field(type = FieldType.Keyword) //Keywor d does not participle
private String category;// classification
@Field(type = FieldType.Keyword)
private String brand; // brand
@Field(type = FieldType.Double)
private Double price; // Price
@Field(index = false, type = FieldType.Keyword)
private String images; // Picture address
}
Create index
@SpringBootTest
@RunWith(SpringRunner.class)
public class ElasticSearctTest {
@Autowired
private ElasticsearchTemplate elasticsearchTemplate;
//Create libraries (indexes (databases) and mappings (tables))
@Test
public void testCreate() {
// When creating an index, it will be created according to the @ Document annotation information of the Item class
elasticsearchTemplate.createIndex(Item.class);
// Configuring the mapping will automatically complete the mapping according to the id, Field and other fields in the Item class
elasticsearchTemplate.putMapping(Item.class);
}
}
Addition, deletion and modification
The strength of Spring Data is that you don't need to write any DAO processing, and automatically perform CRUD operations according to the method name or class information. As long as you define an interface and inherit some sub interfaces provided by the Repository, you can have various basic crud functions
Write ItemRepository
package com.example.elasticsearch.repository;
import com.example.elasticsearch.pojo.Item;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
/**
* @author john
* @date 2019/12/8 - 14:39
*/
public interface ItemRepository extends ElasticsearchRepository<Item,Long> {
}
increase
@Autowired
private ItemRepository itemRepository;
//insert data
@Test
public void testAdd() {
Item item = new Item(1L, "Xiaomi mobile phone 7", " mobile phone",
"millet", 3499.00, "http://image.leyou.com/13123.jpg");
itemRepository.save(item);
}
modify
//Modify (if id exists, it is essentially overwrite, otherwise it is insert)
@Test
public void testUpdate() {
Item item = new Item(1L, "Xiaomi mobile phone 7777", " mobile phone",
"millet", 9499.00, "http://image.leyou.com/13123.jpg");
itemRepository.save(item);
}
Batch add
@Autowired
private ItemRepository itemRepository;
@Test
public void indexList() {
List<Item> list = new ArrayList<>();
list.add(new Item(2L, "Nut phone R1", " mobile phone", "hammer", 3699.00, "http://image.leyou.com/123.jpg"));
list.add(new Item(3L, "Huawei META10", " mobile phone", "Huawei", 4499.00, "http://image.leyou.com/3.jpg"));
// Receive object collection to realize batch addition
itemRepository.saveAll(list);
}
Delete operation
@Autowired
private ItemRepository itemRepository;
@Test
public void testDelete() {
itemRepository.deleteById(1L);
}
Query by id
@Autowired
private ItemRepository itemRepository;
@Test
public void testQuery(){
Optional<Item> optional = itemRepository.findById(2L);
System.out.println(optional.get());
}
Query all and sort by price descending order
@Autowired
private ItemRepository itemRepository;
@Test
public void testFind(){
// Query all and sort by price descending order
Iterable<Item> items = this.itemRepository.findAll(Sort.by(Sort.Direction.DESC, "price"));
items.forEach(item-> System.out.println(item));
}
Custom method
Another powerful feature of Spring Data is the automatic implementation of functions based on method names.
For example, your method name is: findByTitle, then it will know that you query according to the title, and then automatically help you complete it without writing an implementation class.
Of course, the method name must comply with certain conventions:
| Keyword | Sample | Elasticsearch Query String |
|---|---|---|
| And | findByNameAndPrice | {"bool" : {"must" : [ {"field" : {"name" : "?"}}, {"field" : {"price" : "?"}} ]}} |
| Or | findByNameOrPrice | {"bool" : {"should" : [ {"field" : {"name" : "?"}}, {"field" : {"price" : "?"}} ]}} |
| Is | findByName | {"bool" : {"must" : {"field" : {"name" : "?"}}}} |
| Not | findByNameNot | {"bool" : {"must_not" : {"field" : {"name" : "?"}}}} |
| Between | findByPriceBetween | {"bool" : {"must" : {"range" : {"price" : {"from" : ?,"to" : ?,"include_lower" : true,"include_upper" : true}}}}} |
| LessThanEqual | findByPriceLessThan | {"bool" : {"must" : {"range" : {"price" : {"from" : null,"to" : ?,"include_lower" : true,"include_upper" : true}}}}} |
| GreaterThanEqual | findByPriceGreaterThan | {"bool" : {"must" : {"range" : {"price" : {"from" : ?,"to" : null,"include_lower" : true,"include_upper" : true}}}}} |
| Before | findByPriceBefore | {"bool" : {"must" : {"range" : {"price" : {"from" : null,"to" : ?,"include_lower" : true,"include_upper" : true}}}}} |
| After | findByPriceAfter | {"bool" : {"must" : {"range" : {"price" : {"from" : ?,"to" : null,"include_lower" : true,"include_upper" : true}}}}} |
| Like | findByNameLike | {"bool" : {"must" : {"field" : {"name" : {"query" : "?*","analyze_wildcard" : true}}}}} |
| StartingWith | findByNameStartingWith | {"bool" : {"must" : {"field" : {"name" : {"query" : "?*","analyze_wildcard" : true}}}}} |
| EndingWith | findByNameEndingWith | {"bool" : {"must" : {"field" : {"name" : {"query" : "*?","analyze_wildcard" : true}}}}} |
| Contains/Containing | findByNameContaining | {"bool" : {"must" : {"field" : {"name" : {"query" : "**?**","analyze_wildcard" : true}}}}} |
| In | findByNameIn(Collection<String>names) | {"bool" : {"must" : {"bool" : {"should" : [ {"field" : {"name" : "?"}}, {"field" : {"name" : "?"}} ]}}}} |
| NotIn | findByNameNotIn(Collection<String>names) | {"bool" : {"must_not" : {"bool" : {"should" : {"field" : {"name" : "?"}}}}}} |
| Near | findByStoreNear | Not Supported Yet ! |
| True | findByAvailableTrue | {"bool" : {"must" : {"field" : {"available" : true}}}} |
| False | findByAvailableFalse | {"bool" : {"must" : {"field" : {"available" : false}}}} |
| OrderBy | findByAvailableTrueOrderByNameDesc | {"sort" : [{ "name" : {"order" : "desc"} }],"bool" : {"must" : {"field" : {"available" : true}}}} |
For example, let's define a method to query by price range:
public interface ItemRepository extends ElasticsearchRepository<Item,Long> {
/**
* Query by price range
* @param price1
* @param price2
* @return
*/
List<Item> findByPriceBetween(double price1, double price2);
}
Then add some test data:
@Autowired
private ItemRepository itemRepository;
@Test
public void indexList_custom () { //Add some test data
List<Item> list = new ArrayList<>();
list.add(new Item(1L, "Xiaomi mobile phone 7", "mobile phone", "millet", 3299.00, "http://image.leyou.com/13123.jpg"));
list.add(new Item(2L, "Nut phone R1", "mobile phone", "hammer", 3699.00, "http://image.leyou.com/13123.jpg"));
list.add(new Item(3L, "Huawei META10", "mobile phone", "Huawei", 4499.00, "http://image.leyou.com/13123.jpg"));
list.add(new Item(4L, "millet Mix2S", "mobile phone", "millet", 4299.00, "http://image.leyou.com/13123.jpg"));
list.add(new Item(5L, "glory V10", "mobile phone", "Huawei", 2799.00, "http://image.leyou.com/13123.jpg"));
// Receive object collection to realize batch addition
itemRepository.saveAll(list);
}
There is no need to write an implementation class, call the interface, and then we run it directly:
@Autowired
private ItemRepository itemRepository;
@Test
public void queryByPriceBetween(){
List<Item> list = this.itemRepository.findByPriceBetween(2000.00, 3500.00);
for (Item item : list) {
System.out.println("item = " + item);
}
}
Although the basic query and user-defined methods are very powerful, it is not enough if it is a complex query (fuzzy, wildcard, entry query, etc.). At this point, we can only use native queries.
Advanced query
Let's look at the basic playing methods first
@Test
public void testBaseQuery(){
// Term query
MatchQueryBuilder queryBuilder = QueryBuilders.matchQuery("title", "millet");
// Execute query
Iterable<Item> items = this.itemRepository.search(queryBuilder);
items.forEach(System.out::println);
}
QueryBuilders provides a large number of static methods for generating various types of query objects, such as QueryBuilder objects such as entry, fuzzy, wildcard, etc.
Elastic search provides many available query methods, but it is not flexible enough. If you want to play filtering or aggregate query, it is difficult.
Paging query
// Paging query
@Test
public void testNativeQuery2(){
// Build query criteria
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
// Add basic word segmentation query
queryBuilder.withQuery(QueryBuilders.termQuery("category", "mobile phone"));
// Initialize paging parameters
int page = 0;
int size = 3;
// Set paging parameters
queryBuilder.withPageable(PageRequest.of(page, size));
// Perform a search to get results
Page<Item> items = this.itemRepository.search(queryBuilder.build());
// Total number of prints
System.out.println("Total number"+items.getTotalElements());
// Total pages printed
System.out.println("PageCount "+items.getTotalPages());
// Size per page
System.out.println("Size per page"+items.getSize());
// Current page
System.out.println("Current page"+items.getNumber());
items.forEach(System.out::println);
}
Native searchquerybuilder: a query condition builder provided by Spring to help build the request body in json format
Page < item >: the default is a paged query, so a paged result object is returned, including the following attributes:
- totalElements: total number of items
- totalPages: total pages
- Iterator: iterator, which implements the iterator interface, so it can directly iterate to get the data of the current page
- Other properties:
It can be found that the paging in elastic search starts from page 0.
sort
Sorting is also done through NativeSearchQueryBuilder:
//sort
@Test
public void testSort(){
// Build query criteria
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
// Add basic word segmentation query
queryBuilder.withQuery(QueryBuilders.termQuery("category", "mobile phone"));
// sort
queryBuilder.withSort(SortBuilders.fieldSort("price").order(SortOrder.DESC));
// Perform a search to get results
Page<Item> items = this.itemRepository.search(queryBuilder.build());
// Total number of prints
System.out.println(items.getTotalElements());
items.forEach(System.out::println);
}
polymerization
Aggregate into barrels
Barrels are grouped. For example, here we are grouped according to brand:
@Test
public void testAgg(){
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
// Do not query any results
queryBuilder.withSourceFilter(new FetchSourceFilter(new String[]{""}, null));
// 1. Add a new aggregation with the aggregation type of terms, the aggregation name of brands and the aggregation field of brand
queryBuilder.addAggregation(
AggregationBuilders.terms("brands").field("brand"));
// 2. Query, you need to force the result to the AggregatedPage type
AggregatedPage<Item> aggPage = (AggregatedPage<Item>) this.itemRepository.search(queryBuilder.build());
// 3. Analysis
// 3.1. Take the aggregation named brands from the results,
// Because it is term aggregation using String type fields, the result should be strongly converted to StringTerm type
StringTerms agg = (StringTerms) aggPage.getAggregation("brands");
// 3.2. Obtaining barrels
List<StringTerms.Bucket> buckets = agg.getBuckets();
// 3.3 traversal
for (StringTerms.Bucket bucket : buckets) {
// 3.4. Get the key in the bucket, that is, the brand name
System.out.println(bucket.getKeyAsString());
// 3.5. Get the number of documents in the bucket
System.out.println(bucket.getDocCount());
}
}
Nested aggregation, averaging
@Test
public void testSubAgg(){
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
// Do not query any results
queryBuilder.withSourceFilter(new FetchSourceFilter(new String[]{""}, null));
// 1. Add a new aggregation with the aggregation type of terms, the aggregation name of brands and the aggregation field of brand
queryBuilder.addAggregation(
AggregationBuilders.terms("brands").field("brand")
.subAggregation(AggregationBuilders.avg("priceAvg").field("price")) // Perform nested aggregation in the brand aggregation bucket and calculate the average value
);
// 2. Query, you need to force the result to the AggregatedPage type
AggregatedPage<Item> aggPage = (AggregatedPage<Item>) this.itemRepository.search(queryBuilder.build());
// 3. Analysis
// 3.1. Take the aggregation named brands from the results,
// Because it is term aggregation using String type fields, the result should be strongly converted to StringTerm type
StringTerms agg = (StringTerms) aggPage.getAggregation("brands");
// 3.2. Obtaining barrels
List<StringTerms.Bucket> buckets = agg.getBuckets();
// 3.3 traversal
for (StringTerms.Bucket bucket : buckets) {
// 3.4. Get the key in the bucket, i.e. brand name 3.5. Get the number of documents in the bucket
System.out.println(bucket.getKeyAsString() + "，common" + bucket.getDocCount() + "platform");
// 3.6. Obtain sub aggregation results:
InternalAvg avg = (InternalAvg) bucket.getAggregations().asMap().get("priceAvg");
System.out.println("Average selling price:" + avg.getValue());
}
}
Highlight query
//Highlight query (multi field query paging sort)
public List<Item> findByNameAndHighlightAdnPageable(String name, int page,int size) {
if(page <= 0){
page = 1;
}
if( size <= 0){
size = 5; //5 items are displayed on the lowest page
}
HighlightBuilder.Field nameField = new HighlightBuilder
.Field("*")
.preTags("<span style='color:red'>")
.postTags("</span>").requireFieldMatch(false);
//For multi field query, you can query the attribute name in the corresponding entity class in title and category at the same time. Note: highlight can only query text type fields, not numeric type fields
NativeSearchQuery nativeSearchQuery = new NativeSearchQueryBuilder()
.withQuery(QueryBuilders.multiMatchQuery(name, "title","category"))
.withPageable(PageRequest.of(page - 1, size)) //paging
.withSort(SortBuilders.fieldSort("price").order(SortOrder.DESC)) //sort
.withHighlightFields(nameField)
.build();
// In addition, other methods can be used in addition to QueryBuilders.multiMatchQuery
//For example, QueryBuilders.termQuery looks at your actual needs
AggregatedPage<Item> Items = elasticsearchTemplate.queryForPage(nativeSearchQuery, Item.class, new SearchResultMapper() {
@Override
public <T> AggregatedPage<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
SearchHits searchHits = response.getHits();
SearchHit[] hits = searchHits.getHits();
ArrayList<Item> Items = new ArrayList<Item>();
for (SearchHit hit : hits) {
Item Item = new Item();
//Data fields to be displayed in the original map (need to be modified following the entity class)
Map<String, Object> sourceAsMap = hit.getSourceAsMap();
Item.setId(Long.parseLong(sourceAsMap.get("id").toString()));
Item.setTitle(sourceAsMap.get("title").toString());
Item.setCategory(sourceAsMap.get("category").toString());
Item.setBrand(sourceAsMap.get("brand").toString());
Item.setPrice(Double.parseDouble(sourceAsMap.get("price").toString()));
Item.setImages(sourceAsMap.get("images").toString());
//The fields to be highlighted should correspond to the fields to be queried one by one
Map<String, HighlightField> highlightFields = hit.getHighlightFields();
if (highlightFields.get("title") != null) {
String nameHighlight = highlightFields.get("title").getFragments()[0].toString();
Item.setTitle(nameHighlight);
}
if (highlightFields.get("category") != null) {
String contentHighlight = highlightFields.get("category").getFragments()[0].toString();
Item.setCategory(contentHighlight);
}
Items.add(Item);
}
return new AggregatedPageImpl<T>((List<T>) Items);
}
});
return Items.getContent();
}
@Test
public void getHighlight(){
List<Item> sj = findByNameAndHighlightAdnPageable("mobile phone", 1, 10);
sj.forEach(System.out::println);
}
Summary of other queries
Equivalent query
Query name = Xiao Li's
QueryBuilders.termQuery("name", "petty thief")
Range query
Query records older than or equal to 18 and less than or equal to 50
QueryBuilders.rangeQuery("age").gte(18).lte(50);
Fuzzy query
Query the document records with Xiao Li in the name:
QueryBuilders.boolQuery().must(QueryBuilders.wildcardQuery("name", "*petty thief*"));
Multi condition query
Query the documents whose name is Xiao Li and whose age is between 10-50
QueryBuilders.boolQuery()
.must(QueryBuilders.termQuery("name", "petty thief"))
.must(QueryBuilders.rangeQuery("age").gte(10).lte(50));
Collection query
Search for documents with addresses in Beijing, Shanghai and Hangzhou, aged 10 to 50 and named Li Ming
List<String> list = Arrays.asList("Beijing", "Shanghai", "Hangzhou");
QueryBuilders.boolQuery()
.must(QueryBuilders.termQuery("name", "Li Ming"))
.must(QueryBuilders.termsQuery("address", list))
.must(QueryBuilders.rangeQuery("age").gte(10).lte(50));
Use should query
The query name contains Xiao Li or the record whose address is Beijing. should is equivalent to or
QueryBuilders.boolQuery()
.should(QueryBuilders.wildcardQuery("name", "*petty thief*"))
.should(QueryBuilders.termQuery("address", "Beijing"));
should and must cooperate to query
Query the records whose gender is male, whose name contains Xiao Li or whose address is Beijing, * * minimumShouldMatch(1) * * indicates that at least one should condition should be matched
QueryBuilders.boolQuery()
.must(QueryBuilders.termQuery("sex", "male"))
.should(QueryBuilders.wildcardQuery("name", "*petty thief*"))
.should(QueryBuilders.termQuery("address", "Beijing"))
.minimumShouldMatch(1);
Must: conditions that must be met
should: non mandatory condition
minimumShouldMatch(1): at least one should condition must be met
The above queryBuilder can be understood as meeting a must condition and at least one should condition.
Value query
Query the documents with a value for name but no value for tag
QueryBuilders.boolQuery()
.must(QueryBuilders.existsQuery("name"))
.mustNot(QueryBuilders.existsQuery("tag"));
should(QueryBuilders.termQuery("address", "Beijing"));
should and must cooperate to query
Query the records whose gender is male, whose name contains Xiao Li or whose address is Beijing, * * minimumShouldMatch(1) * * indicates that at least one should condition should be matched
QueryBuilders.boolQuery()
.must(QueryBuilders.termQuery("sex", "male"))
.should(QueryBuilders.wildcardQuery("name", "*petty thief*"))
.should(QueryBuilders.termQuery("address", "Beijing"))
.minimumShouldMatch(1);
Must: conditions that must be met
should: non mandatory condition
minimumShouldMatch(1): at least one should condition must be met
The above queryBuilder can be understood as meeting a must condition and at least one should condition.
Value query
Query the documents with a value for name but no value for tag
QueryBuilders.boolQuery()
.must(QueryBuilders.existsQuery("name"))
.mustNot(QueryBuilders.existsQuery("tag"));