Basic usage of springboot springdataelasticsearch

Keywords: Java ElasticSearch Spring Boot

Basic usage of springboot springdataelasticsearch

introduce

Spring Data Elasticsearch is a sub module under the Spring Data project.
Check the official website of Spring Data: http://projects.spring.io/spring-data/
The mission of Spring Data is to provide a unified programming interface for various data access, whether it is a relational database (such as MySQL), a non relational database (such as Redis), or an index database such as Elasticsearch. So as to simplify the developer's code and improve the development efficiency.

Create project

Project structure

POM.xml

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.4.RELEASE</version>
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.75</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

Configure the application.yaml file

spring:
  data:
    elasticsearch:
      cluster-name: elasticsearch
      cluster-nodes: 192.168.0.22:9300

Entity class and annotation

public class Item {
    Long id;
    String title; //title
    String category;// classification
    String brand; // brand
    Double price; // Price
    String images; // Picture address
}

Spring Data declares the mapping attributes of fields through annotations. There are three annotations:

  • @Document

    It acts on the class and marks the entity class as a document object. It generally has four attributes

    • indexName: the name of the corresponding index library (such as database)
    • Type: corresponds to the type in the index library (such as database table)
    • shards: the number of shards. The default value is 5
    • Replicas: number of replicas. The default is 1
  • @id acts on the member variable and marks a field as the id primary key

  • @Field

    Act on the member variable, mark it as the field of the document, and specify the field mapping attribute:

    • Type: field type. The value is enumeration: FieldType
    • Index: whether to index. Boolean type. The default is true
    • Store: whether to store. Boolean type. The default is false
    • analyzer: word breaker Name: ik_max_word

Example:

package cn.es.pojo;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;

@Document(indexName = "item",type = "docs")
@Data //Must have get set
@AllArgsConstructor //Must have full parameters
@NoArgsConstructor   //There must be no tragedy
public class Item {
    @Id
    private Long id;

    @Field(type = FieldType.Text, analyzer = "ik_max_word")
    private String title; //title

    @Field(type = FieldType.Keyword) //Keywor d does not participle
    private String category;// classification

    @Field(type = FieldType.Keyword)
    private String brand; // brand

    @Field(type = FieldType.Double)
    private Double price; // Price

    @Field(index = false, type = FieldType.Keyword)
    private String images; // Picture address
}

Create index

@SpringBootTest
@RunWith(SpringRunner.class)
public class ElasticSearctTest {
    @Autowired
    private ElasticsearchTemplate elasticsearchTemplate;
    //Create libraries (indexes (databases) and mappings (tables))
    @Test
    public void testCreate() {
        // When creating an index, it will be created according to the @ Document annotation information of the Item class
        elasticsearchTemplate.createIndex(Item.class);
        // Configuring the mapping will automatically complete the mapping according to the id, Field and other fields in the Item class
        elasticsearchTemplate.putMapping(Item.class);
    }
}

Addition, deletion and modification

The strength of Spring Data is that you don't need to write any DAO processing, and automatically perform CRUD operations according to the method name or class information. As long as you define an interface and inherit some sub interfaces provided by the Repository, you can have various basic crud functions

Write ItemRepository

package com.example.elasticsearch.repository;

import com.example.elasticsearch.pojo.Item;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;

/**
 * @author john
 * @date 2019/12/8 - 14:39
 */
public interface ItemRepository extends ElasticsearchRepository<Item,Long> {
}

increase

    @Autowired
    private ItemRepository itemRepository;
    //insert data
    @Test 
    public void testAdd() {
        Item item = new Item(1L, "Xiaomi mobile phone 7", " mobile phone",
                "millet", 3499.00, "http://image.leyou.com/13123.jpg");
        itemRepository.save(item);
    }

modify

   //Modify (if id exists, it is essentially overwrite, otherwise it is insert)
    @Test
    public void testUpdate() {
        Item item = new Item(1L, "Xiaomi mobile phone 7777", " mobile phone",
                "millet", 9499.00, "http://image.leyou.com/13123.jpg");
        itemRepository.save(item);
    }

Batch add

    @Autowired
    private ItemRepository itemRepository;

    @Test
    public void indexList() {
        List<Item> list = new ArrayList<>();
        list.add(new Item(2L, "Nut phone R1", " mobile phone", "hammer", 3699.00, "http://image.leyou.com/123.jpg"));
        list.add(new Item(3L, "Huawei META10", " mobile phone", "Huawei", 4499.00, "http://image.leyou.com/3.jpg"));
        // Receive object collection to realize batch addition
        itemRepository.saveAll(list);
    }


Delete operation

    @Autowired
    private ItemRepository itemRepository;

    @Test
    public void testDelete() {
        itemRepository.deleteById(1L);
    }

Query by id

    @Autowired
    private ItemRepository itemRepository;
    @Test
    public void testQuery(){
        Optional<Item> optional = itemRepository.findById(2L);
        System.out.println(optional.get());
    }

Query all and sort by price descending order

    @Autowired
    private ItemRepository itemRepository;

	@Test
    public void testFind(){
        // Query all and sort by price descending order
        Iterable<Item> items = this.itemRepository.findAll(Sort.by(Sort.Direction.DESC, "price"));
        items.forEach(item-> System.out.println(item));
    }

Custom method

Another powerful feature of Spring Data is the automatic implementation of functions based on method names.
For example, your method name is: findByTitle, then it will know that you query according to the title, and then automatically help you complete it without writing an implementation class.
Of course, the method name must comply with certain conventions:

KeywordSampleElasticsearch Query String
AndfindByNameAndPrice{"bool" : {"must" : [ {"field" : {"name" : "?"}}, {"field" : {"price" : "?"}} ]}}
OrfindByNameOrPrice{"bool" : {"should" : [ {"field" : {"name" : "?"}}, {"field" : {"price" : "?"}} ]}}
IsfindByName{"bool" : {"must" : {"field" : {"name" : "?"}}}}
NotfindByNameNot{"bool" : {"must_not" : {"field" : {"name" : "?"}}}}
BetweenfindByPriceBetween{"bool" : {"must" : {"range" : {"price" : {"from" : ?,"to" : ?,"include_lower" : true,"include_upper" : true}}}}}
LessThanEqualfindByPriceLessThan{"bool" : {"must" : {"range" : {"price" : {"from" : null,"to" : ?,"include_lower" : true,"include_upper" : true}}}}}
GreaterThanEqualfindByPriceGreaterThan{"bool" : {"must" : {"range" : {"price" : {"from" : ?,"to" : null,"include_lower" : true,"include_upper" : true}}}}}
BeforefindByPriceBefore{"bool" : {"must" : {"range" : {"price" : {"from" : null,"to" : ?,"include_lower" : true,"include_upper" : true}}}}}
AfterfindByPriceAfter{"bool" : {"must" : {"range" : {"price" : {"from" : ?,"to" : null,"include_lower" : true,"include_upper" : true}}}}}
LikefindByNameLike{"bool" : {"must" : {"field" : {"name" : {"query" : "?*","analyze_wildcard" : true}}}}}
StartingWithfindByNameStartingWith{"bool" : {"must" : {"field" : {"name" : {"query" : "?*","analyze_wildcard" : true}}}}}
EndingWithfindByNameEndingWith{"bool" : {"must" : {"field" : {"name" : {"query" : "*?","analyze_wildcard" : true}}}}}
Contains/ContainingfindByNameContaining{"bool" : {"must" : {"field" : {"name" : {"query" : "**?**","analyze_wildcard" : true}}}}}
InfindByNameIn(Collection<String>names){"bool" : {"must" : {"bool" : {"should" : [ {"field" : {"name" : "?"}}, {"field" : {"name" : "?"}} ]}}}}
NotInfindByNameNotIn(Collection<String>names){"bool" : {"must_not" : {"bool" : {"should" : {"field" : {"name" : "?"}}}}}}
NearfindByStoreNearNot Supported Yet !
TruefindByAvailableTrue{"bool" : {"must" : {"field" : {"available" : true}}}}
FalsefindByAvailableFalse{"bool" : {"must" : {"field" : {"available" : false}}}}
OrderByfindByAvailableTrueOrderByNameDesc{"sort" : [{ "name" : {"order" : "desc"} }],"bool" : {"must" : {"field" : {"available" : true}}}}

For example, let's define a method to query by price range:

public interface ItemRepository extends ElasticsearchRepository<Item,Long> {

    /**
     * Query by price range
     * @param price1
     * @param price2
     * @return
     */
    List<Item> findByPriceBetween(double price1, double price2);
}

Then add some test data:

    @Autowired
    private ItemRepository itemRepository; 

	@Test
    public void indexList_custom () { //Add some test data
        List<Item> list = new ArrayList<>();
        list.add(new Item(1L, "Xiaomi mobile phone 7", "mobile phone", "millet", 3299.00, "http://image.leyou.com/13123.jpg"));
        list.add(new Item(2L, "Nut phone R1", "mobile phone", "hammer", 3699.00, "http://image.leyou.com/13123.jpg"));
        list.add(new Item(3L, "Huawei META10", "mobile phone", "Huawei", 4499.00, "http://image.leyou.com/13123.jpg"));
        list.add(new Item(4L, "millet Mix2S", "mobile phone", "millet", 4299.00, "http://image.leyou.com/13123.jpg"));
        list.add(new Item(5L, "glory V10", "mobile phone", "Huawei", 2799.00, "http://image.leyou.com/13123.jpg"));
        // Receive object collection to realize batch addition
        itemRepository.saveAll(list);
    }

There is no need to write an implementation class, call the interface, and then we run it directly:

 @Autowired
 private ItemRepository itemRepository;

@Test
public void queryByPriceBetween(){
    List<Item> list = this.itemRepository.findByPriceBetween(2000.00, 3500.00);
    for (Item item : list) {
        System.out.println("item = " + item);
    }
}

Although the basic query and user-defined methods are very powerful, it is not enough if it is a complex query (fuzzy, wildcard, entry query, etc.). At this point, we can only use native queries.

Advanced query

Let's look at the basic playing methods first

@Test
public void testBaseQuery(){
    // Term query
    MatchQueryBuilder queryBuilder = QueryBuilders.matchQuery("title", "millet");
    // Execute query
    Iterable<Item> items = this.itemRepository.search(queryBuilder);
    items.forEach(System.out::println);
}

QueryBuilders provides a large number of static methods for generating various types of query objects, such as QueryBuilder objects such as entry, fuzzy, wildcard, etc.

Elastic search provides many available query methods, but it is not flexible enough. If you want to play filtering or aggregate query, it is difficult.

Paging query

// Paging query
    @Test
    public void testNativeQuery2(){
        // Build query criteria
        NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
        // Add basic word segmentation query
        queryBuilder.withQuery(QueryBuilders.termQuery("category", "mobile phone"));

        // Initialize paging parameters
        int page = 0;
        int size = 3;
        // Set paging parameters
        queryBuilder.withPageable(PageRequest.of(page, size));

        // Perform a search to get results
        Page<Item> items = this.itemRepository.search(queryBuilder.build());
        // Total number of prints
        System.out.println("Total number"+items.getTotalElements());
        // Total pages printed
        System.out.println("PageCount "+items.getTotalPages());
        // Size per page
        System.out.println("Size per page"+items.getSize());
        // Current page
        System.out.println("Current page"+items.getNumber());
        items.forEach(System.out::println);
    }

Native searchquerybuilder: a query condition builder provided by Spring to help build the request body in json format

Page < item >: the default is a paged query, so a paged result object is returned, including the following attributes:

  • totalElements: total number of items
  • totalPages: total pages
  • Iterator: iterator, which implements the iterator interface, so it can directly iterate to get the data of the current page
  • Other properties:

It can be found that the paging in elastic search starts from page 0.

sort

Sorting is also done through NativeSearchQueryBuilder:

//sort
    @Test
    public void testSort(){
        // Build query criteria
        NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
        // Add basic word segmentation query
        queryBuilder.withQuery(QueryBuilders.termQuery("category", "mobile phone"));

        // sort
        queryBuilder.withSort(SortBuilders.fieldSort("price").order(SortOrder.DESC));

        // Perform a search to get results
        Page<Item> items = this.itemRepository.search(queryBuilder.build());
        // Total number of prints
        System.out.println(items.getTotalElements());
        items.forEach(System.out::println);
    }

polymerization

Aggregate into barrels

Barrels are grouped. For example, here we are grouped according to brand:

@Test
public void testAgg(){
    NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
    // Do not query any results
    queryBuilder.withSourceFilter(new FetchSourceFilter(new String[]{""}, null));
    // 1. Add a new aggregation with the aggregation type of terms, the aggregation name of brands and the aggregation field of brand
    queryBuilder.addAggregation(
        AggregationBuilders.terms("brands").field("brand"));
    // 2. Query, you need to force the result to the AggregatedPage type
    AggregatedPage<Item> aggPage = (AggregatedPage<Item>) this.itemRepository.search(queryBuilder.build());
    // 3. Analysis
    // 3.1. Take the aggregation named brands from the results,
    // Because it is term aggregation using String type fields, the result should be strongly converted to StringTerm type
    StringTerms agg = (StringTerms) aggPage.getAggregation("brands");
    // 3.2. Obtaining barrels
    List<StringTerms.Bucket> buckets = agg.getBuckets();
    // 3.3 traversal
    for (StringTerms.Bucket bucket : buckets) {
        // 3.4. Get the key in the bucket, that is, the brand name
        System.out.println(bucket.getKeyAsString());
        // 3.5. Get the number of documents in the bucket
        System.out.println(bucket.getDocCount());
    }

}

Nested aggregation, averaging

@Test
public void testSubAgg(){
    NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
    // Do not query any results
    queryBuilder.withSourceFilter(new FetchSourceFilter(new String[]{""}, null));
    // 1. Add a new aggregation with the aggregation type of terms, the aggregation name of brands and the aggregation field of brand
    queryBuilder.addAggregation(
        AggregationBuilders.terms("brands").field("brand")
        .subAggregation(AggregationBuilders.avg("priceAvg").field("price")) // Perform nested aggregation in the brand aggregation bucket and calculate the average value
    );
    // 2. Query, you need to force the result to the AggregatedPage type
    AggregatedPage<Item> aggPage = (AggregatedPage<Item>) this.itemRepository.search(queryBuilder.build());
    // 3. Analysis
    // 3.1. Take the aggregation named brands from the results,
    // Because it is term aggregation using String type fields, the result should be strongly converted to StringTerm type
    StringTerms agg = (StringTerms) aggPage.getAggregation("brands");
    // 3.2. Obtaining barrels
    List<StringTerms.Bucket> buckets = agg.getBuckets();
    // 3.3 traversal
    for (StringTerms.Bucket bucket : buckets) {
        // 3.4. Get the key in the bucket, i.e. brand name 3.5. Get the number of documents in the bucket
        System.out.println(bucket.getKeyAsString() + "´╝îcommon" + bucket.getDocCount() + "platform");

        // 3.6. Obtain sub aggregation results:
        InternalAvg avg = (InternalAvg) bucket.getAggregations().asMap().get("priceAvg");
        System.out.println("Average selling price:" + avg.getValue());
    }

}

Highlight query

  //Highlight query (multi field query paging sort)
    public List<Item> findByNameAndHighlightAdnPageable(String name, int page,int size) {

        if(page <= 0){
            page = 1;
        }
        if( size <= 0){
            size = 5; //5 items are displayed on the lowest page
        }
                HighlightBuilder.Field nameField = new HighlightBuilder
                .Field("*")
                .preTags("<span style='color:red'>")
                .postTags("</span>").requireFieldMatch(false);

  //For multi field query, you can query the attribute name in the corresponding entity class in title and category at the same time. Note: highlight can only query text type fields, not numeric type fields
        NativeSearchQuery nativeSearchQuery = new NativeSearchQueryBuilder()
                .withQuery(QueryBuilders.multiMatchQuery(name, "title","category"))
                .withPageable(PageRequest.of(page - 1, size))   //paging
                .withSort(SortBuilders.fieldSort("price").order(SortOrder.DESC))   //sort
                .withHighlightFields(nameField)
                .build();
// In addition, other methods can be used in addition to QueryBuilders.multiMatchQuery 
//For example, QueryBuilders.termQuery looks at your actual needs
        
        
        

        AggregatedPage<Item> Items = elasticsearchTemplate.queryForPage(nativeSearchQuery, Item.class, new SearchResultMapper() {
                    @Override
                    public <T> AggregatedPage<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
                        SearchHits searchHits = response.getHits();
                        SearchHit[] hits = searchHits.getHits();
                        ArrayList<Item> Items = new ArrayList<Item>();
                        for (SearchHit hit : hits) {
                            Item Item = new Item();
                            //Data fields to be displayed in the original map (need to be modified following the entity class)
                            Map<String, Object> sourceAsMap = hit.getSourceAsMap();
                            Item.setId(Long.parseLong(sourceAsMap.get("id").toString()));
                            Item.setTitle(sourceAsMap.get("title").toString());
                            Item.setCategory(sourceAsMap.get("category").toString());
                            Item.setBrand(sourceAsMap.get("brand").toString());
                            Item.setPrice(Double.parseDouble(sourceAsMap.get("price").toString()));
                            Item.setImages(sourceAsMap.get("images").toString());



                            //The fields to be highlighted should correspond to the fields to be queried one by one
                            Map<String, HighlightField> highlightFields = hit.getHighlightFields();
                            if (highlightFields.get("title") != null) {
                                String nameHighlight = highlightFields.get("title").getFragments()[0].toString();
                                Item.setTitle(nameHighlight);
                            }
                            if (highlightFields.get("category") != null) {
                                String contentHighlight = highlightFields.get("category").getFragments()[0].toString();
                                Item.setCategory(contentHighlight);
                            }
                            Items.add(Item);
                        }
                        return new AggregatedPageImpl<T>((List<T>) Items);
                    }
                });
        return Items.getContent();
    }


    @Test
    public void getHighlight(){
        List<Item> sj = findByNameAndHighlightAdnPageable("mobile phone", 1, 10);
        sj.forEach(System.out::println);

    }

Summary of other queries

Equivalent query

Query name = Xiao Li's

QueryBuilders.termQuery("name", "petty thief")

Range query

Query records older than or equal to 18 and less than or equal to 50

QueryBuilders.rangeQuery("age").gte(18).lte(50);

Fuzzy query

Query the document records with Xiao Li in the name:

QueryBuilders.boolQuery().must(QueryBuilders.wildcardQuery("name", "*petty thief*"));

Multi condition query

Query the documents whose name is Xiao Li and whose age is between 10-50

QueryBuilders.boolQuery()
    .must(QueryBuilders.termQuery("name", "petty thief"))
    .must(QueryBuilders.rangeQuery("age").gte(10).lte(50));

Collection query

Search for documents with addresses in Beijing, Shanghai and Hangzhou, aged 10 to 50 and named Li Ming

List<String> list = Arrays.asList("Beijing", "Shanghai", "Hangzhou");
QueryBuilders.boolQuery()
   .must(QueryBuilders.termQuery("name", "Li Ming"))
   .must(QueryBuilders.termsQuery("address", list))
   .must(QueryBuilders.rangeQuery("age").gte(10).lte(50));

Use should query

The query name contains Xiao Li or the record whose address is Beijing. should is equivalent to or

QueryBuilders.boolQuery()
    .should(QueryBuilders.wildcardQuery("name", "*petty thief*"))
    .should(QueryBuilders.termQuery("address", "Beijing"));

should and must cooperate to query

Query the records whose gender is male, whose name contains Xiao Li or whose address is Beijing, * * minimumShouldMatch(1) * * indicates that at least one should condition should be matched

QueryBuilders.boolQuery()
   .must(QueryBuilders.termQuery("sex", "male"))
   .should(QueryBuilders.wildcardQuery("name", "*petty thief*"))
   .should(QueryBuilders.termQuery("address", "Beijing"))
   .minimumShouldMatch(1);

Must: conditions that must be met

should: non mandatory condition

minimumShouldMatch(1): at least one should condition must be met

The above queryBuilder can be understood as meeting a must condition and at least one should condition.

Value query

Query the documents with a value for name but no value for tag

QueryBuilders.boolQuery()
     .must(QueryBuilders.existsQuery("name"))
     .mustNot(QueryBuilders.existsQuery("tag"));
should(QueryBuilders.termQuery("address", "Beijing"));

should and must cooperate to query

Query the records whose gender is male, whose name contains Xiao Li or whose address is Beijing, * * minimumShouldMatch(1) * * indicates that at least one should condition should be matched

QueryBuilders.boolQuery()
   .must(QueryBuilders.termQuery("sex", "male"))
   .should(QueryBuilders.wildcardQuery("name", "*petty thief*"))
   .should(QueryBuilders.termQuery("address", "Beijing"))
   .minimumShouldMatch(1);

Must: conditions that must be met

should: non mandatory condition

minimumShouldMatch(1): at least one should condition must be met

The above queryBuilder can be understood as meeting a must condition and at least one should condition.

Value query

Query the documents with a value for name but no value for tag

QueryBuilders.boolQuery()
     .must(QueryBuilders.existsQuery("name"))
     .mustNot(QueryBuilders.existsQuery("tag"));
Like - collect - pay attention - easy to review and receive the latest content in the future If you have other questions to discuss in the comment area - or send me a private letter - you will reply as soon as you receive them In case of infringement, please contact me by private letter Thank you for your cooperation. I hope my efforts will be helpful to you^_^

Posted by The Merg on Tue, 21 Sep 2021 16:31:24 -0700