ElasticSearch-helper

It has been a while since ElasticSearch was used as the data source for analysis and retrieval. It has experienced several large and small projects. From the first contact with this technology, I was amazed at its ingenious design and its disgusting access api; When I was a beginner, I often looked at the endless chain calls developed by my predecessors and wondered what to do, looking for parameters that needed to be adjusted everywhere; Later, in a project, I came across an anti sky search box with dozens of conditions, filtering and blurring, and the scope should be reasonable (without discussing the rationality of the design, maybe the existence is reasonable), so the number of lines of code of the search method was reduced to 1000 +, so I wondered if I could design such a framework, The query statement of es is described by objects, and the entity class of query conditions can explain what kind of query they want to do. Therefore, I developed elastic search helper.

Project address warehouse address: https://gitee.com/JohenTeng/elasticsearch-helper.git
Use example address: https://gitee.com/JohenTeng/elasticsearch-helper-sample.git

Adaptation scenario

ElasticSearch helper mainly encapsulates queries. The current version does not provide too much support for DDL and addition, deletion and modification, because addition, deletion and modification involve more proprietary APIs, such as batch, asynchronous, callback, etc; Therefore, the framework is suitable for systems that use ElasticSearch as the data source and have a large number of query criteria.

structure

EsQueryEngine: accepts the query object and generates es query object. The object needs to be described by @ EsQueryIndex, and the annotation under org/pippi/elasticsearch/helper/core/beans/annotation/query/module is used to describe the attributes: @ Match,@MultiMatch,@Term

RequestHook: extension interface to implement the extension described in Request
ResponseHook: extension interface, custom interpretation of SearchResponse

usage method

In the main method:
@EnableEsHelper // Enable EsHelper framework 
@SpringBootApplication(scanBasePackages = "com.sun")
public class SampleApplication {
    public static void main(String[] args) {
        SpringApplication.run(SampleApplication.class, args);
    }
}

stay spring-boot Defined in configuration:

.properties to configure:
## Configure whether the log prints the query statement of elasticsearch
es.helper.queryLogOut.enable = true 
## Configure the location of the custom processing class:
es.helper.ext.handle-packages = *** 

.yml to configure:
es:
  helper:
    queryLogOut:
      enable: true
    ext:
      handle-packages: com.***.***.handles1,com.***.***.handles2 

Query object:

@EsQueryIndex(index = "test", model = QueryModel.BOOL)
// Enable highlighting
@HighLight(fields = {"title", "describe"}) 
public class DemoSearchParam {
    // Range query
    @Range(value = @Base, tag = Range.LE_GE)
    private RangeParam intensity;
    
    @MultiMatch(value = @Base,fields = {"title", "describe"})
    private String title;
}
Return object:
// The default query result processing result class needs to inherit BaseResp.BaseHit
public class Content extends BaseResp.BaseHit {

    private int intensity;

    private String title;
}

Proxy interface:
@EsHelperProxy
public interface TestQueryService {
    BaseResp<Content> queryRecordByIntensity(ContentSearchParam param)
}

Call:
@RunWith(SpringRunner.class)
@SpringBootTest(classes = SampleApplication.class)
public class TestQueryBeanServiceTest {

    @Resource
    private TestQueryService testQueryService;

    @Test
    public void testQueryService(){
        ContentSearchParam param = new ContentSearchParam();
        RangeParam rangeParam = new RangeParam();
        rangeParam.setLeft(12);
        rangeParam.setRight(15);
        param.setIntensity(rangeParam);
        param.setTitle("Test test");
        BaseResp<Content> baseResp = testQueryService.queryRecordByIntensity(param);
        System.out.println(SerializerUtils.parseObjToJson(baseResp));
    }

}

extend

=> Package: org.pippi.elasticsearch.helper.core.beans.annotation.query.mapping.extend The following classes are defined:
MoreLikeThisParam: @MoreLikeThis The modified class attribute needs to be defined as this type;
PageParam: @PageAndOrder The modified class attribute needs to be defined as this type;
RangeParam: @Range The modified class attribute needs to be defined as this type;

=> About hook method:
Package: org.pippi.elasticsearch.helper.core.hook
 Function: we cannot define all query scenarios, so we need to use the hook method to customize the query class and result processing method
 Interface:
@FunctionalInterface RequestHook#handleRequest
@FunctionalInterface ResponseHook#handleResponse

We can make the query object inherit HookQuery abstract class,Realize the set method,For example:

public class DemoParam extends HookQuery {
    
    @Match(value = @Base)
    private String Field;
    
    @Override
    public void setRequestHook() {
        super.requestHook = (h, p) -> h;
    }
    @Override
    public void setResponseHook() {
        super.responseHook = resp -> null;
    }
}

Or define public hook methods in the code, such as:
public interface MySearchHooks extends UserHooks {
    RequestHook reqHook1 = (h, p) -> h;
    ResponseHook respHook1 = resp -> null;
}
use:
@EsHelperProxy
public interface TestQueryService {
    // The query executes the RequestHook hook after processing the annotation definition in the query class
    // After the Es returns the original object, the result processing hook will be used to return the object defined by the hook
    @UseRequestHook("reqHook1")
    @UseResponseHook("respHook1")
    BaseResp<Content> queryRecordByIntensity(ContentSearchParam param);
}

=> About highlighting:
Highlight definitions should be similar in the same system;
We can configure Bean The global highlight configuration class is called, and different highlight configurations are defined.

class AutoConfiguration {
    
    void config {
    // definition
        GlobalEsQueryConfig.configHighLight(DEFAULT_KEY ,() -> SearchSourceBuilder.highlight());
        GlobalEsQueryConfig.configHighLight("html" ,() ->
                SearchSourceBuilder.highlight().fragmentSize(10).numOfFragments(5)
        );
    }
}

have access to @HighLight(fields = {"title", "describe"}, highLightKey = "html")
Medium highLightKey To define the HightLight to configure

epilogue

At present, I haven't packed it into the central warehouse, because considering that there are some changes in various subdivision versions of Elasticsearch's high-level rest client, the current client version of the project is 7.13.3; Theoretically, it supports 7. * elastic serch server;
For the support of aggregate query, because the aggregation scenario is complex, it is difficult to describe it properly through annotations. Therefore, we need to define the aggregation behavior of query in the hook method.

I hope this wheel can be brought to your convenience and point out more problems and deficiencies. Thank you, sir.