Detailed explanation of watermark in flink EventTime

window operation on data In the window operation of flow data, when event time passes the window time, it will close the window. Then in the actual production environment, due to various reasons such as the network, late data may appear, resulting in data disorder. As shown in the figure below, the ...

Posted by plex303 on Thu, 25 Jun 2020 22:46:08 -0700

Learn scala--Map&Tuple together

Map is an iterative set of key value pairs. Using maps in scala is very simple. Construct Map //Immutable map (immutable map, immutable value) // With initial value // In scala, the - > operator is used to create key value pairs of tuples. val inmutableMap = Map(("yoleen" -> 20), ("hq", 26)) // It does not have an initial value, and it ...

Posted by pklover on Wed, 27 May 2020 08:13:38 -0700

Learn scala--Map&Tuple together

Map is an iterative set of key value pairs. Using maps in scala is very simple. Construct Map //Immutable map (immutable map, immutable value) // With initial value // In scala, the - > operator is used to create key value pairs of tuples. val inmutableMap = Map(("yoleen" -> 20), ("hq", 26)) // It does not have an initial value, and it is ...

Posted by curtis_b on Wed, 27 May 2020 07:12:36 -0700

Instances of conversion operations for the core DStream of Spark Streaming

Conversion operation of DStream The DStream API provides the following methods related to transformation operations: Examples of transform(func) and updateStateByKey(func) methods are given below: (1), transform(func) method transform methods and similar transformWith(func) methods allow any RDD-to-RDD function to be applied on DStream and can ...

Posted by buildakicker on Sat, 23 May 2020 12:15:37 -0700

search (14) - elastic4s - statistical range: global, filter, post filter bucket

Aggregation generally works within the scope of query. The aggregation request without query is actually counted within the query range of match [all {}: GET /cartxns/_search { "aggs": { "all_colors": { "terms": {"field" : "color.keyword"} } } } } GET /cartxns/_search { "query": { "match_all": {} }, "aggs": { ...

Posted by something on Sat, 16 May 2020 08:26:40 -0700

Search (10) - elastic4s multi match: multi field full text search

In full-text search, we often match the same query criteria in multiple fields or different criteria in different fields. For example: GET /books/_search { "query": { "bool": { "should": [ { "match": { "title": "Peace and war" }}, { "match": { "author": "Tostei" }} ] } } } We can use boolQuery to c ...

Posted by xgab on Sat, 09 May 2020 00:45:12 -0700

Spark DataFrame is not a real DataFrame

The article was originally written in Mars team column , welcome to follow. From this article, we start a new series of reading paper. Today's paper is Towards Scalable Dataframe Systems , is still a preprint. By Devin Petersohn from Riselab , formerly known as APMLab, the lab has produced a series of famous open source projects, such as Apache ...

Posted by daniel_grant on Sun, 26 Apr 2020 00:52:37 -0700

Recommendation Engine for SparkML (2) - Evaluation of Recommendation Model

The content and code for this article follow Last article To write, we recommend that you take a look at Ha~.We wrote the implementation of the movie recommendation in the last article, but is the recommendation reasonable? This requires us to evaluate the model.For the recommended models, the models are evaluated based on the mean square devia ...

Posted by chiprivers on Thu, 23 Apr 2020 10:52:57 -0700

Python high order function practice [user defined high order function, map, reduce, filter, sorted]

I. overview scala is similar to higher-order function, which is a function that can receive another function as a parameter. 2, Import base package import random from functools import reduce 3, Custom higher order function #Define common functions and generate lists automatically def getList(): hList = list(range(5)) return ...

Posted by rledieu on Fri, 17 Apr 2020 07:59:20 -0700

Scala Actor concurrent programming WordCount

Scala Actor concurrent programming WordCount Demand: Write a stand-alone WordCount with actor concurrent programming, take multiple files as input, summarize multiple tasks after calculation, and get the final result. General steps: 1. Continuously receive messages through loop +react 2. Using case class sample cla ...

Posted by php_b34st on Sat, 04 Apr 2020 09:28:51 -0700