Spark SQL/DataFrame/DataSet operation ----- read data

1, Read data source (1) Read json and use spark.read. Note: the path is from HDFS by default. If you want to read the native file, you need to prefix it file: / /, as follows scala> val people = spark.read.format("json").load("file:///opt/software/data/people.json") people: org.apache.spark.sql.DataFrame = [age: bigint, name: string] scal ...

Posted by Pie on Sun, 02 Feb 2020 08:18:33 -0800

Encryption protection of data on super ledger Fabric chain

Hyperledger Fabric is one of the most popular blockchain development frameworks. It has unique positioning and some distinctive features, such as licensing architecture, pluggable components, channels supporting private transactions, modularity and scalability, so it is suitable for the development of enterprise alliance chain applications. In ...

Posted by regoch on Sun, 02 Feb 2020 04:35:21 -0800

Visualization of new coronavirus data

At present, Tencent's Real time tracking of epidemic situation The web page can see the real-time data of new coronavirus, and the visual display is also very clear. But every time you open it, you can only see the data of that day, so you want to save the data of every day. After being saved, pyechart ...

Posted by DoddsAntS on Sat, 01 Feb 2020 03:46:32 -0800

Remote http file upload and file2multipartfile in java

There are times when you encounter various needs in your work, and you have to change your approach to solving them. Of course, what is important is in what scenarios to complete them. For example, how the file type in Strut2 is converted to the multipartfile type, I looked for a few days and found a workaround documented below (although it did ...

Posted by PascalNouma on Fri, 31 Jan 2020 12:38:47 -0800

Spring Security Series (7) remember user name login function

Remember the basics of my function The user sends a request to UsernamePasswordAuthenticationFilter. When the user authenticates successfully, a service like RemeberMeService will be called. There is a Token repository in the service, which will generate a Token, write the Token to the Cookie of the browser, and at the same ...

Posted by genistaff on Fri, 31 Jan 2020 07:29:57 -0800

21 days to build a distributed crawler (I) urllib Library

1.1.usage of urlopen function #encoding:utf-8 from urllib import request res = request.urlopen("https://www.cnblogs.com/") print(res.readlines()) #urlopen Parameters #def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, # *, cafile=None, capath=None, cadefault=False, context=None): 1.2.urlretriev ...

Posted by greywire on Thu, 30 Jan 2020 20:29:55 -0800

Come on, Wuhan! Crawling Baidu migration map data

Recently, in order to build a model to predict the spread of the virus, we need to crawl the data on Baidu's migration, and write here 1, Find the law First of all, we open http://qianxi.baidu.com/, and randomly query the information of a city, such as Beijing 29: You can find the data here. Let's lo ...

Posted by heckenschutze on Thu, 30 Jan 2020 07:49:47 -0800

Implement dataTable plug-in in SSM to realize paging

01 download resource file The following three JS and CSS resource files are introduced into the project in the BootStrap front-end framework <!-- DataTables --> <script src="/static/assets/bower_components/datatables.net/js/jquery.dataTables.min.js"></script> <script src="/stat ...

Posted by DaZZleD on Thu, 30 Jan 2020 06:43:38 -0800

Web front-end knowledge combing -- Javascript common syntax

JavaScript, like HTML and CSS, is the three necessary modules for Web front-end development. In addition, JavaScript is also the basic language of Vue, Ext, jQuery and other frameworks, so the learning of JavaScript is crucial. In fact, after learning the C + + language and Java, I feel that the basi ...

Posted by smiley_kool on Wed, 29 Jan 2020 21:06:05 -0800

[blockchain] Tendermint - local single node deployment

Tendermint Due to work reasons, I began to contact Hyperledger Fabric two years ago, and its community is still very active. It has developed from v0.7 in that year to v2.0 now. But I have to say that Fabric's consideration is still very big, which is difficult for secondary development. So recently, ...

Posted by dayang on Wed, 29 Jan 2020 03:26:18 -0800