Configuration and startup of Hadoop pseudo distributed environment

1. Environmental preparation On a linux machine, install the hadoop running environment. For the installation method, see: Establishment of HADOOP operation environment 2. Start HDFS and run MapReduce 2.1. Configure cluster 1. Configuration: hadoop-env.sh Get the installation path of JDK in Linux system: [root@ hadoop101 ~]# echo $JAVA_HOME /op ...

Posted by spooke2k on Tue, 25 Feb 2020 19:31:27 -0800

Hadoop running environment building tutorial

1. Environmental preparation: A linux machine can be a virtual machine installed by local VMware or a real linux machine. If it is a locally installed virtual machine, the following points need to be pre configured: Configure the static IP of the machine (to prevent IP changes during restart) Modify host name (easy to configure) Turn off the ...

Posted by crazylegseddie on Tue, 25 Feb 2020 19:02:32 -0800

Spring Boot bottom level Exploration Series 04 - Web Development

Article directory Handling JSON data 1. Edit the Book class 2. Edit BookController controller 3. Convert set data 4. Replace the converter 1) Use Gson 2) Using fastjson Jiang Shuai, Naixue education, is good at system architecture design, big data, operation and maintenance and other technica ...

Posted by glitch003 on Sat, 22 Feb 2020 03:57:14 -0800

oracle 19c virtual automatic index test

1. About testing Recently oracle held a series of online sharing, one of the new features of 19c is automatic indexing. I tested it in docker and failed. [oracle@8aa96a41b58b ~]$ sqlplus / as sysdba SQL*Plus: Release 19.0.0.0.0 - Production on Thu Feb 20 09:44:52 2020 Version 19.3.0.0.0 SQL> EXEC DBMS_AUTO_INDEX.CONFIGURE('AUTO_INDEX_MODE ...

Posted by jabbaonthedais on Fri, 21 Feb 2020 04:58:37 -0800

Python Crawler Serial 5-Proxy, Cookie Resolution

1. ProxyHandler Processing (Proxy Server) 1. Using proxy IP is a common means of crawling 2. Get the address of the proxy server: www.xicidaili.com www.goubanjia.com 3. Proxies are used to hide real access. Proxies do not allow frequent access to a fixed site, so proxies must be many. 4. Basic usage steps: (1) Set proxy address (2) Create a Pox ...

Posted by landavia on Wed, 05 Feb 2020 08:49:12 -0800

Spring cloud series - get started quickly hystrix

What is hystrix? hystrix is a fuse protection middleware used by netflix for microservice distributed system. hystrix provides elegant response results to clients after the service is not reachable or the request times out. Why use hystrix? In the microservice scenario, many services depend on e ...

Posted by Sturm on Tue, 04 Feb 2020 06:02:08 -0800

Algorithm learning series - hash table, bloom filter, consistent hash, island problem, and concurrent query set

Article directory 1. Understand hash function and hash table 1.1 what is Hash 1.2 why there should be Hash 1.3 for example, chestnuts: 1. To use array storage, you need to create new arrays new int[]{2,5,9,13}, and then write a loop to traverse the search: 2. If the hash function is used for calcul ...

Posted by duk on Tue, 04 Feb 2020 00:09:04 -0800

Find the number of adjacent words in large amount of data

This topic is similar to some of the search topics in Leetcode. The problem you want to deal with is: count the number of two adjacent digits of a word. If there are w1,w2,w3,w4,w5,w6, then: The final output is (word,neighbor,frequency). We implement it in five ways: MapReduce Spark Spark SQL method Scala method Spark SQL for Scala MapReduce ...

Posted by olechka on Sun, 02 Feb 2020 08:18:59 -0800

Spark SQL/DataFrame/DataSet operation ----- read data

1, Read data source (1) Read json and use spark.read. Note: the path is from HDFS by default. If you want to read the native file, you need to prefix it file: / /, as follows scala> val people = spark.read.format("json").load("file:///opt/software/data/people.json") people: org.apache.spark.sql.DataFrame = [age: bigint, name: string] scal ...

Posted by Pie on Sun, 02 Feb 2020 08:18:33 -0800

Baidu feijiang deep learning

Catalog 1. In depth learning development process 1.1 Turing test 1.2 hierarchical processing information 1.3 in depth learning 2. Machine learning 2.1 category 2.2 classification 3. Neural network 3.1 initial understanding of neural network 3.2 example of neural network - paddle learning handwri ...

Posted by TWD on Wed, 29 Jan 2020 01:30:39 -0800