Gearman was used by LiveJournal to make picture resize. You also know that picture resize is a high CPU operation. If you let the web site do this high CPU function, it may bring you down.
In web applications, let's look at how gearman solves this problem. Its architecture is similar to the following:
From the picture above, you should see that Gearman is made up of three parts:
1. Job Server
This is Gearman's Job Server, which bridges Client and jobwork. Does it remind you of the mediator model?
2. Client
Gearman provides Client API to call the client. Client only needs to drop the name of a high CPU business function to Job Server and wait for Job Server to return the execution result.
3. jobwork
Gearman provides the work API for the work client to make calls. jobserver will distribute to an appropriate work to execute and wait for the results according to the load of the back-end work cluster.
At this point, you should understand that it's essentially a distributed RPC call, and what's so compelling is that Client and Work can be implemented in different languages.
First: Installation and Deployment
1. Download address: https://github.com/gearman/gearmand/releases
At present, gearman's Job Server is implemented in three languages: C, JAVA and Perl. Because the C version of Job Server is the most active, we use the latest 1.1.17 version of gearman in CentOS.
Installation and deployment.
2. Fast installation
Quick installation can be done through get-start in http://gearman.org/get-start/on the official website.
Basic Dependency Library Installation and gearmand Download
1 yum -y install boost-devel gperf libevent-devel libuuid-devel gcc44 gcc-c++ 2 wget https://github.com/gearman/gearmand/releases/download/1.1.17/gearmand-1.1.17.tar.gz 3 cd gearmand-1.1.17.tar.gz 4 tar xzvf gearmand-1.1.17.tar.gz 5 cd gearmand-1.1.17 6 [root@localhost gearmand-1.1.17]# ls 7 aclocal.m4 build-aux configure.ac gear_config.in libgearman-1.0 libhashkit-1.0 Makefile.am rpm THANKS 8 AUTHORS ChangeLog COPYING gearmand libgearmancore libhostile Makefile.in scripts util 9 benchmark configmake.h docs HACKING libgearman-server libtest man support version.m4 10 bin configure examples libgearman libhashkit m4 NEWS tests
<2> Then there's the conventional. / configure -- prefix=/usr/myapp/gearman & make & make install, which is a very slow process to go out and smoke.
By the way, go and shit.
1 ./configure --prefix=/usr/myapp/gearman && make && make install
<3> Years later, when you see this, you will be able to install it successfully... Congratulations... At least it doesn't let you step on the interface that lacks all kinds of dependency packages.
1 See any operating system documentation about shared libraries for 2 more information, such as the ld(1) and ld.so(8) manual pages. 3 ---------------------------------------------------------------------- 4 /usr/bin/mkdir -p '/usr/myapp/gearman/sbin' 5 /usr/bin/install -c -m 644 man/gearman_worker_create.3 man/gearman_worker_define_function.3 man/gearman_worker_echo.3 man/gearman_worker_errno.3 man/gearman_worker_error.3 man/gearman_worker_free.3 man/gearman_worker_function_exist.3 man/gearman_worker_grab_job.3 man/gearman_worker_options.3 man/gearman_worker_register.3 man/gearman_worker_remove_options.3 man/gearman_worker_remove_servers.3 man/gearman_worker_set_context.3 man/gearman_worker_set_log_fn.3 man/gearman_worker_set_namespace.3 man/gearman_worker_set_options.3 man/gearman_worker_set_timeout.3 man/gearman_client_has_option.3 man/gearman_client_options_t.3 man/gearman_task_attr_init.3 man/gearman_task_attr_init_background.3 man/gearman_task_attr_init_epoch.3 man/gearman_task_attr_t.3 man/gearman_worker_set_identifier.3 man/gearman_worker_set_workload_free_fn.3 man/gearman_worker_set_workload_malloc_fn.3 man/gearman_worker_st.3 man/gearman_worker_timeout.3 man/gearman_worker_unregister.3 man/gearman_worker_unregister_all.3 man/gearman_worker_wait.3 man/gearman_worker_work.3 man/libgearman.3 '/usr/myapp/gearman/share/man/man3' 6 /bin/sh ./libtool --mode=install /usr/bin/install -c gearmand/gearmand '/usr/myapp/gearman/sbin' 7 libtool: install: /usr/bin/install -c gearmand/gearmand /usr/myapp/gearman/sbin/gearmand 8 /usr/bin/mkdir -p '/usr/myapp/gearman/bin' 9 /bin/sh ./libtool --mode=install /usr/bin/install -c bin/gearman bin/gearadmin '/usr/myapp/gearman/bin' 10 libtool: install: /usr/bin/install -c bin/.libs/gearman /usr/myapp/gearman/bin/gearman 11 libtool: install: /usr/bin/install -c bin/gearadmin /usr/myapp/gearman/bin/gearadmin 12 make[3]: Leaving directory `/usr/myapp/gearmand-1.1.17' 13 make[2]: Leaving directory `/usr/myapp/gearmand-1.1.17' 14 make[1]: Leaving directory `/usr/myapp/gearmand-1.1.17'
<4> Start gearmand, you can also use - d to turn on the background mode. Add DEBUG here to just look at the real-time DEBUG information, as follows:
1 [root@localhost myapp]# cd /usr/myapp/gearman 2 [root@localhost gearman]# ls 3 bin include lib sbin share 4 [root@localhost gearman]# cd bin 5 [root@localhost bin]# ls 6 gearadmin gearman 7 [root@localhost bin]# cd /usr/myapp/gearman 8 [root@localhost gearman]# cd sbin 9 [root@localhost sbin]# ls 10 gearmand 11 [root@localhost sbin]# ./gearmand --verbose DEBUG 12 ./gearmand: Could not open log file "/usr/myapp/gearman/var/log/gearmand.log", from "/usr/myapp/gearman/sbin", switching to stderr. (No such file or directory) 13 DEBUG 2017-08-29 02:31:10.796259 [ main ] THREADS: 4 -> libgearman-server/gearmand.cc:263 14 INFO 2017-08-29 02:31:10.796374 [ main ] Initializing Gear on port 4730 with SSL: false 15 INFO 2017-08-29 02:31:10.796487 [ main ] Starting up with pid 40299, verbose is set to DEBUG 16 DEBUG 2017-08-29 02:31:10.796637 [ main ] Method for libevent: epoll -> libgearman-server/gearmand.cc:364 17 DEBUG 2017-08-29 02:31:10.798874 [ main ] Trying to listen on 0.0.0.0:4730 -> libgearman-server/gearmand.cc:646 18 INFO 2017-08-29 02:31:10.800151 [ main ] Listening on 0.0.0.0:4730 (8) 19 DEBUG 2017-08-29 02:31:10.800175 [ main ] Trying to listen on :::4730 -> libgearman-server/gearmand.cc:646 20 INFO 2017-08-29 02:31:10.800307 [ main ] Listening on :::4730 (9) 21 DEBUG 2017-08-29 02:31:10.800333 [ main ] Creating wakeup pipe -> libgearman-server/gearmand.cc:915 22 DEBUG 2017-08-29 02:31:10.800344 [ main ] Creating 4 threads -> libgearman-server/gearmand.cc:378 23 DEBUG 2017-08-29 02:31:10.800357 [ main ] Initializing libevent for IO thread -> libgearman-server/gearmand_thread.cc:224 24 DEBUG 2017-08-29 02:31:10.800406 [ main ] Creating IO thread wakeup pipe -> libgearman-server/gearmand_thread.cc:495 25 DEBUG 2017-08-29 02:31:10.800467 [ main ] Thread 1 created -> libgearman-server/gearmand_thread.cc:273 26 DEBUG 2017-08-29 02:31:10.800507 [ main ] Initializing libevent for IO thread -> libgearman-server/gearmand_thread.cc:224 27 DEBUG 2017-08-29 02:31:10.800550 [ main ] Creating IO thread wakeup pipe -> libgearman-server/gearmand_thread.cc:495 28 DEBUG 2017-08-29 02:31:10.800585 [ main ] Thread 2 created -> libgearman-server/gearmand_thread.cc:273 29 DEBUG 2017-08-29 02:31:10.800594 [ main ] Initializing libevent for IO thread -> libgearman-server/gearmand_thread.cc:224 30 DEBUG 2017-08-29 02:31:10.800632 [ main ] Creating IO thread wakeup pipe -> libgearman-server/gearmand_thread.cc:495 31 DEBUG 2017-08-29 02:31:10.800669 [ main ] Thread 3 created -> libgearman-server/gearmand_thread.cc:273 32 DEBUG 2017-08-29 02:31:10.800677 [ main ] Initializing libevent for IO thread -> libgearman-server/gearmand_thread.cc:224 33 DEBUG 2017-08-29 02:31:10.800714 [ main ] Creating IO thread wakeup pipe -> libgearman-server/gearmand_thread.cc:495 34 DEBUG 2017-08-29 02:31:10.800753 [ main ] Thread 4 created -> libgearman-server/gearmand_thread.cc:273 35 DEBUG 2017-08-29 02:31:10.800761 [ main ] replaying queue: begin -> libgearman-server/gearmand.cc:391 36 DEBUG 2017-08-29 02:31:10.800766 [ main ] __replay -> libgearman-server/plugins/queue/default/queue.cc:101 37 DEBUG 2017-08-29 02:31:10.800774 [ main ] replaying queue: end -> libgearman-server/gearmand.cc:397 38 INFO 2017-08-29 02:31:10.800780 [ main ] Adding event for listening socket (8) 39 INFO 2017-08-29 02:31:10.800787 [ main ] Adding event for listening socket (9) 40 DEBUG 2017-08-29 02:31:10.800794 [ main ] Adding event for wakeup pipe -> libgearman-server/gearmand.cc:966 41 DEBUG 2017-08-29 02:31:10.800801 [ main ] Entering main event loop -> libgearman-server/gearmand.cc:406 42 DEBUG 2017-08-29 02:31:10.801186 [ 2 ] Entering thread event loop -> libgearman-server/gearmand_thread.cc:463 43 DEBUG 2017-08-29 02:31:10.801277 [ 3 ] Entering thread event loop -> libgearman-server/gearmand_thread.cc:463 44 DEBUG 2017-08-29 02:31:10.801507 [ main ] staring up Epoch thread -> libgearman-server/timer.cc:61 45 DEBUG 2017-08-29 02:31:10.801635 [ 1 ] Entering thread event loop -> libgearman-server/gearmand_thread.cc:463 46 DEBUG 2017-08-29 02:31:10.802426 [ 4 ] Entering thread event loop -> libgearman-server/gearmand_thread.cc:463
<5> Finally, we can find out the port number of gearmand by netstat, lsof and ps-ef, just as you can see the default port number of 4370.
Of course, you can also use help command at startup.
1 [root@localhost ~]# netstat -tln 2 Active Internet connections (only servers) 3 Proto Recv-Q Send-Q Local Address Foreign Address State 4 tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 5 tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6 tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 7 tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 8 tcp 0 0 0.0.0.0:4730 0.0.0.0:* LISTEN 9 tcp6 0 0 :::8009 :::* LISTEN 10 tcp6 0 0 :::8080 :::* LISTEN 11 tcp6 0 0 :::22 :::* LISTEN 12 tcp6 0 0 ::1:631 :::* LISTEN 13 tcp6 0 0 ::1:25 :::* LISTEN 14 tcp6 0 0 :::4730 :::* LISTEN 15 tcp6 0 0 127.0.0.1:8005 :::* LISTEN 16 [root@localhost ~]# ps -ef | grep gearmand 17 root 40299 15869 0 02:31 pts/1 00:00:00 ./gearmand --verbose DEBUG 18 root 40364 40327 0 02:33 pts/2 00:00:00 grep --color=auto gearmand 19 [root@localhost ~]# lsof -i :4730 20 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME 21 gearmand 40299 root 8u IPv4 322550 0t0 TCP *:gearman (LISTEN) 22 gearmand 40299 root 9u IPv6 322551 0t0 TCP *:gearman (LISTEN) 23 [root@localhost ~]#
2: The use of Java Driver on Gearman
For demonstration, I can make a simple "string. ToUpper" business logic to verify whether the architecture can run.
1. java acts as Gearman's work
First, you need to pull the jar package in the mvn warehouse: http://www.mvnrepository.com/artifact/org.gearman/gearman-java/0.6.
<1> UpperFunction class, which defines the specific business logic of work:
1 package com.datamip.gearmanwork; 2 3 import java.text.SimpleDateFormat; 4 import java.util.Date; 5 6 import org.gearman.client.GearmanJobResult; 7 import org.gearman.client.GearmanJobResultImpl; 8 import org.gearman.util.ByteUtils; 9 import org.gearman.worker.AbstractGearmanFunction; 10 11 //String capitalization business Function 12 public class UpperFunction extends AbstractGearmanFunction { 13 14 @Override 15 public GearmanJobResult executeFunction() { 16 17 String param = ByteUtils.fromUTF8Bytes((byte[]) this.data); 18 19 byte[] mybytes = param.toUpperCase().getBytes(); 20 21 GearmanJobResultImpl result = new GearmanJobResultImpl(mybytes, true, mybytes, null, null, -1, -1); 22 23 SimpleDateFormat formatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); 24 25 String dateString = formatter.format(new Date()); 26 27 System.out.println(String.format("current time:%s, Coming String:%s，Return string:%s", dateString, param,new String(mybytes))); 28 29 return result; 30 } 31 }
<2> Register UpperFunction in gearmand. As you can see from the red code, it's actually a kv mode, where key="myUpperFunc" implements a new UpperFunction.
Client only needs to pass a "myUpperFunc" and Gearmand knows which handler the "string" corresponds to...
1 public class App { 2 public static void main(String[] args) { 3 4 GearmanWorker worker = new GearmanWorkerImpl(); 5 6 GearmanNIOJobServerConnection conn = new GearmanNIOJobServerConnection("192.168.23.170", 4730); 7 worker.addServer(conn); 8 9 // take'Register functions that will be capitalized' reach gearmand in 10 worker.registerFunctionFactory(new GearmanFunctionFactory() { 11 12 public String getFunctionName() { 13 return "myUpperFunc"; 14 } 15 16 public GearmanFunction getFunction() { 17 return new UpperFunction(); 18 } 19 }); 20 21 System.out.println("Start the service..."); 22 23 worker.work(); 24 } 25 }
2. java acts as Gearman's client
GearSubmit class
1 package com.datamip.gearmanclient; 2 3 import java.util.concurrent.ExecutionException; 4 import java.util.concurrent.ExecutorService; 5 import java.util.concurrent.Executors; 6 import java.util.concurrent.Future; 7 8 import org.gearman.client.GearmanClient; 9 import org.gearman.client.GearmanClientImpl; 10 import org.gearman.client.GearmanJob; 11 import org.gearman.client.GearmanJobImpl; 12 import org.gearman.client.GearmanJobResult; 13 import org.gearman.common.GearmanJobServerConnection; 14 import org.gearman.common.GearmanNIOJobServerConnection; 15 import org.gearman.util.ByteUtils; 16 17 public class Gearsubmit { 18 19 public void process() throws InterruptedException, ExecutionException { 20 21 GearmanJobServerConnection conn = new GearmanNIOJobServerConnection("192.168.23.170", 4730); 22 23 GearmanClient client = new GearmanClientImpl(); 24 25 client.addJobServer(conn); // add connections 26 27 String functionName = "myUpperFunc"; 28 29 byte[] data = ByteUtils.toUTF8Bytes("hello,world"); 30 31 // Create background tasks 32 GearmanJob job = GearmanJobImpl.createJob(functionName, data, null); 33 34 GearmanJobResult jobResult = null; 35 36 Future<GearmanJobResult> gearmanJobResult = client.submit(job); 37 38 jobResult = gearmanJobResult.get(); 39 40 byte[] resultBytes = jobResult.getResults(); 41 42 // Obtain job Return value 43 String value = ByteUtils.fromUTF8Bytes(resultBytes); 44 45 System.out.println(value); 46 47 System.out.println("end of execution"); 48 49 client.shutdown(); 50 } 51 }
<2> Main program, multi-threaded concurrent execution.
1 public class App { 2 public static void main(String[] args) throws InterruptedException, ExecutionException, IOException { 3 4 ExecutorService executorService = Executors.newFixedThreadPool(100); 5 6 for (int i = 0; i < 10000; i++) { 7 executorService.execute(new Runnable() { 8 9 @Override 10 public void run() { 11 Gearsubmit submit=new Gearsubmit(); 12 13 try { 14 submit.process(); 15 } catch (InterruptedException | ExecutionException e) { 16 // TODO Auto-generated catch block 17 e.printStackTrace(); 18 } 19 } 20 }); 21 } 22 23 System.in.read(); 24 } 25 }
Well, everything is ready. Next, for demonstration, the demonstration is explanation. I export the work program into jar and convert it into exe with Jar2Exe. Here's the following picture:
Then I opened 3.exe into five instances, and client called it concurrently with a thread pool of 100 threads, of course, everything was simulated... You can see, but when my client started, all five work s were executed.
If at this point you stop a work, the job server will drop the task to a relatively small workload to execute.
Okay, I'm going to talk about it in this article. I hope it will be helpful to you.