Business background

Three Tomcats are deployed on one server, and the server.xml configuration for each tomcat is as follows, except for inconsistent ports.

<Executor name="tomcatThreadPool" namePrefix="wkApi-exec-"  maxThreads="1200" minSpareThreads="400"/>
<Connector  executor="tomcatThreadPool"  port="9000" 
               protocol="org.apache.coyote.http11.Http11NioProtocol"
               connectionTimeout="60000"
               redirectPort="8443"
               URIEncoding="UTF-8"
               maxPostSize="0"
               maxThreads="1200" 
               minSpareThreads="256"
               acceptCount="300" 
               compression="on" 
               compressionMinSize="2048" 
               noCompressionUserAgents="gozilla,traviata" 
               CompressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,text/json,application/x-javascript,application/javascript,application/json,image/png,image/jpeg"
               />

Problems arise

1) Caused by:java.lang.OutOfMemoryError: unable to create new native thread, but the program is still running.
2) Users who are currently serving cannot operate any commands and report: -bash: fork: retry: Resource temporarily unavailable

ok, it's not a server configuration error. Is the total process of service creation from the current user > the maximum of the current user

By: ulimit-a command

[appuser@123.57.86.172/10.173.41.156 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7415
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

max user processes (-u) 1024

You can see that the current user can only start 1024 processes

You can use pstree-a | wc-l to count the total number of current processes

[appuser@123.57.86.172/10.173.41.156 ~]$ pstree -a|wc -l
75
 #Export all process trees
[appuser@123.57.86.172/10.173.41.156 ~]$ pstree -a > 1.log

You can view all processes

Solution:

1. Limit the total number of tomcat processes to less than 1024 (ps: the maximum is about 9000).
2. Adjust the maximum number of processes through / etc/security/limits.d/90-nproc.conf.

You can see that other users are limited to 1024

[appuser@123.57.86.172/10.173.41.156 ~]$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024
root       soft    nproc     unlimited