The web backend on the server often needs to run multiple processes at the same time, and each process needs to exchange data. This function can be realized in python with the help of multiprocessing. Further, after the process is interrupted, we want developers to receive messages for improvement and maintenance at the first time, which needs to be implemented with the help of process management package psutil and Python's own e-mail sending module smtplib and email.
1, Multiprocessing programming and communication
In order to simulate multi process communication, we design three modules to realize this function: receiver, parser, and publish. The data is transmitted from the receiving module to the parsing module through the mutiprocessing Queue, and then from the parsing module to the publishing module.
The Receiver continuously generates natural numbers through a while loop and puts them in the queue:
def receiver(self, queue):
while True:
self.count += 1
if not queue.full():
print(multiprocessing.current_process().pid, "Process playback data", self.count)
queue.put(self.count)
time.sleep(1)
else:
time.sleep(2)
The parser further processes the data. It needs two queue parameters, one is responsible for receiving from the receiver and the other is responsible for transmitting to the publisher. When the data passed by the receiver is equal to 20, the program deliberately reports an error and the simulation process reports an error down.
def parser(self, queue, queue2):
while True:
if not queue.empty():
num = queue.get()
if num == 20: # Here, when num reaches 20, deliberately let the program report an error and simulate the process to report an error down
a = int("hello world")
self.info = "Now count is %d" % a
queue2.put(self.info)
else:
print(multiprocessing.current_process().pid, "Process receiving data", num)
# lock.acquire()
self.info = "Now count is {}".format(num)
# lock.release()
queue2.put(self.info)
else:
pass
publisher is responsible for publishing data
def publish(self, queue):
while True:
if not queue.empty():
msg = queue.get()
# lock.acquire()
print("process", multiprocessing.current_process().pid, "Publish info:", msg)
# lock.release()
else:
pass
2, Process monitoring
analysis
In this case, we can use the following scheme:
Scheme 1: use the Zabbix/Prometheus monitoring system to detect the TCP port of Java applications. If the detection port is blocked, set the trigger for detection failure. Then realize the alarm
Scheme 2: use Zabbix's custom key to realize the alarm. The content of the custom key is to execute a shell script. The shell script is used to detect whether the process exists. If it does not exist, the output is 0 and the existence output is 1. Then, Zabbix's trigger sets the nearest T value not to be 1, so as to realize the alarm of process state detection.
Scheme 3: write Python script to detect the status of the process, and then use Python's built-in library to realize e-mail alarm.
Here we choose scheme 3, the function to be realized: to detect whether a process has become a zombie process
Idea:
1. First, the Python program needs to check whether the given process is running normally.
2. If it is detected that the process is normal, do not do any processing; If it has become a zombie process, you need to trigger the email alarm function
3.Python programs need to execute detection scripts regularly. (optional, I haven't implemented it here. It can be implemented by executing python files regularly through shell script, or python's own sleep, and then set it as a daemon in the linux background)
# Process detection function, cannot run alone
def checkprocess(self, target_pid):
self.target_pid = target_pid
pid1, pid2, pid3 = self.target_pid
while True:
print('Monitoring')
process1 = psutil.Process(pid1) # Instantiate three processes according to pid
process2 = psutil.Process(pid2)
process3 = psutil.Process(pid3)
if process1.is_running and process1.status() != psutil.STATUS_ZOMBIE:
# and is added here because the official document of psutil says, is_running will return the result of the zombie process to True. As for the specific status of the process, you can refer to the official documentation of psutil.
pass
else:
self.sendmail(self.receiver, "Process monitoring", "receiver The process has become a zombie process!, Please check the reason")
if process2.is_running and process2.status() != psutil.STATUS_ZOMBIE:
pass
else:
self.sendmail(self.receiver, "Process monitoring", "parser The process has become a zombie process! Please check the reason")
if process3.is_running and process3.status() != psutil.STATUS_ZOMBIE:
pass
else:
self.sendmail(self.receiver, "Process monitoring", "receiver The process has become a zombie process! Please check the reason")
time.sleep(5)
3, Python mail sending function
Use python to send e-mail and encapsulate it into a module using classes. You need to use python's own email and SMTP lib library
- SMTP lib is responsible for creating the SMTP operation object, connecting to the SMTP target server, calling the methods in the object and sending mail to the target address
- The email library is used to create mail objects (common mail objects include plain text mail objects, picture objects as attachments, and mail objects mixed with multiple objects)
- You need to change your qq mailbox settings to log in to your qq mailbox and send mail with python agent. Here, you need to do two things: open the SMTP function of the mailbox and obtain the authorization code. Refer to: https://zhuanlan.zhihu.com/p/25565454
from email.mime.text import MIMEText
from email.header import Header
import smtplib
class SendEmail(object):
def __init__(self):
self.host_server = 'smtp.qq.com'
# Sender's qq number
self.sender_qq = 'xxxxx'
# The authorization code of QQ mailbox needs to receive the smtp service to open QQ mailbox. On the home page, it is set
self.pwd = 'xxx'
self.sender_qq_mail = 'xxx@qq.com' # Sender mailbox
# Create an SMTP operation object and connect to the SMTP target server, which can be 163, QQ, etc
self.smtp = smtplib.SMTP_SSL(self.host_server)
# set_debuglevel() is used for debugging. A parameter value of 1 indicates that the debugging mode is turned on, and a parameter value of 0 indicates that the debugging mode is turned off
# self.smtp.set_debuglevel(1)
self.smtp.ehlo(self.host_server)
self.smtp.login(self.sender_qq, self.pwd) # validate logon
def sendmail(self, receivers, mail_title, mail_content):
msg = MIMEText(mail_content, "plain", 'utf-8') # Construct a text mail object
# Set the header below
# Mail subject
msg["Subject"] = Header(mail_title, 'utf-8')
# Sender
msg["From"] = self.sender_qq_mail
# addressee
msg["To"] = ",".join(receivers)
try:
# msg.as_ As in string()_ String () is to change msg(MIMEText or MIMEMultipart object) to str.
self.smtp.sendmail(self.sender_qq_mail, msg['To'].split(','), msg.as_string())
print("mail has been post successfully")
except smtplib.SMTPException as e:
print(repr(e))
if __name__ == "__main__":
sm = SendEmail()
sm.sendmail(['xxx@163.com', 'xxx@gmail.com'], "python test","Hello, I'm working on a project python Sign in qq Test of e-mail from mailbox")
4, Complete code
The following is the complete code encapsulated into a class:
receiver.py
# Create an analog receiver, generate data circularly and store it in the cache
import time
import multiprocessing
class Recver:
def __init__(self):
self.count = 0
def recver(self, queue):
while True:
self.count += 1
if not queue.full():
print(multiprocessing.current_process().pid, "Process playback data", self.count)
queue.put(self.count)
time.sleep(1)
else:
time.sleep(2)
parser.py
# Receive the data generated by the cycle, analyze it, and then store it in the cache
import multiprocessing
class Parser:
def __init__(self):
self.info = "null"
def parser(self, queue, queue2):
while True:
if not queue.empty():
num = queue.get()
if num == 20:
a = int("hello world")
self.info = "Now count is %d" % a
queue2.put(self.info)
else:
print(multiprocessing.current_process().pid, "Process receiving data", num)
# lock.acquire()
self.info = "Now count is {}".format(num)
# lock.release()
queue2.put(self.info)
else:
pas
publisher.py
# Read from cache and publish
import multiprocessing
class Publisher():
def __init__(self):
pass
def publish(self, queue):
while True:
if not queue.empty():
msg = queue.get()
# lock.acquire()
print("process", multiprocessing.current_process().pid, "Publish info:", msg)
# lock.release()
else:
pass
Monitoring script monitor.py
from email.mime.text import MIMEText
from email.header import Header
import smtplib
import psutil
import time
class Monitor(object):
def __init__(self):
self.target_pid = [] # Target pid to monitor
self.host_server = 'smtp.qq.com' # smtp service using QQ mailbox
self.sender_qq = 'xxx' # Own QQ number
self.pwd = 'xxxxx' # SMTP authorization code of QQ mailbox
self.sender_qq_mail = 'xxx@qq.com' # Sender mailbox, test my own QQ mailbox
self.receiver = ['xxx'] # There can be multiple recipient mailboxes, which are written in the list and separated by commas
self.smtp = smtplib.SMTP_SSL(self.host_server) # Create an SMTP operation object and connect to the SMTP target server, which can be 163, QQ, etc
# set_debuglevel() is used for debugging. A parameter value of 1 indicates that the debugging mode is turned on, and a parameter value of 0 indicates that the debugging mode is turned off
# self.smtp.set_debuglevel(1)
self.smtp.ehlo(self.host_server)
self.smtp.login(self.sender_qq, self.pwd) # validate logon
def sendmail(self, receivers, mail_title, mail_content):
# Constructing a MIMEText object represents a text mail object
msg = MIMEText(mail_content, "plain", 'utf-8')
# Set the header below
# Mail subject
msg["Subject"] = Header(mail_title, 'utf-8')
# Sender
msg["From"] = self.sender_qq_mail
# addressee
msg["To"] = ",".join(receivers)
try:
# msg.as_ As in string()_ String () is to change msg(MIMEText or MIMEMultipart object) to str.
self.smtp.sendmail(self.sender_qq_mail, msg['To'].split(','), msg.as_string())
self.smtp.quit()
print("mail has been post successfully")
except smtplib.SMTPException as e:
print(repr(e))
def checkprocess(self, target_pid):
self.target_pid = target_pid
pid1, pid2, pid3 = self.target_pid
while True:
print('Monitoring')
process1 = psutil.Process(pid1) # Instantiate three processes according to pid
process2 = psutil.Process(pid2)
process3 = psutil.Process(pid3)
if process1.is_running and process1.status() != psutil.STATUS_ZOMBIE:
# print('process1 status:', process1.status)
pass
else:
self.sendmail(self.receiver, "Process monitoring", "receiver The process has become a zombie process!, Please check the reason")
if process2.is_running and process2.status() != psutil.STATUS_ZOMBIE:
pass
else:
self.sendmail(self.receiver, "Process monitoring", "parser The process has become a zombie process! Please check the reason")
if process3.is_running and process3.status() != psutil.STATUS_ZOMBIE:
pass
else:
self.sendmail(self.receiver, "Process monitoring", "receiver The process has become a zombie process! Please check the reason")
time.sleep(5)
Start script start.py
from receiver import Recver
from parser import Parser
from publisher import Publisher
from monitor import Monitor
import multiprocessing
queue1 = multiprocessing.Queue(maxsize=5)
queue2 = multiprocessing.Queue(maxsize=5)
queue_pid = multiprocessing.Queue(maxsize=3)
if __name__ == '__main__':
jieshou = Recver()
jiexi = Parser()
publisher = Publisher()
monitor = Monitor()
process1 = multiprocessing.Process(target=jieshou.recver, args=(queue1,), daemon=True)
process2 = multiprocessing.Process(target=jiexi.parser, args=(queue1, queue2), daemon=True)
process3 = multiprocessing.Process(target=publisher.publish, args=(queue2,), daemon=True)
process1.start()
process2.start()
process3.start()
target_pid = (process1.pid, process2.pid, process3.pid)
process4 = multiprocessing.Process(target=monitor.checkprocess, args=(target_pid,), daemon=True)
process4.start()
process1.join()
# process2.join()