background
In performance test, sometimes an API request can only be processed once for the same data (such as user registration), or can only be executed sequentially. Multiple users are not allowed to operate on the data at the same time. Different data can be requested concurrently. Therefore, data sampling needs to be handled to avoid multiple users sampling the same data at the same time.
Python queue
Queue is a standard library in Python, commonly known as queue, which can be directly referenced by import.
In Python, data between multiple threads is shared. When multiple threads exchange data, the security and consistency of data cannot be guaranteed. Therefore, when multiple threads need to exchange data, queues appear. Queues can perfectly solve the data exchange between threads and ensure the security and consistency of data between threads.
The queue will adopt the first in first out or first in last out mode to ensure that a single data will not be accessed by multiple threads at the same time.
The queue module has three types of queues and constructors:
If maxsize is less than 1, the queue length is infinite
FIFO Queue first in first out. class queue.Queue(maxsize=0) LIFO Similar to heap, i.e. first in and last out. class queue.LifoQueue(maxsize=0) The lower the priority queue level, the first out. class queue.PriorityQueue(maxsize=0)
Common methods in the queue module:
queue.qsize() Returns the size of the queue queue.empty() If the queue is empty, return True,conversely False queue.full() If the queue is full, return True,conversely False queue.full And maxsize Size correspondence queue.get([block[, timeout]])Get queue, timeout waiting time queue.get_nowait() Quite queue.get(False) Queue.put(item, [block[, timeout]]) Write to queue, timeout waiting time queue.put_nowait(item) Quite queue.put(item, False) queue.task_done() After completing a job, queue.task_done()Function sends a signal to the queue where the task has completed queue.join() In fact, it means waiting until the queue is empty before performing other operations
Locust application
In the HttpUser class, generate queue data
class TestLocust(HttpUser):
queue_list = queue.Queue()
for i in range(1, 10):
queue_list.put_nowait(i)
In the task method, pass self.parent.queue_list.get() to access the data in the queue. If you want to cycle the queue data, you can put the retrieved data back into the queue, self.parent.queue_list.put_nowait(data)
class UserBehavior(TaskSet):
@task
def get_root(self):
data = self.parent.queue_list.get()
print("queue data:{}".format(data))
# self.parent.queue_list.put_nowait(data)
Let's take a look at the application of Locust multi-user concurrent queue queue through a practical example. In order to better show the concurrency process, add some log information. The interval between each task is 5s to simulate the concurrency of two users.
1. Cycle data retrieval, and the data is not repeated
from locust import TaskSet, task, HttpUser
import os
from locust.user.wait_time import between
import queue
class UserBehavior(TaskSet):
def on_start(self):
print('taskset start')
self.root_index = 0
def on_stop(self):
print('taskset end')
@task
def get_root(self):
print("get_root task")
print("root index : " + str(self.root_index))
self.root_index += 1
if not self.parent.queue_list.empty():
data = self.parent.queue_list.get()
print("queue data:{}".format(data))
response = self.client.get('',name='get_root')
else:
print("no data exist")
exit(0)
if not response.ok:
print(response.text)
response.failure('Got wrong response')
class TestLocust(HttpUser):
# If no wait_time is specified, the next task will be executed as soon as one finishes.
wait_time = between(5,5)
def on_start(self):
print('locust user start')
def on_stop(self):
print('locust user stop')
tasks = [UserBehavior]
host = "https://cn.bing.com"
queue_list = queue.Queue()
for i in range(1, 6):
queue_list.put_nowait(i)
if __name__ == "__main__":
# -u concurrency user number
# -r generate user number per second
# --run-time or -t
os.system("locust -f test_locust.py --headless -u 2 -r 1 --run-time 20s --stop-timeout 5")
Output:
It can be seen that both user s maintain their own variable root index, but the data of the queue is taken out in the order of first in first out until the queue is empty, and there is no duplicate data.
locust user start taskset start get_root task root index : 0 queue data:1 locust user start taskset start get_root task root index : 0 queue data:2 get_root task root index : 1 queue data:3 get_root task root index : 1 queue data:4 get_root task root index : 2 queue data:5 get_root task root index : 2 no data exist
2. Cyclic data retrieval, data duplication
Put the data retrieved from the queue back to the end of the queue, so the sequence data will not be empty.
from locust import TaskSet, task, HttpUser
import os
from locust.user.wait_time import between
import queue
class UserBehavior(TaskSet):
def on_start(self):
print('taskset start')
self.root_index = 0
def on_stop(self):
print('taskset end')
@task
def get_root(self):
print("get_root task")
print("root index : " + str(self.root_index))
self.root_index += 1
if not self.parent.queue_list.empty():
data = self.parent.queue_list.get()
print("queue data:{}".format(data))
# put the data back to the queue
self.parent.queue_list.put_nowait(data)
response = self.client.get('',name='get_root')
else:
print("no data exist")
exit(0)
if not response.ok:
print(response.text)
response.failure('Got wrong response')
class TestLocust(HttpUser):
# If no wait_time is specified, the next task will be executed as soon as one finishes.
wait_time = between(5,5)
def on_start(self):
print('locust user start')
def on_stop(self):
print('locust user stop')
tasks = [UserBehavior]
host = "https://cn.bing.com"
queue_list = queue.Queue()
for i in range(1, 6):
queue_list.put_nowait(i)
if __name__ == "__main__":
# -u concurrency user number
# -r generate user number per second
# --run-time or -t
os.system("locust -f performance_test/test_locust.py --headless -u 2 -r 1 --run-time 20s --stop-timeout 5 --logfile log.txt --csv=example")
Output:
It can be seen that both user s maintain their own variable root index, but the data of the queue is taken out in the order of first in first out, and then the number taken out is put back to the end of the queue. Therefore, the queue will not be empty, and the data will be repeated by cyclic data retrieval.
locust user start taskset start get_root task root index : 0 queue data:1 locust user start taskset start get_root task root index : 0 queue data:2 get_root task root index : 1 queue data:3 get_root task root index : 1 queue data:4 get_root task root index : 2 queue data:5 get_root task root index : 2 queue data:1 get_root task root index : 3 queue data:2 get_root task root index : 3 queue data:3 taskset end locust user stop taskset end locust user stop