Python collections.deque vs. Queue.Queue vs. multiprocessing.Queue

Keywords: Python

Generally speaking, multiprocessing.Queue is used when communication between processes is needed; Queue.Queue is used when communication between threads is needed in the same process; and collections.deque is usually used as a data structure in the same thread. Here's how they are used:

multiprocessing.Queue

multiprocessing provides two mechanisms for interprocess communication, one is Queue, the other is Pipe. In fact, Queue is also implemented through Pipe. Specific reference Interprocess communication
Queue commonly used methods:

  • Queue.qsize(): Return the number of item s in queue. Note that the number is not reliable.
  • Queue.empty(): return True if queue is empty, not reliable
  • Queue.full(): return True if queue is full, not reliable
  • Queue.put(item[, block[, timeout]]): block indicates whether or not blocking is True, which means blocking. If timeout is set, the blocking timeout time is long. If there is still no free slot, raise Queue.full exception. If block=False, then there is no blocking. When queue full, report Queue.full exception directly.
  • Queue.put_nowait(item): Equivalent to put(item, False)
  • Queue.get([block[, timeout]])
  • Queue.get_nowait(): Equivalent to get(False)

Example:

from multiprocessing import Process, Queue

def f(q):
    q.put([42, None, 'hello'])

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    print q.get()    # prints "[42, None, 'hello']"
    p.join()

Note that Queue does not provide join() and task_done(), so it is not possible to ensure that all tasks are processed in the producer process. If you need join and task_done, you need to use multiprocessing.JoinableQueue. See details. JoinableQueue

Queue.Queue

Queue.Queue is usually used for communication between different threads in the same process. It provides a method similar to multiprocessing.Queue, but there are two more methods as follows:

  • task_done(): Used to inform task completion
  • join(): used to wait for all tasks in the queue to complete. The specific use is shown in the figure below.


    queue

collections.deque

This data structure is mainly used for queues. The FIFO mechanism of queues is implemented by append and popleft. Common methods are as follows:

  • extendleft
  • appendleft
  • popleft
  • extend
  • append
  • pop
    Specific reference Official website
    Example:
>>> from collections import deque
>>> d = deque('ghi')                 # make a new deque with three items
>>> for elem in d:                   # iterate over the deque's elements
...     print elem.upper()
G
H
I

>>> d.append('j')                    # add a new entry to the right side
>>> d.appendleft('f')                # add a new entry to the left side
>>> d                                # show the representation of the deque
deque(['f', 'g', 'h', 'i', 'j'])

>>> d.pop()                          # return and remove the rightmost item
'j'
>>> d.popleft()                      # return and remove the leftmost item
'f'
>>> list(d)                          # list the contents of the deque
['g', 'h', 'i']
>>> d[0]                             # peek at leftmost item
'g'
>>> d[-1]                            # peek at rightmost item
'i'

>>> list(reversed(d))                # list the contents of a deque in reverse
['i', 'h', 'g']
>>> 'h' in d                         # search the deque
True
>>> d.extend('jkl')                  # add multiple elements at once
>>> d
deque(['g', 'h', 'i', 'j', 'k', 'l'])
>>> d.rotate(1)                      # right rotation
>>> d
deque(['l', 'g', 'h', 'i', 'j', 'k'])
>>> d.rotate(-1)                     # left rotation
>>> d
deque(['g', 'h', 'i', 'j', 'k', 'l'])

>>> deque(reversed(d))               # make a new deque in reverse order
deque(['l', 'k', 'j', 'i', 'h', 'g'])
>>> d.clear()                        # empty the deque
>>> d.pop()                          # cannot pop from an empty deque
Traceback (most recent call last):
  File "<pyshell#6>", line 1, in -toplevel-
    d.pop()
IndexError: pop from an empty deque

>>> d.extendleft('abc')              # extendleft() reverses the input order
>>> d
deque(['c', 'b', 'a'])

Posted by willwill100 on Mon, 17 Dec 2018 04:03:04 -0800