[toc]
Example:
In [1]: import asyncio
In [2]: async def f(i):
...: await asyncio.sleep(i)
...: print(i)
...:
In [3]: async def func():
...: tasks = []
...: for i in range(10):
...: await asyncio.sleep(0)
...: print('create process parameter ', i)
...: tasks.append(asyncio.create_task(f(i)))
...: _ = [await t for t in tasks]
...:
In [4]: asyncio.run(func())
Create orchestration parameter 0
Create process parameter 1
0
Create process parameter 2
Create process parameter 3
Create orchestration parameter4
Create orchestration parameter5
Create process parameter 6
Create orchestration parameter7
Create process parameter8
Create orchestration parameter9
1
2
3
4
5
6
7
8
9Start looking at the source code
The source code starts from asyncio.run():
# asyncio.runners.py def run(main, *, debug=False): """Run a coroutine. ... """ if events._get_running_loop() is not None: # Check whether there is a current loop instance. If there is, an exception will be reported raise RuntimeError( "asyncio.run() cannot be called from a running event loop") if not coroutines.iscoroutine(main): # Check whether main is a collaboration object, otherwise an exception will be reported raise ValueError("a coroutine was expected, got {!r}".format(main)) loop = events.new_event_loop() # Create event loop instance try: events.set_event_loop(loop) # Bind event loop instance loop.set_debug(debug) # Set debug mode return loop.run_until_complete(main) # Start executing the main function in the event loop until the main function finishes running finally: try: _cancel_all_tasks(loop) loop.run_until_complete(loop.shutdown_asyncgens()) finally: events.set_event_loop(None) loop.close()
The above process is to create a loop and execute functions in the loop. Let's take a look at the event loop instance created by events. New? Event? Loop():
# asyncio.events.py def new_event_loop(): """Equivalent to calling get_event_loop_policy().new_event_loop().""" return get_event_loop_policy().new_event_loop() # Call the new event loop() method to return a loop instance def get_event_loop_policy(): """Get the current event loop policy.""" if _event_loop_policy is None: _init_event_loop_policy() # Initialize loop if it is empty return _event_loop_policy # _Event loop policy is an event loop policy, a global variable, initially None def _init_event_loop_policy(): # Initialize global variable "event" loop "policy global _event_loop_policy with _lock: # Lock up if _event_loop_policy is None: # pragma: no branch from . import DefaultEventLoopPolicy _event_loop_policy = DefaultEventLoopPolicy() # Initialize and get an instance using the default DefaultEventLoopPolicy
The loop instance is created by the new event loop() method of the DefaultEventLoopPolicy() instance.
The DefaultEventLoopPolicy class is different in linux and window s:
If my computer is linux, please refer to defaulteventlooppo - > defaulteventlooppolicy = unixdefaulteventlooppolicy under linux:
# asyncio.unix_events.py class _UnixDefaultEventLoopPolicy(events.BaseDefaultEventLoopPolicy): """UNIX event loop policy with a watcher for child processes.""" _loop_factory = _UnixSelectorEventLoop # loop factory ... ... # asyncio.events.py class BaseDefaultEventLoopPolicy(AbstractEventLoopPolicy): _loop_factory = None class _Local(threading.local): # Save a global variable, only the current thread can access it _loop = None _set_called = False def __init__(self): self._local = self._Local() def get_event_loop(self): # Get the loop of the current thread, otherwise create """Get the event loop. This may be None or an instance of EventLoop. """ if (self._local._loop is None and not self._local._set_called and isinstance(threading.current_thread(), threading._MainThread)): self.set_event_loop(self.new_event_loop()) if self._local._loop is None: raise RuntimeError('There is no current event loop in thread %r.' % threading.current_thread().name) return self._local._loop def set_event_loop(self, loop): # Set the loop of the current thread and mark that it has been created """Set the event loop.""" self._local._set_called = True assert loop is None or isinstance(loop, AbstractEventLoop) self._local._loop = loop def new_event_loop(self): # Create loop """Create a new event loop. You must call set_event_loop() to make this the current event loop. """ return self._loop_factory() # Loop is created by the "loop" factory, i.e. "UnixSelectorEventLoop"
This loop is an instance of the UnixSelectorEventLoop. With the loop, the next step is to run \ until \ complete. The run \ until \ complete method does not define and view its parent class in the \ UnixSelectorEventLoop class:
unix_events._UnixSelectorEventLoop ---> selector_events.BaseSelectorEventLoop ---> base_events.BaseEventLoop --> events.AbstractEventLoop
Find the method in the BaseEventLoop:
# asyncio.base_events.py class BaseEventLoop(events.AbstractEventLoop): ... ... def run_until_complete(self, future): """Run until the Future is done. If the argument is a coroutine, it is wrapped in a Task. WARNING: It would be disastrous to call run_until_complete() with the same coroutine twice -- it would wrap it in two different Tasks and that can't be good. Return the Future's result, or raise its exception. """ self._check_closed() # Check whether the current loop loop is closed and an exception has been reported new_task = not futures.isfuture(future) # Determine whether the future passed in is a future or task type future = tasks.ensure_future(future, loop=self) # If the future is coroutine, call loop. Create? Task() to create the task and add it to the loop, then return the task if new_task: # An exception is raised if the future didn't complete, so there # is no need to log the "destroy pending task" message future._log_destroy_pending = False future.add_done_callback(_run_until_complete_cb) # Add the callback function when the task is completed, and stop the loop after the task is completed try: self.run_forever() # Start loop events except: if new_task and future.done() and not future.cancelled(): # Task completed and not cancelled # The coroutine raised a BaseException. Consume the exception # to not log a warning, the caller doesn't have access to the # local task. future.exception() # Trigger exception in task execution raise finally: future.remove_done_callback(_run_until_complete_cb) # Remove callback function of task if not future.done(): # Report error when the task is not completed raise RuntimeError('Event loop stopped before Future completed.') return future.result() # Return task execution results
Next, let's see how task s are created. Ensure (future, loop = self):
# asyncio.tasks.py def ensure_future(coro_or_future, *, loop=None): """Wrap a coroutine or an awaitable in a future. If the argument is a Future, it is returned directly. """ if coroutines.iscoroutine(coro_or_future): # If Coro or future is a coprocessing function if loop is None: loop = events.get_event_loop() task = loop.create_task(coro_or_future) # Call the create task method of loop if task._source_traceback: del task._source_traceback[-1] return task elif futures.isfuture(coro_or_future): # If it is future, return directly if loop is not None and loop is not futures._get_loop(coro_or_future): raise ValueError('loop argument must agree with Future') return coro_or_future elif inspect.isawaitable(coro_or_future): # If it is a wait object, i.e. a contract object, use the wrap awaitable package to call this function again return ensure_future(_wrap_awaitable(coro_or_future), loop=loop) else: raise TypeError('An asyncio.Future, a coroutine or an awaitable is ' 'required') @coroutine def _wrap_awaitable(awaitable): # After being wrapped by this function, it becomes a co process function """Helper for asyncio.ensure_future(). Wraps awaitable (an object with __await__) into a coroutine that will later be wrapped in a Task by ensure_future(). """ return (yield from awaitable.__await__())
Next, let's look at the loop. Create ﹣ task (core ﹣ or ﹣ future). How the task is created is shown in the BaseEventLoop():
# asyncio.base_events def create_task(self, coro): """Schedule a coroutine object. Return a task object. """ self._check_closed() if self._task_factory is None: # If the task factory is empty (the task factory can be set) task = tasks.Task(coro, loop=self) # Call tasks.Task to create a task if task._source_traceback: del task._source_traceback[-1] else: # Use this factory function to create if the task factory is not empty task = self._task_factory(self, coro) return task
Then look at how the Task class is initialized:
class Task(futures._PyFuture): # Inherit Python Task implementation # Collaboration inherited from Future # from a Python Future implementation. """A coroutine wrapped in a Future.""" _log_destroy_pending = True @classmethod def current_task(cls, loop=None): # Returns the currently running task in the event loop, or none. warnings.warn("Task.current_task() is deprecated, " "use asyncio.current_task() instead", PendingDeprecationWarning, stacklevel=2) if loop is None: loop = events.get_event_loop() return current_task(loop) def __init__(self, coro, *, loop=None): super().__init__(loop=loop) if self._source_traceback: del self._source_traceback[-1] if not coroutines.iscoroutine(coro): # raise after Future.__init__(), attrs are required for __del__ # prevent logging for pending task in __del__ self._log_destroy_pending = False raise TypeError(f"a coroutine was expected, got {coro!r}") self._must_cancel = False self._fut_waiter = None self._coro = coro # coro is a coprocessing function self._context = contextvars.copy_context() # contextvars is used to store the management context. The method here is to copy the current context self._loop.call_soon(self.__step, context=self._context) # Invoke the self.__step method in the iteration of the next event cycle. The environment variable is self._context. _register_task(self) # Store record current task def set_result(self, result): raise RuntimeError('Task does not support set_result operation') def set_exception(self, exception): raise RuntimeError('Task does not support set_exception operation') def get_stack(self, *, limit=None): """Return the list of stack frames for this task's coroutine. ... """ return base_tasks._task_get_stack(self, limit) def print_stack(self, *, limit=None, file=None): """Print the stack or traceback for this task's coroutine. ... """ return base_tasks._task_print_stack(self, limit, file) def cancel(self): """Request that this task cancel itself. ... """ self._log_traceback = False if self.done(): return False if self._fut_waiter is not None: if self._fut_waiter.cancel(): # Leave self._fut_waiter; it may be a Task that # catches and ignores the cancellation so we may have # to cancel it again later. return True # It must be the case that self.__step is already scheduled. self._must_cancel = True return True def __step(self, exc=None): if self.done(): # __step has run and reported an exception raise futures.InvalidStateError( f'_step(): already done: {self!r}, {exc!r}') if self._must_cancel: # if not isinstance(exc, futures.CancelledError): exc = futures.CancelledError() self._must_cancel = False coro = self._coro # The main body of a coprocessor self._fut_waiter = None _enter_task(self._loop, self) # Set the currently running task # Call either coro.throw(exc) or coro.send(None). try: if exc is None: # If there is no exception # We use the `send` method directly, because coroutines # don't have `__iter__` and `__next__` methods. result = coro.send(None) # Execute the send method of the coroutine function and return the result else: result = coro.throw(exc) # Send an exception to the coroutine function (the coroutine function may handle the exception and then trigger Stopiteration) except StopIteration as exc: # The execution of the coroutine function is completed. If any returned result exists in exc.value if self._must_cancel: # If the task is cancelled # Task is cancelled right before coro stops. self._must_cancel = False super().set_exception(futures.CancelledError()) # Set cancelederror() exception to Future (task) else: super().set_result(exc.value) # Set the result to Future (task) and set the status to complete except futures.CancelledError: super().cancel() # I.e., Future.cancel(self). except Exception as exc: # The following are exceptions to Future settings super().set_exception(exc) except BaseException as exc: super().set_exception(exc) raise # Trigger exception else: blocking = getattr(result, '_asyncio_future_blocking', None) # Whether the result has the 'asyncio' and 'future' blocking 'property, and if it is not false, it is a future?? , see the asyncio.base'futures.isfuture function if blocking is not None: # blocking is not None may be False or True # Yielded Future must come from Future.__iter__(). if futures._get_loop(result) is not self._loop: # Whether the loop of the task is the current loop is not to pass the error to the coroutine function new_exc = RuntimeError( f'Task {self!r} got Future ' f'{result!r} attached to a different loop') self._loop.call_soon( self.__step, new_exc, context=self._context) # Pass the error to the coroutine function elif blocking: # True, result is a future object (at this time, the future has been added to the next loop loop loop?) if result is self: # If the result is the current task, pass the exception to the coroutine function to handle new_exc = RuntimeError( f'Task cannot await on itself: {self!r}') self._loop.call_soon( self.__step, new_exc, context=self._context) # Pass the exception to the coroutine function to handle else: result._asyncio_future_blocking = False # Set result's asyncio future blocking to False result.add_done_callback( self.__wakeup, context=self._context) # Set the callback function to result in the same context as this task self._fut_waiter = result # Set "ft" waiter. If "ft" waiter is not None, the current task will not end if self._must_cancel: # If the current status is cancelled, then result future will also be cancelled if self._fut_waiter.cancel(): self._must_cancel = False else: # If "asyncio" future "blocking is false, then result future is not represented by yield from? new_exc = RuntimeError( f'yield was used instead of yield from ' f'in task {self!r} with {result!r}') self._loop.call_soon( self.__step, new_exc, context=self._context) # Give the exception to the coroutine function elif result is None: # If there is no result output after the. send function, then. send will be executed in the next loop loop loop # Bare yield relinquishes control for one event loop iteration. self._loop.call_soon(self.__step, context=self._context) elif inspect.isgenerator(result): # If the result returned is a generator # Yielding a generator is just wrong. new_exc = RuntimeError( # The generator should be represented by yield from f'yield was used instead of yield from for ' f'generator in task {self!r} with {result!r}') self._loop.call_soon( self.__step, new_exc, context=self._context) else: # It's wrong to produce other results # Yielding something else is an error. new_exc = RuntimeError(f'Task got bad yield: {result!r}') self._loop.call_soon( self.__step, new_exc, context=self._context) finally: _leave_task(self._loop, self) # Switching task s in the loop self = None # Needed to break cycles when an exception occurs. def __wakeup(self, future): # Callback function added to result future try: future.result() # Get the result of result future except Exception as exc: # This may also be a cancellation. self.__step(exc) # In case of any exception, it shall be handled by the co procedure function else: # Don't pass the value of `future.result()` explicitly, # as `Future.__iter__` and `Future.__await__` don't need it. # If we call `_step(value, None)` instead of `_step()`, # Python eval loop would use `.send(value)` method call, # instead of `__next__()`, which is slower for futures # that return non-generator iterators from their `__iter__`. self.__step() # When the result future of the cooperation function is finished, continue to execute the \ self = None # Needed to break cycles when an exception occurs.
⇑ ps: the ⇑ asyncio ﹐ future ﹐ blocking flag will be set to True in ﹐ future, and ﹐ wait ﹐ will be represented by ﹐ wrap ﹐ await able function above to yield from
# asyncio.future.Future() def __await__(self): if not self.done(): self._asyncio_future_blocking = True yield self # This tells Task to wait for completion. if not self.done(): raise RuntimeError("await wasn't used with future") return self.result() # May raise too.
Next, let's see how the loop loops through the task s:
# base_event.BaseEventLoop def call_soon(self, callback, *args, context=None): """Arrange for a callback to be called as soon as possible. This operates as a FIFO queue: callbacks are called in the order in which they are registered. Each callback will be called exactly once. Any positional arguments after the callback will be passed to the callback when it is called. """ self._check_closed() if self._debug: self._check_thread() self._check_callback(callback, 'call_soon') handle = self._call_soon(callback, args, context) # Call on if handle._source_traceback: del handle._source_traceback[-1] return handle # Return to handle def _call_soon(self, callback, args, context): handle = events.Handle(callback, args, self, context) # Create handle if handle._source_traceback: del handle._source_traceback[-1] self._ready.append(handle) # Add handle to self. \ return handle ... def run_forever(self): """Run until stop() is called.""" self._check_closed() if self.is_running(): raise RuntimeError('This event loop is already running') if events._get_running_loop() is not None: raise RuntimeError( 'Cannot run the event loop while another loop is running') self._set_coroutine_origin_tracking(self._debug) self._thread_id = threading.get_ident() # Thread id old_agen_hooks = sys.get_asyncgen_hooks() sys.set_asyncgen_hooks(firstiter=self._asyncgen_firstiter_hook, finalizer=self._asyncgen_finalizer_hook) # I don't know what these two lines do.. try: events._set_running_loop(self) # Set event loop while True: # Start loop execution self._run_once() # Perform a cycle if self._stopping: break finally: self._stopping = False self._thread_id = None events._set_running_loop(None) self._set_coroutine_origin_tracking(False) sys.set_asyncgen_hooks(*old_agen_hooks) ... def _run_once(self): """Run one full iteration of the event loop. This calls all currently ready callbacks, polls for I/O, schedules the resulting callbacks, and finally schedules 'call_later' callbacks. """ sched_count = len(self._scheduled) # Number of scheduled tasks if (sched_count > _MIN_SCHEDULED_TIMER_HANDLES and # Clean up the cancelled timed handle? self._timer_cancelled_count / sched_count > _MIN_CANCELLED_TIMER_HANDLES_FRACTION): # Remove delayed calls that were cancelled if their number # is too high new_scheduled = [] for handle in self._scheduled: if handle._cancelled: handle._scheduled = False else: new_scheduled.append(handle) heapq.heapify(new_scheduled) self._scheduled = new_scheduled self._timer_cancelled_count = 0 else: # Remove delayed calls that were cancelled from head of queue. while self._scheduled and self._scheduled[0]._cancelled: self._timer_cancelled_count -= 1 handle = heapq.heappop(self._scheduled) handle._scheduled = False timeout = None if self._ready or self._stopping: timeout = 0 elif self._scheduled: # Compute the desired timeout. when = self._scheduled[0]._when timeout = max(0, when - self.time()) if self._debug and timeout != 0: t0 = self.time() event_list = self._selector.select(timeout) dt = self.time() - t0 if dt >= 1.0: level = logging.INFO else: level = logging.DEBUG nevent = len(event_list) if timeout is None: logger.log(level, 'poll took %.3f ms: %s events', dt * 1e3, nevent) elif nevent: logger.log(level, 'poll %.3f ms took %.3f ms: %s events', timeout * 1e3, dt * 1e3, nevent) elif dt >= 1.0: logger.log(level, 'poll %.3f ms took %.3f ms: timeout', timeout * 1e3, dt * 1e3) else: event_list = self._selector.select(timeout) # Under linux, select the fb with event by SelectSelector self._process_events(event_list) # Add the handle of event to self. \ # Handle 'later' callbacks that are ready. end_time = self.time() + self._clock_resolution while self._scheduled: handle = self._scheduled[0] if handle._when >= end_time: # Timed task time is greater than current event exit cycle break handle = heapq.heappop(self._scheduled) # pop out the header of the scheduled task handle._scheduled = False # Mark executed self._ready.append(handle) # Add this time to self. # This is the only place where callbacks are actually *called*. # All other places just add them to ready. # Note: We run all currently scheduled callbacks, but not any # callbacks scheduled by callbacks run this time around -- # they will be run the next time (after another I/O poll). # Use an idiom that is thread-safe without using locks. ntodo = len(self._ready) for i in range(ntodo): # Start processing tasks in "ready" one by one handle = self._ready.popleft() if handle._cancelled: # If it has been cancelled, next continue if self._debug: try: self._current_handle = handle t0 = self.time() handle._run() dt = self.time() - t0 if dt >= self.slow_callback_duration: logger.warning('Executing %s took %.3f seconds', _format_handle(handle), dt) finally: self._current_handle = None else: handle._run() # Execute the code managed by this processor, i.e. task handle = None # Needed to break cycles when an exception occurs.
Example analysis
After reading the source code, start to analyze step by step how the code in the example is written by switching the process and execution:
await + wait object is to create a Task and then add it to the loop. For example, asyncio.sleep is a function defined by async def. We can see from the source code that when creating a Task, functions defined by async def compound statements are all represented by yield from. So
await asyncio.sleep(n)
It can be written as:
yield from asyncio.sleep(n).__await__() # . it can be omitted
But yield from can't be written in async def, so you can use the co process function (non co process object), so the above example can be written as:
In [2]: @asyncio.coroutine ...: def f(i): ...: yield from asyncio.sleep(i).__await__() ...: print(i) ...: In [3]: @asyncio.coroutine ...: def func(): ...: tasks = [] ...: for i in range(10): ...: yield from asyncio.sleep(0) ...: print('Create a process parameter',i) ...: tasks.append(asyncio.create_task(f(i))) ...: for t in tasks: ...: yield from t ...: In [4]: asyncio.run(func())
The cooperation function is different from the cooperation object (waiting object). In the ensure future function, if it is a cooperation function, the Task object will be created directly. If it is a cooperation object, it will be encapsulated by the wrap awaitable first and then the Task object will be created.
It's easier to analyze when await is converted to yield from. Start to analyze the example step by step:
- Create loop, the first loop loop loop = []: create func() Task and record it as m, and execute "step -- > send (none) to return the first result;
- The second loop loop=[m]: the first result is the delay Task(Future type) created by asyncio.sleep(0), which is recorded as -- > SM, and then its "step" is added to the loop, which is executed in the next loop, sleep 0 seconds;
- Loop start next loop=[sm]: sm is finished, and then the last suspended position is executed downward. Print the creation process parameter 0, and then create the task (f(0). At this time, the task. Step has been directly added to the loop and recorded as --- > F1. Then the next for loop returns the second result, sleep future, -- > sm.
- Loop start next loop=[f1, sm]: execute f1, and sleep > SF1 is encountered,. Switch the cooperation sm, and then execute it from the last suspended position. Print the creation cooperation parameter 1, create the cooperation -- > F2, and the next for loop sleep -- > sm.
- Loop start next loop=[sf1, f2, sm]: sf1, then f(0) function is executed where it was last suspended, printing 0, switching f2, sleep -- > SF2 is suspended. Switch the cooperation sm, and then execute it from the last suspended position. Print the creation cooperation parameter 2, create the cooperation -- > F3, and the next for loop sleep -- > sm.
- Loop start next loop=[sf2, f3, sm]: