The Development Process of Python Coroutines and an In-depth Analysis of Old and New Coroutines

1. The Historical Evolution of Python Coroutines

Throughout the long development of Python, the implementation of coroutines has undergone several significant changes. Understanding these changes helps us better grasp the essence of Python asynchronous programming.

1.1 Early Exploration and Introduction of Basic Functions

• Python 2.5: This version introduced the .send(), .throw(), and .close() methods to generators. The appearance of these methods made generators more than just simple iterators; they began to possess more complex interaction capabilities, laying a certain foundation for the development of coroutines. For example, the .send() method can send data into the generator, breaking the previous limitation that generators could only output unidirectionally.
• Python 3.3: The yield from syntax was introduced, which was an important milestone. It enabled generators to receive return values and allowed coroutines to be defined directly using yield from. This feature simplified the writing of coroutines and enhanced the readability and maintainability of the code. For instance, with yield from, a complex generator operation can be conveniently delegated to another generator, achieving code reuse and logical separation.

1.2 Standard Library Support and Syntax Refinement

• Python 3.4: The asyncio module was added, which was a crucial step for Python towards modern asynchronous programming. asyncio provides an event-loop-based asynchronous programming framework, enabling developers to write highly efficient asynchronous code more conveniently. It offers a powerful infrastructure for running coroutines, including core functions such as event loops and task management.
• Python 3.5: The async and await keywords were added, providing more direct and clearer support for asynchronous programming at the syntax level. The async keyword is used to define an asynchronous function, indicating that the function is a coroutine; the await keyword is used to pause the execution of a coroutine and wait for the completion of an asynchronous operation. This syntactic sugar makes the writing style of asynchronous code closer to that of synchronous code, greatly reducing the threshold for asynchronous programming.

1.3 Maturity and Optimization Stages

• Python 3.7: The way of defining coroutines using async def + await was formally established. This method is more concise and straightforward and has become the standard way to define coroutines in Python. It further strengthened the syntactic structure of asynchronous programming, enabling developers to write asynchronous code more naturally.
• Python 3.10: Defining coroutines with yield from was removed, marking a new stage in the development of Python coroutines. It became more focused on the new coroutine system based on async and await, reducing the confusion caused by different implementation methods.

1.4 Concepts and Impacts of Old and New Coroutines

Old coroutines are implemented based on generator syntax such as yield and yield from. New coroutines, on the other hand, are based on keywords like asyncio, async, and await. In the history of coroutine development, the two implementation methods had an intersection period. However, the generator-based syntax of old coroutines made it easy to confuse the concepts of generators and coroutines, causing some difficulties for learners. Therefore, deeply understanding the differences between the two implementation methods of coroutines is crucial for mastering Python asynchronous programming.

2. Review of Old Coroutines

2.1 Core Mechanism: The Magic of the yield Keyword

The core of old coroutines lies in the yield keyword, which endows functions with powerful capabilities, including pausing and resuming code execution, alternating execution between functions, and transferring CPU resources.

2.2 Analysis of Code Examples

import time


def consume():
    r = ''
    while True:
        n = yield r
        print(f'[consumer] Starting to consume {n}...')
        time.sleep(1)
        r = f'{n} consumed'


def produce(c):
    next(c)
    n = 0
    while n < 5:
        n = n + 1
        print(f'[producer] Produced {n}...')
        r = c.send(n)
        print(f'[producer] Consumer return: {r}')
    c.close()


if __name__ == '__main__':
    c = consume()
    produce(c)

In this example, the consume function is a consumer coroutine, and the produce function is a producer function. The while True loop in the consume function allows it to keep running. The line n = yield r is crucial. When the execution reaches this line, the execution flow of the consume function pauses, returns the value of r to the caller (i.e., the produce function), and saves the current state of the function.

The produce function starts the consume coroutine via next(c) and then enters its own while loop. In each loop, it produces a piece of data (n) and sends the data to the consume coroutine through c.send(n). c.send(n) not only sends data to the consume coroutine but also resumes the execution of the consume coroutine, making it continue from the line n = yield r.

2.3 Execution Results and Analysis

Execution results:

[producer] Produced 1...
[consumer] Starting to consume 1...
[producer] Consumer return: 1 consumed
[producer] Produced 2...
[consumer] Starting to consume 2...
[producer] Consumer return: 2 consumed
[producer] Produced 3...
[consumer] Starting to consume 3...
[producer] Consumer return: 3 consumed
[producer] Produced 4...
[consumer] Starting to consume 4...
[producer] Consumer return: 4 consumed
[producer] Produced 5...
[consumer] Starting to consume 5...
[producer] Consumer return: 5 consumed

When the consumer consume executes to n = yield r, the process pauses and returns the CPU to the caller produce. In this example, the while loops in consume and produce work together to simulate a simple event loop function. And the yield and send methods implement the pausing and resuming of tasks.

To summarize the implementation of old coroutines:

• Event loop: Implemented by manually writing while loop code. Although this method is relatively basic, it gives developers a deeper understanding of the principle of the event loop.
• Code pausing and resuming: Achieved with the help of the characteristics of the yield generator. yield can not only pause the function execution but also save the function state, enabling the function to continue from where it was paused when resumed.

3. Review of New Coroutines

3.1 A Powerful System Based on the Event Loop

New coroutines are implemented based on keywords such as asyncio, async, and await, with the event loop mechanism at its core. This mechanism provides more powerful and efficient asynchronous programming capabilities, including event loops, task management, and callback mechanisms.

3.2 Analysis of the Functions of Key Components

• asyncio: Provides an event loop, which is the foundation for running new coroutines. The event loop is responsible for managing all asynchronous tasks, scheduling their execution, and ensuring that each task is executed at the appropriate time.
• async: Used to mark a function as a coroutine function. When a function is defined as async def, it becomes a coroutine, and the await keyword can be used to handle asynchronous operations.
• await: Provides the ability to suspend the process. In a coroutine function, when await is executed, the execution of the current coroutine pauses, waits for the completion of the asynchronous operation following await, and then resumes execution.

3.3 Detailed Explanation of Code Examples

import asyncio


async def coro1():
    print("start coro1")
    await asyncio.sleep(2)
    print("end coro1")


async def coro2():
    print("start coro2")
    await asyncio.sleep(1)
    print("end coro2")


# Create an event loop
loop = asyncio.get_event_loop()

# Create tasks
task1 = loop.create_task(coro1())
task2 = loop.create_task(coro2())

# Run coroutines
loop.run_until_complete(asyncio.gather(task1, task2))

# Close the event loop
loop.close()

In this example, two coroutine functions coro1 and coro2 are defined. After printing "start coro1", the coro1 function pauses for 2 seconds via await asyncio.sleep(2). Here, await suspends the execution of coro1 and returns the CPU to the event loop. The event loop will schedule other executable tasks, such as coro2, within these 2 seconds. After printing "start coro2", the coro2 function pauses for 1 second via await asyncio.sleep(1) and then prints "end coro2". When coro2 is paused, if there are no other executable tasks in the event loop, it will wait for the pause time of coro2 to end and continue to execute the remaining code of coro2. When both coro1 and coro2 are executed, the event loop ends.

3.4 Execution Results and Analysis

Results:

start coro1
start coro2
end coro2
end coro1

When coro1 executes to await asyncio.sleep(2), the process is suspended, and the CPU is returned to the event loop, waiting for the next scheduling of the event loop. At this time, the event loop schedules coro2 to continue execution.

To summarize the implementation of new coroutines:

• Event loop: Achieved through the loop provided by asyncio, which is more efficient and flexible and can manage a large number of asynchronous tasks.
• Program suspension: Achieved through the await keyword, making the asynchronous operations of coroutines more intuitive and easier to understand.

4. Comparison of Old and New Coroutine Implementations

4.1 Differences in Implementation Mechanisms

• yield: It is a keyword for generator (Generator) functions. When a function contains a yield statement, it returns a generator object. The generator object can be iterated step by step to obtain the values in the generator function by calling the next() method or using a for loop. Through yield, a function can be divided into multiple code blocks, and the execution can be switched between these blocks, thus achieving the pausing and resuming of function execution.
• asyncio: It is a standard library provided by Python for writing asynchronous code. It is based on the event loop (Event Loop) pattern, allowing multiple concurrent tasks to be processed in a single thread. asyncio uses the async and await keywords to define coroutine functions. In a coroutine function, the await keyword is used to pause the execution of the current coroutine, wait for the completion of an asynchronous operation, and then resume execution.

4.2 Summary of Differences

• Old coroutines: Mainly achieve coroutines through the ability of the yield keyword to pause and resume execution. Its advantage is that, for developers familiar with generators, it is easy to understand based on the generator syntax; its disadvantages are that it is easy to confuse with the generator concept, and the way of manually writing the event loop is not flexible and efficient enough.
• New coroutines: Achieve coroutines through the event loop mechanism combined with the ability of the await keyword to suspend the process. Its advantages are that it provides more powerful and flexible asynchronous programming capabilities, the code structure is clearer, and it better meets the needs of modern asynchronous programming; its disadvantage is that for beginners, the concepts of the event loop and asynchronous programming may be relatively abstract and require some time to understand and master.

5. The Relationship between await and yield

5.1 Similarities

• Control flow pause and resume: Both await and yield have the ability to pause the code execution at a certain point and continue it at a later time. This characteristic plays a crucial role in asynchronous programming and generator programming.
• Coroutine support: Both are closely related to coroutines (Coroutine). They can be used to define and manage coroutines, making the writing of asynchronous code simpler and more readable. Whether it is old or new coroutines, they rely on these two keywords to achieve the core functions of coroutines.

5.2 Differences

• Syntax differences: The await keyword was introduced in Python 3.5 and is specifically used to pause execution in an asynchronous function, waiting for the completion of an asynchronous operation. The yield keyword is for early coroutines and is mainly used in generator (Generator) functions to create iterators and implement lazy evaluation. Early coroutines were realized through the capabilities of generators.
• Semantics:
  • await means that the current coroutine needs to wait for the completion of an asynchronous operation and suspend execution, giving other tasks a chance to execute. It emphasizes waiting for the result of an asynchronous operation and is a waiting mechanism in asynchronous programming.
  • yield hands over the control of execution to the caller while saving the state of the function so that it can resume execution from the paused position in the next iteration. It focuses more on the control and state preservation of function execution.
  • await suspends the program and lets the event loop schedule new tasks; yield suspends the program and waits for the next instruction from the caller.
• Context: await must be used in an asynchronous context, such as in an asynchronous function or in an async with block. While yield can be used in an ordinary function, as long as the function is defined as a generator function, even without the context of using coroutines.
• Return values: yield returns a generator object, and the values in the generator can be iterated step by step by calling the next() method or using a for loop. The await keyword returns an awaitable object (Awaitable), which can be a Future, Task, Coroutine, etc., representing the result or state of an asynchronous operation.

5.3 Summary

await does not implement program pausing and execution through yield. Although they have similar capabilities, they have no calling relationship at all and are both Python keywords. await is suitable for asynchronous programming scenarios, used to wait for the completion of asynchronous operations, and supports more flexible coroutine management; while yield is mainly used in generator functions to implement iterators and lazy evaluation. There are some differences in their application scenarios and syntax, but both provide the ability to pause and resume the control flow.

By reviewing the development process of Python coroutines, comparing the implementation methods of old and new coroutines, and analyzing the relationship between await and yield in depth, we have a more comprehensive and in-depth understanding of the principles of Python coroutines. Although these contents are somewhat difficult to understand, mastering them will lay a solid foundation for our exploration in the field of Python asynchronous programming.

Leapcell: The Best Serverless Platform for Python app Hosting

Finally, let me introduce Leapcell, the most suitable platform for deploying Python applications.

1. Multi-Language Support

• Develop with JavaScript, Python, Go, or Rust.

2. Deploy unlimited projects for free

• pay only for usage — no requests, no charges.

3. Unbeatable Cost Efficiency

• Pay-as-you-go with no idle charges.
• Example: $25 supports 6.94M requests at a 60ms average response time.

4. Streamlined Developer Experience

• Intuitive UI for effortless setup.
• Fully automated CI/CD pipelines and GitOps integration.
• Real-time metrics and logging for actionable insights.

5. Effortless Scalability and High Performance

• Auto-scaling to handle high concurrency with ease.
• Zero operational overhead — just focus on building.

Explore more in the documentation!

Leapcell Twitter: https://x.com/LeapcellHQ