Managing Concurrent Tasks in Go with `errgroup`

Key Takeaways

1. errgroup simplifies concurrent goroutine management with error handling and context cancellation.
2. Using WithContext allows early termination of all goroutines upon failure.
3. SetLimit controls the number of active concurrent goroutines to optimize resource usage.

The errgroup package in Go provides synchronization, error propagation, and context cancellation for groups of goroutines working on subtasks of a common task. It is part of the golang.org/x/sync module and offers a convenient way to manage concurrent operations that may return errors.

Overview of errgroup

The errgroup package is designed to simplify the management of multiple goroutines performing related tasks. It extends the functionality of sync.WaitGroup by adding error handling and context cancellation features. This means that if any goroutine encounters an error, the entire group can be canceled, and the error can be propagated back to the caller.

Key Components

The primary components of the errgroup package include:

• Group: A collection of goroutines working on subtasks of the same overall task.
• WithContext: Creates a new Group and an associated Context derived from the provided context.
• Go: Registers a function to be run in a new goroutine.
• Wait: Waits for all registered functions to complete and returns the first non-nil error (if any).

Basic Usage

Here's an example of how to use errgroup to fetch multiple URLs concurrently:

package main

import (
        "fmt"
        "net/http"
        "golang.org/x/sync/errgroup"
)

func main() {
        var g errgroup.Group
        var urls = []string{
                "http://www.golang.org/",
                "http://www.google.com/",
                "http://www.somestupidname.com/",
        }

        for _, url := range urls {
                url := url // create a new variable to avoid closure issues
                g.Go(func() error {
                        resp, err := http.Get(url)
                        if err == nil {
                                resp.Body.Close()
                        }
                        return err
                })
        }

        if err := g.Wait(); err == nil {
                fmt.Println("Successfully fetched all URLs.")
        } else {
                fmt.Printf("Failed to fetch URLs: %v\n", err)
        }
}

In this example, an errgroup.Group is created, and for each URL, a new goroutine is launched to perform an HTTP GET request. The g.Go method registers the function to be executed concurrently. The g.Wait method waits for all goroutines to complete and returns the first encountered error, if any.

Using errgroup with Context

The WithContext function allows for context cancellation, enabling all goroutines to be canceled if any of them encounters an error. This is particularly useful when the tasks are interdependent or when early termination is desired upon failure.

package main

import (
        "context"
        "fmt"
        "golang.org/x/sync/errgroup"
        "time"
)

func main() {
        ctx, cancel := context.WithCancel(context.Background())
        defer cancel()
        g, ctx := errgroup.WithContext(ctx)

        for i := 0; i < 3; i++ {
                i := i // avoid closure capture
                g.Go(func() error {
                        if i == 1 {
                                return fmt.Errorf("error in goroutine %d", i)
                        }
                        select {
                        case <-time.After(2 * time.Second):
                                fmt.Printf("goroutine %d completed\n", i)
                        case <-ctx.Done():
                                fmt.Printf("goroutine %d canceled\n", i)
                        }
                        return nil
                })
        }

        if err := g.Wait(); err != nil {
                fmt.Printf("Group finished with error: %v\n", err)
        } else {
                fmt.Println("Group finished successfully")
        }
}

In this code, if any goroutine returns an error, the context is canceled, causing other goroutines to terminate early if they respect the context's cancellation.

Controlling Concurrency with SetLimit

The SetLimit method allows you to control the maximum number of active goroutines at any given time. This is useful for limiting resource usage when dealing with a large number of tasks.

package main

import (
        "fmt"
        "golang.org/x/sync/errgroup"
)

func main() {
        var g errgroup.Group
        g.SetLimit(2) // limit to 2 concurrent goroutines

        for i := 0; i < 5; i++ {
                i := i // avoid closure capture
                g.Go(func() error {
                        fmt.Printf("Starting task %d\n", i)
                        // simulate work
                        return nil
                })
        }

        if err := g.Wait(); err != nil {
                fmt.Printf("Group finished with error: %v\n", err)
        } else {
                fmt.Println("Group finished successfully")
        }
}

Here, SetLimit(2) ensures that no more than two goroutines are active simultaneously. New goroutines will wait until an active one completes before starting.

Common Pitfalls

When using errgroup, be mindful of the following:

• Variable Capture in Closures: Ensure that loop variables are correctly captured in closures to avoid unexpected behavior. This is typically done by creating a new variable within the loop.

• Context Cancellation: When using WithContext, goroutines should respect the context's cancellation by checking ctx.Done() to terminate promptly when canceled.

• Error Handling: errgroup captures only the first non-nil error. If you need to collect all errors, you'll need to implement additional error aggregation logic.

By leveraging the errgroup package, Go developers can manage complex concurrent workflows more effectively, with built-in error handling and context-aware cancellation.

FAQs

How does errgroup differ from sync.WaitGroup?

errgroup propagates errors and supports context cancellation, while sync.WaitGroup only waits for goroutines to complete.

What happens when one goroutine fails in errgroup?

The first error is returned, and if using WithContext, other goroutines receive a cancellation signal.

How can I limit the number of active goroutines with errgroup?

Use SetLimit(n) to ensure that only n goroutines run concurrently.

---

We are Leapcell, your top choice for hosting Go projects.

Leapcell is the Next-Gen Serverless Platform for Web Hosting, Async Tasks, and Redis:

Multi-Language Support

• Develop with Node.js, Python, Go, or Rust.

Deploy unlimited projects for free

• pay only for usage — no requests, no charges.

Unbeatable Cost Efficiency

• Pay-as-you-go with no idle charges.
• Example: $25 supports 6.94M requests at a 60ms average response time.

Streamlined Developer Experience

• Intuitive UI for effortless setup.
• Fully automated CI/CD pipelines and GitOps integration.
• Real-time metrics and logging for actionable insights.

Effortless Scalability and High Performance

• Auto-scaling to handle high concurrency with ease.
• Zero operational overhead — just focus on building.

Explore more in the Documentation!

Follow us on X: @LeapcellHQ