Worker Pool
A concurrency pattern that limits parallelism by distributing work across a fixed number of goroutines.
Contributors
When to Use
- When processing many tasks that can run concurrently
- When you need to limit resource usage (connections, memory, CPU)
- When tasks are I/O bound and benefit from parallelism
When NOT to Use
- For a small number of tasks where goroutine overhead matters
- When tasks must be processed sequentially
- When each task requires exclusive access to a shared resource
The Worker Pool pattern manages a fixed number of goroutines that process tasks from a shared channel. This limits concurrency to prevent resource exhaustion while still enabling parallel processing.
Implementation
package pool
import (
"context"
"sync"
)
type Task func() error
type Pool struct {
workers int
tasks chan Task
wg sync.WaitGroup
}
func New(workers int) *Pool {
return &Pool{
workers: workers,
tasks: make(chan Task),
}
}
func (p *Pool) Start(ctx context.Context) {
for i := 0; i < p.workers; i++ {
p.wg.Add(1)
go p.worker(ctx)
}
}
func (p *Pool) worker(ctx context.Context) {
defer p.wg.Done()
for {
select {
case task, ok := <-p.tasks:
if !ok {
return
}
task()
case <-ctx.Done():
return
}
}
}
func (p *Pool) Submit(task Task) {
p.tasks <- task
}
func (p *Pool) Stop() {
close(p.tasks)
p.wg.Wait()
}
Usage
package main
import (
"context"
"fmt"
"time"
)
func main() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Create pool with 3 workers
pool := New(3)
pool.Start(ctx)
// Submit tasks
for i := 0; i < 10; i++ {
id := i
pool.Submit(func() error {
fmt.Printf("Processing task %d\n", id)
time.Sleep(100 * time.Millisecond)
return nil
})
}
// Wait for completion
pool.Stop()
}
Benefits
- Resource control: Limits concurrent operations to prevent exhaustion
- Backpressure: Producers block when all workers are busy
- Graceful shutdown: Context cancellation stops workers cleanly
- Reusable workers: Goroutines are recycled, reducing allocation overhead
- Simple coordination: Uses channels for natural synchronization