Background work with Background Service

The problem: jobs with no request

Our shop has a few jobs that no user is waiting for:

1. Deleting expired carts. Once every few minutes.
2. Sending the order confirmation email. The user must not wait three seconds for the email server.
3. Sending Outbox messages. Every few seconds, send the saved messages to the message queue.

We do not put these jobs inside an endpoint. We build a service that starts with the app, works in the background, and shuts down with the app. We call it a Background Service.

The idea: a service with a start and an end

In .NET, the Host runs the app. Every service that implements the IHostedService interface has two methods:

• The StartAsync method: it is called when the app starts.
• The StopAsync method: it is called when the app shuts down.

The ready-made BackgroundService class makes this simpler. You only write one method: ExecuteAsync. This method gets a token called stoppingToken. When the app wants to shut down, this token is cancelled.

Code: deleting expired carts

public sealed class ExpiredCartCleaner(
    IServiceScopeFactory scopeFactory,
    ILogger<ExpiredCartCleaner> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromMinutes(5));

        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            try
            {
                // A new scope (and a new DbContext) for every round.
                await using var scope = scopeFactory.CreateAsyncScope();
                var db = scope.ServiceProvider.GetRequiredService<ShopDb>();

                var deleted = await db.Carts
                    .Where(c => c.ExpiresAt < DateTime.UtcNow)
                    .ExecuteDeleteAsync(stoppingToken);

                logger.LogInformation("Deleted {Count} expired carts", deleted);
            }
            catch (Exception ex) when (ex is not OperationCanceledException)
            {
                // One bad round must not kill the service. Try again next tick.
                logger.LogError(ex, "Cart cleanup failed");
            }
        }
    }
}

// Program.cs
builder.Services.AddHostedService<ExpiredCartCleaner>();

There are three points in this code. Each one has its own section below:

1. A new Scope for each round.
2. Errors are caught inside the loop.
3. The stoppingToken is passed everywhere.

Point one: a Scope for each round

A background service is built once and lives until the app ends. So it behaves like a Singleton. But DbContext is Scoped. If you take the DbContext straight from the constructor, in the Development environment the app gives an error at start. In Production this check is off by default, so you see no error, and the problem below happens. If you create one Scope at the start of the method and keep it until the end, you have a worse problem:

1. One DbContext stays alive forever.
2. Everything read with it stays in its Change Tracker. Even after saving.
3. Memory slowly grows. After a few days, the pod restarts with an out-of-memory error.
4. It also gets slower. Each save checks all those things again.

Point two: an error must not kill the service

If an error comes out of the ExecuteAsync method, by default the whole app stops (this has been the default since .NET 6). This is good, because the error does not stay hidden. But it means that a short database outage shuts down the whole service.

So:

1. Catch the error of each round inside the loop. Log it and try again in the next round.
2. Do not catch the cancel error. The OperationCanceledException error means the app is shutting down. Let the loop end.
3. If the error keeps repeating, create an alert. A background service that silently fails every time is worse than a service that is off.

In-memory queue: sending emails

For the order confirmation email, the endpoint must not wait for the email server. So the endpoint only puts the job in a queue and replies right away. A background service takes it from the queue and sends the email. The Channel class is made for this job.

public sealed class EmailQueue
{
    // Bounded: if the queue is full, writers wait instead of using all memory.
    private readonly Channel<OrderEmail> _channel = Channel.CreateBounded<OrderEmail>(1_000);

    public ValueTask EnqueueAsync(OrderEmail email, CancellationToken ct) =>
        _channel.Writer.WriteAsync(email, ct);

    public IAsyncEnumerable<OrderEmail> ReadAllAsync(CancellationToken ct) =>
        _channel.Reader.ReadAllAsync(ct);
}

public sealed class EmailSender(
    EmailQueue queue, IServiceScopeFactory scopeFactory, ILogger<EmailSender> logger)
    : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var email in queue.ReadAllAsync(stoppingToken))
        {
            try
            {
                await using var scope = scopeFactory.CreateAsyncScope();
                var client = scope.ServiceProvider.GetRequiredService<IEmailClient>();
                await client.SendAsync(email, stoppingToken);
            }
            catch (Exception ex) when (ex is not OperationCanceledException)
            {
                logger.LogError(ex, "Sending email for order {OrderId} failed", email.OrderId);
            }
        }
    }
}

// Program.cs
builder.Services.AddSingleton<EmailQueue>();
builder.Services.AddHostedService<EmailSender>();

Point three: safe shutdown

In Kubernetes, every deploy means the old pods shut down. The order is:

1. The SIGTERM signal arrives. The .NET host starts to shut down.
2. New requests are not accepted. Running requests get time to finish.
3. The stoppingToken is cancelled. The background service loop must see this and end.
4. The host waits up to a time limit. This limit is set with the ShutdownTimeout setting in HostOptions. Its default value is 30 seconds.
5. If Kubernetes does not wait longer, it sends SIGKILL. The Kubernetes grace period is also 30 seconds by default. After that, the process is killed at once.

builder.Services.Configure<HostOptions>(options =>
{
    // Must be shorter than the Kubernetes grace period.
    options.ShutdownTimeout = TimeSpan.FromSeconds(20);
});

Even with all of this, a pod sometimes dies suddenly. So every background job must be Idempotent. This means that if it runs twice, the result is not broken.

Several pods, several copies of the same job

A financial report is built every night at 2 AM and emailed to customers. To handle the load, we raise the number of pods from 1 to 3. Now what happens?

1. Each pod is a full copy of the app. So each pod has its own background service.
2. All three wake up at 2 AM. None of them knows about the others.
3. Each customer gets three identical emails.

The options, from simple to complex:

1. Move the job out of the service. A CronJob in Kubernetes that runs a short-lived pod every night. Now there are no multiple copies.
2. Record the run in the database with a Unique Constraint. Before it starts, each pod adds a “today’s report” row. Only the first write succeeds.
3. A distributed lock. With a database lock or Redis. Or a ready-made tool like Hangfire or Quartz.NET in cluster mode.

Important rules

1. Create a new Scope for each unit of work. Use the CreateAsyncScope method on IServiceScopeFactory.
2. Pass the stoppingToken everywhere. To the database, HttpClient and delays.
3. Catch and log the error of each round. But do not catch the cancel error.
4. Set the shutdown limit lower than the Kubernetes grace period.
5. Write the job to be Idempotent. A restart and a second run are always possible.
6. Think about the number of pods. A scheduled job on several pods runs several times.

Common mistakes

When to use a Background Service?