Azure Durable Functions Best Practices for Production
In Part 1, I explained the Durable Functions execution model. In Part 2, I explored the patterns available for shaping a workflow.
Choosing the correct pattern is important, but it does not automatically make the workflow production-ready. The difficult lessons usually appear around payload size, duplicate delivery, retries, history growth, deployments, observability, and recovery.
This final part brings those lessons together as the checklist I would use when designing or reviewing a Durable Functions application.
Navigate the Durable Functions Series
-
Part 1: Understanding Basics
-
Part 2: Azure Durable Functions Design Patterns
- Part 3: Azure Durable Functions Best Practices [👈 You are here]
Keep Payloads Small and Non-Sensitive
Durable Functions persists much more than the initial orchestration input. Inputs, outputs, external event payloads, custom status, entity state, durable HTTP data, and unhandled exception details can all end up in the configured backend.
That has three direct consequences:
- Large payloads increase storage transactions, memory use, and replay cost.
- Persisted data can become stale while a long-running workflow waits.
- Secrets and personally identifiable information gain a much wider exposure surface.
The default pattern should be:
Store the data in the system that owns it, pass a small reference through the orchestration, and materialise the data inside an activity only when it is needed.
Passing a customer ID is usually better than passing a customer object. Passing a blob reference is usually better than passing the entire document. This also makes retries safer because an activity can check the current source-of-truth state before acting.
Some providers can offload large payloads, but that is a useful escape hatch, not permission to turn orchestration history into a document store.
Treat the Task Hub as a security boundary. Prefer managed identity over storage keys, apply least-privilege RBAC, restrict network access, and tightly control write access. The framework trusts the state it reads from the backend; write access is far more powerful than it first appears.
Let Failures Be Visible
The default failure behaviour is normally the right one. An unhandled activity exception fails the activity call. If the orchestrator does not deliberately handle it, the orchestration becomes failed and can be found by monitoring.
The dangerous pattern is catching an exception, logging it, returning a default value, and allowing the workflow to report success.
Only catch an error when the workflow has a decision to make:
- Retry a transient failure.
- Branch on a known business outcome.
- Run compensating activities.
- Set useful status before failing.
- Route the case for manual intervention.
Retries should be reserved for failures that can reasonably succeed without changing the request: throttling, temporary network problems, timeouts, or short-lived service unavailability. Validation failures, rejected business rules, and malformed data should normally fail fast.
Configure retry policies at the call site because different dependencies have different limits. A payment provider and an internal read-only API should not automatically share the same retry count and backoff.
Also remember that exception details may be persisted. Avoid putting secrets or full payloads in exception messages, just as you would avoid putting them in logs.
Scaling Is More Than Increasing Concurrency
There are several different limits involved in a Durable Functions workload:
- How many work items a worker can process concurrently.
- How many workers the hosting plan can add.
- How the chosen storage provider partitions and delivers work.
- How much concurrency the language runtime can actually execute.
- How much load downstream systems can safely accept.
- How large each orchestration history becomes.
Turning up maxConcurrentActivityFunctions is not automatically a performance improvement. If activities spend most of their time throttled by a database or API, more concurrency can make throughput and reliability worse.
Measure end-to-end latency, queue backlog, activity duration, retry rates, dependency throttling, replay time, and memory use. Tune concurrency from evidence, not from the number of items waiting.
For the Azure Storage provider, partition count affects orchestration scale-out and cannot simply be changed after the Task Hub has been created. That makes capacity testing an architecture activity, not a last-minute production setting change. If Azure Storage does not meet the tested throughput requirements, evaluate the Durable Task Scheduler rather than endlessly tuning queue settings.
Activities can perform I/O, CPU work, and multithreaded work, but they are still Azure Functions and remain subject to the hosting plan and function timeout rules. A durable orchestration can live for months; one activity invocation cannot run forever.
Control History Growth
History is what makes replay possible, but unbounded history eventually becomes a performance problem.
Common causes include:
- Huge fan-outs from one parent orchestration.
- Large inputs and outputs repeated across many calls.
- Polling loops that never finish.
- Recurring workflows that never reset.
- Excessive use of custom status or very chatty orchestration designs.
Use ContinueAsNew for truly eternal workflows so each cycle begins with a fresh history. Use sub-orchestrations to divide a large workflow into sensible phases. Batch large collections and keep messages small.
Completed, failed, and terminated instance data also needs an explicit retention policy. With bring-your-own storage providers, terminal histories remain until the application or an operator purges them. Schedule purging through the supported instance-management APIs based on business, support, privacy, and audit requirements. The Durable Task Scheduler supports configurable automatic purge retention, but retention still needs to be chosen deliberately.
Treat Deployment as Part of Orchestration Design
Long-running workflows and normal deployment habits do not always mix well.
Before changing an orchestrator, ask whether any in-flight instance could replay through the changed code. Treat the following as potentially breaking:
- Adding, removing, or reordering durable calls.
- Renaming an orchestrator, activity, entity, or sub-orchestration.
- Changing an input or output type already stored in history.
- Changing branching logic that affects which durable operation comes next.
- Changing serialisation behaviour during an isolated worker migration.
For supported SDK versions, built-in orchestration versioning is now the recommended approach for most breaking changes. It associates an instance with a version and allows newer workers to preserve old code paths while new instances use the new path.
The important word there is preserve. Once a version is live, its orchestrator path should remain deterministic. Do not quietly edit the old branch because the new branch exists.
Where built-in versioning is not suitable, side-by-side applications with separate Task Hubs remain an option. A deployment slot by itself is not a complete versioning strategy; the state and in-flight instances still need a deliberate plan.
Every function app or deployment slot sharing a backend should have its own Task Hub unless it is an intentional multi-region copy of the same application for disaster recovery. Two unrelated apps competing for the same Task Hub can process each other’s messages and produce undefined behaviour.
If You Are Migrating from the In-Process Model
Support for the .NET in-process model ends on 10 November 2026. Migrating to the isolated worker is more than replacing package names until the application compiles.
Pay particular attention to persisted inputs and outputs, System.Text.Json behaviour, retries, entities, ContinueAsNew, custom status, and management code. Test what happens to orchestration instances that are already running when the new application is deployed, and choose an orchestration versioning or side-by-side deployment strategy before switching production traffic.
The backend options have also evolved. Azure Storage remains the mature default provider, while Microsoft now recommends the managed Durable Task Scheduler for new Durable Functions workloads. Existing orchestration data cannot currently be migrated from one storage provider to another, so a backend change needs its own state and cutover plan.
Make Operations a First-Class Feature
If the only way to understand a stuck workflow is to open a storage account and inspect internal tables, the application is not operationally ready.
At a minimum:
- Use replay-safe logging inside orchestrators.
- Include the orchestration instance ID, function name, version, and business correlation ID in structured logs.
- Keep PII and secrets out of instance IDs, logs, custom status, inputs, outputs, and exceptions.
- Enable Application Insights tracking and distributed tracing where appropriate.
- Monitor failed, terminated, pending, and unusually long-running instances.
- Alert on queue backlog, dependency throttling, repeated retries, and age of the oldest work item.
- Expose a safe operational path to query, suspend, resume, terminate, and purge instances.
- Define what support teams can retry automatically and what needs business approval.
Replay-safe logging prevents duplicate application log lines during orchestration replay. It does not remove the need for correlation, meaningful event names, or sensible telemetry volume.
Custom status is useful for showing a small piece of progress to a caller, but it is not a second logging system and should never contain a full domain object.
Test the Failure Paths, Not Just the Happy Path
Durable Functions often looks perfect in a local happy-path demo. The interesting behaviour begins when a host stops halfway through the workflow.
I would include these cases in the test strategy:
- The same activity input is delivered twice.
- A transient dependency fails until the final retry.
- A terminal business error fails immediately.
- The host restarts after an external side effect but before activity completion is recorded.
- An expected external event never arrives.
- An event arrives twice, late, or with the wrong correlation.
- A compensation activity also fails.
- A large fan-out hits the real downstream rate limit.
- A new deployment arrives while old orchestration instances are still running.
- Terminal history is purged according to the retention policy.
Unit-test business logic in the application service layer. Test orchestrator decisions with deterministic inputs. Then run integration tests against the actual Durable Functions runtime and a representative backend, because replay, delivery, serialisation, and scale behaviour cannot all be proven with mocks.
The Code Review Checklist
With the runtime rules, design patterns, and production concerns covered, this is the checklist I would use if I had only a few minutes to review a Durable Functions design.
| Do | Do not |
|---|---|
| Keep the orchestrator deterministic and focused on coordination | Perform I/O, read changing configuration, or use normal delays inside it |
| Make every activity side effect idempotent | Assume an activity will execute exactly once |
| Pass IDs and small reference DTOs | Pass large or sensitive domain objects through history |
| Use durable timers and external events | Block a thread or poll without an upper bound |
| Retry transient failures with dependency-specific policies | Retry validation and business-rule failures blindly |
| Let failures surface and monitor them | Catch exceptions and return a false success |
| Batch fan-out and respect downstream capacity | Start unbounded parallel work from one orchestration |
Use sub-orchestrations and ContinueAsNew to control history |
Let one orchestration history grow forever |
| Use replay-safe, structured, correlated logging | Log full payloads, credentials, or PII |
| Plan orchestration versioning before deployment | Change live orchestrator paths and hope running instances cope |
| Give each app and slot the correct Task Hub isolation | Point unrelated applications at the same Task Hub |
| Secure the backend with identity, least privilege, and network controls | Treat the state store as harmless implementation detail |
| Define retention and purge terminal histories | Keep every completed instance forever by accident |
| Load-test the complete workflow and its dependencies | Tune concurrency from guesswork |
| Test duplicates, restarts, timeouts, and compensation | Test only the happy path |
The Five Takeaways
If the full series is too much to remember, remember these five:
Orchestrators coordinate; activities work.
Orchestrators replay; activities may be delivered more than once.
Pass references, not large or sensitive objects.
A deployment can break an orchestration that started days ago.
If we cannot observe and safely recover the workflow, it is not finished.
Final Thoughts
Durable Functions gives us excellent patterns for expressing long-running workflows, but choosing a pattern is only the beginning. The best practices are what make that workflow safe to replay, retry, scale, deploy, observe, and recover.
Reliable designs still begin with the same questions. Where does the state live? What gets replayed? What can run twice? How does the workflow recover? What happens to an in-flight instance during deployment? How will the team support it at 2 a.m.?
This three-part series has focused on the mental model and design decisions. I am planning a follow-up post with current .NET isolated worker code examples covering the patterns and practices discussed here. Stay tuned for that.
In the meantime, what have I missed? If your team has a Durable Functions rule, production lesson, or edge case that deserves to be in this guide, please leave a comment or reach out. I would love to add the things that only show up after running these workflows in the real world.
Leave a comment