Local Development with Azure Service Bus

For teams new to Azure Service Bus, one of the first questions you have to answer is "how do I develop against this?" And it turns out the answer isn't that straightforward - because it's currently impossible to run Azure Service Bus outside of Azure. There's no install. There's no Docker image. There's no emulator. This is a bit alien to most developers, as typically, we can run the entire production workload locally. My team uses:

• Azure SQL
• Azure Functions
• ASP.NET Core (and .NET Core Workers)
• Node.js (SPA)
• Docker
• Kubernetes
• CosmosDB
• Azure Service Bus

Of all these, only the last one on the list can't be run external to Azure. This poses a challenge to the team - we want to ensure that environments are isolated from each other, and each developer's environment won't interfere with another's:

When developers are forced to use shared resources, we introduce contention. We can no longer reason about our code versus someone else's. This is especially problematic for stateful resources, where I have to worry about someone else changing the state I'm working with.

Back to Azure Service Bus, what we don't want to happen is have multiple developers all reading and writing from the same queues. I don't want one developer producing messages, and another developer consuming them, while the original developer is left waiting for those messages to arrive.

To develop effectively, we need figure out a strategy to ensure isolated developer environments. Unfortunately for us, there's not really any guidance on the Microsoft documents on how to do this. Compare to Azure Functions, that has a top-level documentation page on doing exactly this:

Or Azure Cosmos DB:

But there's nothing on the Azure Service Bus docs on local development. There's lots of guides on how to develop against Azure Service Bus, but nothing around strategies for isolating individual developers from each other.

I wound up creating a GitHub issue for this, just to understand what exactly the local development story is:

There is no documented story, so what are our options? As the top-level resource bucket in Azure Service Bus is a "namespace", we can:

• Isolate developers in a common namespace
• Create a namespace per developer
• Abstract our usage of Azure Service Bus and run something else locally

These each have their benefits and drawbacks, however. Namespaces aren't free, and each namespace tier has different features, so we have to consider costs and capabilities.

Isolate Developers in Common Namespace

In this approach, we pick an appropriate tier of Azure Service Bus (probably Standard or Premium), and create isolated resources for each individual developer (or machine). We have to use a naming convention to separate these resources, perhaps using the machine name or developer's login:

One advantage to this approach is we can use a higher Service Bus pricing tier (perhaps even Premium, around $700/mo), ensuring our developers use the exact same resource during development as production.

The downside is all this additional naming complexity and the pain of setting all this parallel infrastructure up. You'll have to roll your own convention and setup, and ensure your code can handle topics, subscriptions, and queues having somewhat dynamic names.

Namespace per developer

Instead of having a single namespace that everyone shares, we can instead create isolated namespaces per developer:

Then inside each namespace, the names of our individual resources (topic/subscription/queue) will be consistent across all environments:

This is a big benefit - we can ensure that our application code only really needs to change a connection string but the rest of the code deals with consistent resource names.

The downside is creating this service bus resource in the first place. When you create a Service Bus namespace, you have a few decisions:

• What Azure subscription to use
• Namespace name
• Pricing tier

Even the first question can be difficult. MSDN and Visual Studio subscriptions come with Azure credits. I've found that unless they need to, teams don't activate those subscription credits. You can use a corporate subscription as well, but typically the process for creating Azure resources isn't open to developers. If it is, it's usually in a sandbox/dev Azure Resource Group.

Next, you have the namespace name. This name isn't just unique to the subscription - it has to be unique across all Azure Service Bus namespaces. You'll need to come up with a naming convention that's distinct per developer/environment/company.

Finally, the pricing tier. Premium namespaces are not cheap - ~$700/mo. However, Standard namespaces aren't expensive - ~$10/mo. And that cost is shared across an entire subscription. For multiple developers on a single subscription, that $10/mo is a one-time charge.

Not insurmountable challenges, but again, we don't have to do this with a SQL database or CosmosDB database.

Transport abstraction

Finally, we can abstract our transport and swap our usage locally. We can do this at the protocol (AMQP) level, or at a higher-level abstraction using an OSS library like NServiceBus. With this, our application code can swap messaging transports to a local version when we're developing locally:

if (context.HostingEnvironment.IsDevelopment()) {
    endpoint.UseTransport<RabbitMQTransport>()
        .ConnectionString(context.Configuration["Messaging:ConnectionString"])
        .UseConventionalRoutingTopology()
        .EnableInstallers();
} else {
    endpoint.UseTransport<AzureServiceBusTransport>()
        .ConnectionString(context.Configuration["Messaging:ConnectionString"]);
}

Hopefully, the transports are close enough (or it's just pure AMQP protocol), and I'm not relying on too many transport-specific features in local development. Or, transport-specific limitations that you only encounter when running real-life workloads. But this can work, and we can even use containerized resources.

I asked the question on Twitter on these options, and got roughly an even split between these:

With a "single shared namespace" winning out (which I thought was the hardest one to do).

So what should we do? It really depends, since we don't have LocalStack available for us here. It would be super nice of course to have a local/emulator. And if you don't, I guess you can get into the sea: