Joseph Masengesho · Follow
Published in · 9 min read · Jul 18, 2020
In Post 1, we looked at Event-Driven Architecture (EDA) and its benefits. We were introduced to message brokers in general. In this post, we will focus on four main message brokers in Azure: Azure Service Bus, Azure Event Hubs, Azure Event Grid, and Azure Queue Storage.
Azure Service Bus, an Azure PaaS offering, is a fully managed enterprise integration message broker. It is used to decouple applications of any size hosted natively on Azure, on-premise, or from any other cloud vendor such as AWS or GCP. Messages are sent and kept into queues or topics until requested by consumers in a “poll” mode (i.e. only delivered when requested).
Concepts and Features
- Queues
In a simple implementation, message producers send messages to a queue. Messages are kept in the queue until picked up by consumers. The retrieval is on a first-in-first-out (FIFO) basis. A queue can have one or many competing consumers, but a message is consumed only once.
- Topics, Subscriptions, Rules, and Filters
In a sophisticated scenario, you might want to send a message to predefined consumers based on some conditions (message attributes). In this case, all messages are sent to a topic. A topic enables you to create multiple subscriptions. Subscriptions receive messages based on defined Rules and Filters.
- Namespace
A namespace is a container of all messaging components. It acts as an application container. Queues, topics, subscriptions, rules, and filters are scoped in a container. Creating a Service Bus automatically creates a namespace with the name you gave to the resource.
Other advanced Features
- Dead Letter Queue: Azure Service Bus creates DLQ sub-queue to hold messages that could not be delivered or processed.
- Consumption Mode: Azure Service Bus supports several consumption modes: pub/sub with a pull model, competing consumers, and partitioning can be achieved with the use of topics, subscriptions, and actions.
- Duplicate Detection: Azure Service Bus is the only message broker that supports duplicate detection natively.
- Delivery Guarantee: Azure Service Bus supports three delivery guarantees: At-least-once, At-most-once, and Effectively once.
- Message Ordering: Azure Service Bus can guarantee first-in-first-out using sessions.
Azure Event Hubs is a big-data streaming platform and ingestion service for applications that produce a lot of events. It can receive and process millions of records per second. Upon receipt, data can be transformed and stored using a real-time analytics provider or batching/storage adapters. Azure Event Hubs represent the “front door” for an event pipeline. Like Azure Service Bus, Event Hubs sit between event producers and event consumers. It decouples the process of producing data from the process of consuming data. You can publish events individually or in batches.
Concepts and Features
Let’s say we are monitoring electricity usage of an electric grid with millions of connected devices that emit a consumption reading every minute. Can’t we use a Service Bus and set up a queue and voila? Remember the architecture of a Service Bus: consumers pull data for processing on a competing-consumer pattern. For data streaming, there is a massive influx of events every second. We need to “read it” and make sense of it. There is no expectation about who the consumers are. Consumers who are interested in incoming data can start reading from the time they join. One consumer group may need to perform aggregations on our incoming data while another wants to perform anomaly detection. Partitions and consumer groups are illustrated in the next figure.
- Partitions and Consumer Groups
Azure Event Hubs are built around the concept of partitions and consumer groups. Inside an Event Hub, events are sent to partitions by specifying the partition key or partition id. Partition count of an Event Hub cannot be changed after creation so is mindful of this limitation.
Receivers are grouped into consumer groups. A consumer group represents a view (state, position, or offset) of an entire event hub. It can be thought of as a set of parallel applications that consume events at the same time.
Consumer groups enable receivers to each have a separate view of the event stream. They read the stream independently at their own pace and with their own offsets. Event Hub uses a partitioned consumer pattern; events are spread across partitions to allow horizontal scale.
- Throughput Units
The event hub capacity is measured by the throughput units. This capacity is simply a measure of how many events or memory size an event hub can take in (ingress) or send (egress). A single throughput unit has:
- Ingress: Up to 1 MB per second or 1000 events per second (whichever comes first).
- Egress: Up to 2 MB per second or 4096 events per second.
Event Grid is a serverless message broker that enables native reactive interactions between Azure resources. You can also connect your custom events from outside of Azure.
Azure Event Grid allows you to easily build applications with event-based architectures. First, select the Azure resource you would like to subscribe to and then specify the event handler or webhook endpoint to send the event to. Event Grid has built-in support for events coming from Azure services like Blob Storage, Event Hubs, IoT Hubs, and more. Event Grid also has support for your own events using custom topics. You can use filters to route specific events to different endpoints, multicast to multiple endpoints, and make sure your events are reliably delivered.
As shown in the figure below, Event Grid conceptual architecture is based on Event Sources and Event Handlers. The list of services in the figure below is not exhaustive. Find a list of all supported event sources here and a list of all supported event handlers here.
Concepts and Features
- Event subscriptions
The endpoint or built-in mechanism to route events, sometimes to more than one handler. Subscriptions are also used by handlers to intelligently filter incoming events.
- Event handlers
The application or service reacting to the event. Any services in Azure that can subscribe to Event Grid or webhooks. They are also called event endpoints.
Other Event Grid Considerations
- Durability: Event Grid provides durable delivery. It tries to deliver each message at least once for each matching subscription immediately. If a subscriber’s endpoint doesn’t acknowledge receipt of an event or if there’s a failure, Event Grid retries delivery based on a fixed retry schedule and retry policy. By default, the Event Grid module delivers one event at a time to the subscriber. The payload is however an array with a single event.
- Consumption Mode: Event Grid supports three consumption modes: pub/sub with a push model, concurrent consumers, and partitioning can be achieved with event filtering.
- High Throughput: Event Grid can take up to 10 million events per second. Always, the first 10 thousand events are free every month.
- Event Schema: Events consist of a set of five required string properties and a required data object: topic, subject, id, event type, and data. The properties are common to all events from any publisher. The data object has properties that are specific to each publisher. For system topics, these properties are specific to the resource provider, such as Azure Storage or Azure Event Hubs (Microsoft).
Azure Queue Storage is different from the three previous message brokers because it is part of the Azure Storage Service. It carries Azure Storage core concepts and features around security, concurrency, pricing tiers, and storage size limitations. The Azure Queue Storage service supports HTTP and HTTPS only. One of the most compelling benefits of a queue storage service is the effectively large storage account capacity. You can have one giant queue or many smaller queues. If your scenario involves a large number of messages such as a backlog of work or snapshot state data, queue storage is probably your best option.
Concepts and Features
- Storage Account
Azure Storage solution offers four services: Blob for binary data, Files for managed file systems, Queues for messages, and Table for NoSQL databases. In our article, we will not discuss storage solution concepts; we will focus on why you would need to use Queue Storage as a messaging solution and how it compares to other offerings discussed in this article.
- Queues
The Queue Storage service implements a queue that stores messages for up to seven days unless otherwise specified. Messages are added and dequeued similar to other queues. The Queue Storage service presents three important features:
- Message Editing: As opposed to other message broker services, the queue service offers the ability to edit a message and keep it in the queue. This makes Queue Storage a perfect candidate for scenarios where we need to manipulate messages before we delete them.
- Message Deletion: Dequeuing a message from queue storage does not delete it. After the message has been processed by the consumer successfully the message needs to be deleted, otherwise, it will be deleted when it expires. This makes queue storage efficient at handling errors. If a process fails before the message is deleted, it will be picked up by other consumers.
- Security: Azure Storage encrypts your data at rest by default and you cannot disable it. Like other storage services, you can rely on Microsoft-managed keys or bring your own keys for data encryption. You can authorize access to queue storage using a Shared Key (storage account key), Shared Access Signature (SAS), and Azure Active Directory (Azure AD). As you would guess, anonymous public read access is not supported.
Other Event Grid Considerations
- Consumption Mode: Queue Storage supports two consumption modes: pub/sub with a pull mode and competing consumers.
- Big Queues: Queue storage queues can have up to 80 GB in size.
In this section, we reviewed architecture concepts and features for each of the four Azure messaging solutions. Let’s conclude by recapping at their practical use cases and specializations.
Some resources stand out for specific use cases:
- Event Hubs stands out for event streaming and high-volume real-time scenarios. It is very well suited for device telemetry streaming, Internet of Things (IoT), logging for high traffic applications such as social media and e-commerce, and real-time analytics pipelines.
- Azure Service Bus stands out for high-reliability scenarios because of its advanced features like duplication detection, dead-letter queues, topics, and sessions. It is suited for financial transaction messages, e-commerce, logistics, insurance, and healthcare.
- Azure Event Grid is the catalyst for reactive programming within Azure. It extends Azure serverless computing by triggering Azure Functions, Logic Apps, and Webhooks from any resource that supports Event Grid topics. It is used in building serverless applications and operations automation in Azure.
- Azure Queue Storage is the most basic messaging solution in Azure, but it offers its set of benefits. Queue Storage is specialized for scenarios where messages can be updated before being de-queued. This is a compelling feature for workflow-based applications. Queue Storage also offers the biggest queue among other queue-based solutions.
In this post, we reviewed four main message brokers in Azure. For each, we examined architecture concepts and practical use cases. The next post combines the two previous posts in order to answer the original question: When to use what service broker and why? We used 12 common factors to compare the four message brokers and we conclude by providing a decision tree that can assist in the decision-making process.
Continue to read the series:
Post 1: Event-Driven Architecture: Review Event-Driven Architecture (EDA) and Message Brokers concepts and benefits in a distributed environment.
Post 2: Azure Messaging Solutions: Review of each service’s architecture concepts and where it fits in the Azure cloud. We explore the types of problems each service is designed to solve.
Post 3: Choosing Messaging Solution (Coming soon): Analysis of 12 considerations that could impact your decision when choosing a cloud messaging service including variations, examples, and recommended services.
