Reliable Messaging Based on ESB – How to Approach It?

Messaging is now the most popular style of integrating applications in companies. Based on a service bus, this solution handles messaging applications divided into three groups: Senders, recipients, and a message broker. The latter’s role is often to ensure the reliability of the process, and guarantee that the message is delivered. This article will tell you how to organize the processing of messages on an ESB so that it could operate like a trusted, reliable courier company.


A few words about integration

The integration of applications can be approached in several ways. And each comes with its own problems. Basically, there are four styles of integration: 

  • file transfer 
  • database sharing by applications 
  • remote procedure call 
  • messaging 

This article will look at one of the problems connected with sending messages using the service bus, namely processing reliability. Before presenting the system architecture that will allow reliability to be achieved, we first need to determine the situations in which message processing might fail. What can go wrong when delivering a message from the sender to the recipient? 

The first of these situations will be the temporary unavailability of the target system, e.g. due to a scheduled service window. Another one is performance problems that prevent the recipient from successfully processing the message. At times, bugs might occur in the software itself where the correct message causes processing problems on the recipient’s side. Errors may also be identified in the service bus code itself.

Will the sender have to resend the message in each of these cases? With thousands of messages sent, how do you find those that have failed or caused an error? Fortunately, by using the right tools and designing well-thought-out system architecture, you can guarantee delivery of the message from the sender to the recipient. 

The work of a reliable courier 

The reader is likely to have some associations at this point. When talking about senders, recipients and a broker, a courier company will quickly come to your mind. So let’s briefly think how a reliable courier gets a package delivered. 

First, note one important fact here. Having handed over the package, the sender does not wait idly for the delivery man to return with a signed acknowledgment of receipt. They just return to their normal duties. 

Secondly, when picking up the package, the courier does not always deliver it immediately to the recipient. More often than not, the first place where the package goes is the local courier office and the sorting office. Only in the next step, often after being taken to another office in another city, will the package make its way to the recipient. 

Further, the supplier may face one of two situations – either the recipient accepts the package by signing the confirmation, or the courier does not find the recipient. From the point of view of the further delivery process, the latter situation, where a repeated delivery attempt is made, seems more interesting. Each company has its own rules specifying how many times the courier will try to deliver the package, and the number of days after which the package returns to the sender.

Service bus as a message courier 

When comparing package delivery by courier to messaging, note one important thing. The service bus itself does not allow received messages to be stored indefinitely. To arrange reliable processing, the ESB must use additional tools that allow the received messages to be retained. When messages are stored in memory only, in the event of a bus failure, data may be lost and it will be necessary to resend them. 

One of the tools to tackle this problem is message queues. And the tool ensuring the implementation of efficient queuing is the RabbitMQ message broker.  

Bearing in mind the previously introduced context – package delivery by courier –I’ll tell you how to organize messaging based on Mule ESB and RabbitMQ later in this article.

Asynchronous communication using queues in RabbitMQ 

Let’s return to the first observation regarding the moment when the package is handed over to the courier. The sender of the package does not just wait for the courier to deliver the package to the recipient. In the context of systems integration, this situation is obviously reflected in the use of asynchronous communication. After receiving the message from the sender, the ESB forwards the message to the queue. Then, when the message is already retained in RabbitMQ, the bus acknowledges receipt of the message from the sender.  What happens when we turn off the RabbitMQ server at this point? Let’s talk about guaranteed delivery.

Reliability pattern and guaranteed delivery 

In one of the articles on the DZone portal, the author presents a pattern of reliable processing based on the Mule ESBand the VM queue native to this bus. The point of this solution is to configure the queue in such a way that messages are not stored in memory only, but are retained in one of the so-called object stores provided by Mule ESB. Note, however, that persisting messages on the same virtual machine on which the bus sits is not always the best solution. In the case of physical architecture designed and implemented by Unity Group, individual system components are often installed on several machines, which minimizes the impact of individual system components on each other. Now, you might ask if by using RabbitMQ installed on a dedicated virtual machine, you can also use persistent queues. The answer is – of course you can. Messages are stored in the Mnesia database supplied and installed with this component. 

We already know how to persist messages in RabbitMQ queues. So let’s move on to the next step in the process – delivering the message to the recipient. 

Reliability pattern and guaranteed delivery 

In one of the articles on the DZone portal, the author presents a pattern of reliable processing based on the Mule ESBand the VM queue native to this bus. The point of this solution is to configure the queue in such a way that messages are not stored in memory only, but are retained in one of the so-called object stores provided by Mule ESB. Note, however, that persisting messages on the same virtual machine on which the bus sits is not always the best solution. In the case of physical architecture designed and implemented by Unity Group, individual system components are often installed on several machines, which minimizes the impact of individual system components on each other. Now, you might ask if by using RabbitMQ installed on a dedicated virtual machine, you can also use persistent queues. The answer is – of course, you can. Messages are stored in the Mnesia database supplied and installed with this component. 

We already know how to persist messages in RabbitMQ queues. So let’s move on to the next step in the process – delivering the message to the recipient. 

The producer-consumer pattern and the Dead Letter Queue 

The communication between ESB and RabbitMQ is based on the producer-consumer pattern. In a situation where there are messages queued, the service bus, as a consumer, will retrieve the messages and send them to the recipient. In the example quoted, I mentioned that from the point of view of the delivery process, delivery failure is the most interesting case. For this reason, let’s consider the case where the ESB retrieved a message from the queue and the recipient was unavailable. In this situation, the first thing to do is to handle the already retrieved message in such a way that it is not lost. The acknowledgment mechanism in RabbitMQ comes in handy here. Once a message is retrieved from the queue, there are only two options available: accept or reject the message. So, after being successfully sent to the recipient, the retrieved message must be acknowledged, which will remove it from the queue system. If the processing fails (which may happen for various reasons), the message should be rejected.

Rejecting a message may lead to two situations. In the first case, the message will go back to the original queue and then it will be immediately retrieved and resent. As you can guess, this is not the most efficient way, as a non-working receiving application usually needs some time to recover. The preferred case is where the service bus resends the message to the recipient with a certain delay. Here, the courier delivery metaphor comes to mind again. A delivery man standing at the recipient’s doorstep and persistently ringing the doorbell will not deliver the package if nobody happens to be at home. A solution that will allow you to store a message whose delivery failed is the Dead Letter Queue, which is a queue where the message is stored for a specified period, and then it can return to the original queue, from where it will be resent to the recipient. 

Bounce message 

We need to consider the last case: the situation when a number of attempts were made to send the message to the recipient, and none of those attempts was successful. Usually, it means that something is wrong with the system. And more often than not, the best solution is to send a classic email advising administrators about an error and the need to take specific actions. 

Summary 

Coming back to the question asked at the beginning – can a service bus work like a reliable courier? Yes, but it requires appropriate tools and a properly designed system architecture. You may achieve excellent results in terms of reliable messaging by skillfully combining Mule ESB and the RabbitMQ message broker.

Our Experts
/ Knowledge Shared

12.10.2021

A Single Source of Truth / A Key Way To Eliminate Technical Debt

E-Commerce

Repetition, inaccuracy and inconsistencies are issues every business wants to avoid. They’re also typical signs of poorly chosen – or poorly used – technologies. When used right, good technology can eliminate unnecessary repetition, ensure greater consistency and generally improve results over all. At the core of this is often the concept of “A...

05.10.2021

Personalization in E-Commerce / The Basics

E-Commerce

When talking face to face, personalization often happens at unconscious level. Interacts are smooth, adapted on the fly to suit the customer’s needs. Shop assistants can guide people to where they need to be and, when it comes to homing in on exactly what someone wants, it’s the fundamental secret to any good salesperson. But online? We must adapt....

28.09.2021

Black Friday / A Practical Checklist for E-Commerce Owners

E-Commerce

Everyone gets stressed on Black Friday but, between us, shoppers are the ones that have it easy 😉 Behind the scenes, entire IT, marketing and customer support teams need to prepare a solid strategy. And that’s not even mentioning warehouse, logistics or the subsequent challenge of order fulfilment… Black Friday (and Cyber Monday) are some of the...

Expert Knowledge
For Your Business

As you can see, we've gained a lot of knowledge over the years - and we love to share! Let's talk about how we can help you.

Contact us

<dialogue.opened>