Easy peasy RabbitMQ squeezy

Image from http://bit.ly/2U3PyJz

Back in mid-2018, I had to learn the RabbitMQ. I was trying to understand how it works. Failed miserably. This thing happens too often for me. There are several things I wanted to learn. JS, Elasticsearch, MongoDB or even React. Miserably failed every time. Tried learning those technologies several times but couldn’t concentrate.

A few weeks ago, again I had to study RabbitMQ. That time I was determined. I need to know this thing thoroughly. I started learning again and was trying to experiment on it. This is what I came to know about it. I am going to share my findings below.

It’s a long article. Take your time. Read slowly. Read again if you don’t get it. Write/Note down if possible.

RabbitMQ is a message broker. It takes messages from the producers (your application) and passes it to consumers (another application). Passing the messages to consumers are based on some criteria/algorithm. We’ll discuss it in a few minutes. Suppose you have several microservices and they all need to know a few events. Like a user has registered, a user placed an order and so on. Assume you have 1 microservice who raises an event when a user registers and two other services need to consume that event. Letting them know about the event using an HTTP request is a terrible idea. So, what we can do is to shoot a message to rabbitmq and the consumers will ask rabbitmq for that sort of data and they’ll get it from there if present. So, you no longer require to make *N* HTTP requests and those are time-consuming as well.

Terminologies

Before we dive into the basics, we need to know a few terminologies. Pay attention to the boldly marked terms.

• Connection: A network connection, e.g. a TCP/IP socket connection.
• Channel: A bi-directional stream of communications between two AMQP peers. Channels are multiplexed so that a single network connection can carry multiple channels.
• Content/Message: Application data passed from client to server and from server to client.
• Content body: A specific type of frame that contains raw application data. Content body frames are entirely opaque — the server does not examine or modify these in any way.
• Exchange: The entity within the server which receives messages from producer applications and optionally routes these to message queues within the server.
• Exchange type: The algorithm and implementation of a particular model of exchange. In contrast to the “exchange instance”, which is the entity that receives and routes messages within the server.
• Message queue: A named entity that holds messages and forwards them to consumer applications.
• Binding: An entity that creates a relationship between a message queue and an exchange. Used by the Queues to get messages from the exchange.
• Routing key: A virtual address that an exchange may use to decide how to route a specific message. A message property used by the exchange to route a message to queues.
• Durable: A server resource that survives a server restart.
• Transient: A server resource or message that is wiped or reset after a server restart.
• Persistent: A message that the server holds on reliable disk storage and MUST NOT lose after a server restart.
• Consumer: A client application that requests messages from a message queue.
• Producer: A client application that publishes messages to an exchange.
• Virtual host: A collection of exchanges, message queues, and associated objects. Virtual hosts are independent server domains that share a common authentication and encryption environment. (Simply it’s like the database of a DBMS)

Mostly these are all. If you still need to know more have a look at here.

How the data is passed from producer to the consumer?

• A producer P (your application) produces a message and sends it to an Exchange in RabbitMQ
• Along with your message, you send a routing key. Based on that the exchange decides where to send the message — A queue.
• A consumer C is connected to a queue to receive the produced message.

The way the messages are sent from P to C

Assume you want to drop your CV like old days. You put that in an envelope. Post it in a postbox. Postman takes that and routes it to desired address as you mentioned. Then the envelope is put in the office’s mailbox. And HR or somebody opens it up and gets your CV in their hand.

In this example

• Your CV within that envelope is a Message.
• You are the Producer.
• The mailing address is the Routing key.
• Postbox to post office is the Exchange.
• Office’s address is the Binding Key.
• Office’s mailbox is the Queue.
• HR or receiver of the CV is the Consumer.

When you post a message in RabbitMQ, you post it to an Exchange. All of the exchanges have a name, a type and may have few properties. There are 4 types of exchanges by default.

No above-mentioned exchange holds any message in it. It just passes to the queue if anything is attached to it based on routing key/headers (that algorithm, mentioned above).

NB: You may want to know about Queues before. The queue section is written below in a few lines. If you want to understand the images properly, you can check the queue section before. BTW, ‘ucl.two’ in figures is a queue as well, like the other queues.

Direct exchange: Direct exchange sends the messages as soon as it receives it to the Queues if the routing key + binding are the same. There is a predefined special type of direct exchange which doesn’t have any name. It’s an empty string "". In case of default (“”) exchange, it routes the messages to those queues whose name matches the routing key (Fig:2).

Fig:1 — Direct Exchange Example (type: direct, name: user.actions.direct)

Fig:2 — Nameless Direct (default) exchange example ( type: direct, name: ‘’)

Topic exchange: Topic exchange is like the direct exchanges but the key difference is that the queue can use wildcard bindings to get messages from the exchanges. The routing keys are words separated by dots — user.created / user.loggedin / user.loggedout. Only * and # can be used as the wildcards. * — means one word in any position of the key. # — means zero or more words in any position of the key.

Fig:3 — Topic Exchange Example (type: topic, name: user.actions.topic)

Fanout exchange: Fanout exchange doesn’t bother about the routing key. It passes all the messages it receives to all the connected queues. Even if the routing key is given to bind the queue and the exchange, the routing key will be ignored.

Fig:4 — Fanout Exchange example (type: fanout, name: user.actions.fanout)

Headers exchange: Headers exchange are like the Direct exchange. Direct exchange routes the messages based on the routing key. But the headers exchanges routes the message based on the headers. If any route key is given, the key is simply ignored. Headers are key-value pairs. Any header key having x-* will not be regarded as a valid header to be matched to route message to queues. When a queue tries to bind itself with this exchange, it sets x-match to any of them. all — which means all headers values should match to get the message or any — which means at least of the header's value should match to get the message.

Fig:5 — Headers Exchange example (type: headers, name: users.actions.headers)

x-delayed-message exchange: This exchange is not available by default. You must have to install a plugin and enable it. All the previously explained exchanges come with rabbitmq by default which doesn’t hold the message with any delay. As soon as the messages arrive at the exchange, it passes to the queue. With this exchange, you can delay the message of X milliseconds before it reaches to the queue. But you need to define the type (direct, headers, topic) of exchange using ‘x-delayed-type’, which will define what will happen when the message will be ready to be delivered to the queue.

That’s it for the exchange.

Queue

Queues are the place where the messages get stored. After producer posts a message to the exchange, it’s routed to the queues if the binding key matches to the routing or the header matches with the header. There is no type of queues. But, bear in mind that all the queues are bound to the nameless/default direct exchange with the routing key of the queue’s own name. That means if a message is sent to default exchange having a routing of the any queue’s name, then the queue will receive that message. All the queues must have a name and routing key (depending on exchange it binds to).

Properties

• durable — Used by both the exchange and the queues. Means the queue and exchange survive if the rabbitmq restarts.
• auto_delete — Used by both the exchange and the queues. Means if no queue is bound to any exchange then the exchange will be deleted. Like that if no consumer is bound to the queue, then the queue will be deleted automatically.
• internal — Used by Exchange. If an exchange is internal, no client will be able to publish to the exchange. Only for an exchange to exchange binding.
• alternate-exchange — Used by Exchange. If a message can not be routed to any bound queues (for any reason), the message will then be redirected to that specified exchange.
• exclusive — Used by Queue. Only the current connection can use the queue. As soon as the connection goes off, the queue will be deleted.
• x-message-ttl — Used by Queue. Sets TTL for a message that arrives. Messages will be moved from the queue after expires the TTL. In milliseconds.
• x-expires — Used by Queue. Determines how many milliseconds the queue can be left unused before it’s deleted automatically.
• x-max-length — Used by Queue. A max number of received messages can be stored before it starts dropping from its head.
• x-queue-mode — Used by Queue. Default is “default”. If set “lazy”, then it’ll set the messages to the disk as much as possible to reduce the RAM usage.
• delivery_mode — Used by Message. Persistent, Nonpersistent. Determines if can survive after server restart.
• application_headers — Used by Message. Passed along with the message. These are used by the exchange on headers exchange.

QoS

The QoS means Quality of Service. In rabbitmq, if you’re asking for messages from queues and you have set “no ack” to true, which will let queues to delete the message as soon as you receive it. Even if your application crashes and the message is not handled properly, the message will be deleted for forever. But, by defining the QoS before you start to consume messages from a queue, using a number like 5 (as for an example called prefetch_count), you’re declaring that you should not be given more than 5 unacknowledged messages on the same channel your consumers are connected to. If you can’t acknowledge any of those given messages, you’ll no longer be given any new message to process. As soon as you acknowledge the queue that you’ve processed the message, it’ll then start sending you new messages till you reach that prefetch_count. To acknowledge the delivery, you must have to notify via the same channel they were received. Types of acknowledgments.

• ack — Positive acknowledgment. Instructs RabbitMQ that the message can be discarded.
• reject — Negative acknowledgment. Like positive ack, it’ll delete the message.
• nack — Negative acknowledgment. Should not delete the message.

If a message is not acknowledged by the consumer, it’ll automatically be requeued when the channel or connection on which the delivery happened is closed.

Again, remember that the QoS is set on a channel level.

Things you should know

• Whenever you publish a message, it goes to the exchange. Not in any queue. There is no need to declare a queue while publishing. (Not needed means, you don’t need to consider queue while publishing a message. It’s not required.).

From producer to exchange

• To consume a message, a queue is bound to an exchange. A consumer consumes messages from a queue. That means you should first bind a queue to an exchange and then you can only consume a message. For default exchange (nameless exchange) you cannot bind a queue to that exchange. It’ll be sent directly whenever it finds the routing_key \== queue_name.

Message from exchange to queue to consumer

• RabbitMQ dispatches the messages to its consumer in a round-robin fashion. On average all the consumers will get almost the same amount of messages. That’s why you saw in figures that consumers of the same queue with multiple consumers received half of the messages each.
• If you have multiple queues with the same routing key, you’ll get the same message in both the queues. You’ll not get those messages in round-robin fashion. Because round-robin only works while consuming messages. Not from exchange to queues.
• If no queue is bound to exchange and by that time a message is sent to that exchange. Or a message with a routing key is sent to exchange and no queue is using that key to bind to that exchange. In both cases, the message will be lost. But, to overcome it, we can usealternate-exchange (AE) which will send the message to the defined queue in these scenarios. This argument must be set during the creation of an exchange.
• If the queue is holding any message and no one is listening, then if the queue is about to delete the message (TTL, Drop from the head), then these messages can be sent to a Dead Letter Exchange (DLX). It’s an argument you need to set while creating the queue.
• If a persistent message is sent to a non-durable queue, the message will no longer be persistent.

That’s it. This is what I found and came to know about while learning RabbitMQ.

Questions

I just came to know the following aroused in people’s mind. So, the questions and the answers are given below

Q: What are the mailing address & office address in the example?

• The mailing address is that address you put on the envelope which is the destination/office address.

Q: What is a channel?

• The channel was not included in the terminologies section before. It’s been added now. When your application communicates with RabbitMQ, it uses a connection (also added on terminologies). A channel is kind of a pipe through which these data/streams are passed through.

Q: I understand UCL is a queue but what is UCL.TWO?

• Just like ucl, the ucl.two is also a queue. You can have multiple queues of the same routing key. If you have multiple (N) queues for the same routing key, you’ll receive identical messages to all those N queues. That is why ucl and ucl.two receives the same amount of messages.

Q: Why UCL had two consumers and others had one consumer only?

A: It’s all about how you want to develop the application. Suppose your queue receives too many messages and to keep your queue short you may need to add multiple consumers. If the volume of messages is not that high and one consumer can clean up that queue messages easily you’re good to use one consumer. So, it’s up to you how many consumers you want to your queue.

Finally, a Github repository is created for this purpose. Readme says how to install it. It’s in PHP. Read it, play with our preferred language.

Happy Coding. ❤

Edits:

• When a consumer (subscription) is registered, messages will be delivered (pushed) by RabbitMQ using the basic.deliver method. The method carries a delivery tag, which uniquely identifies the delivery on a channel. Delivery tags are therefore scoped per channel.
• Because delivery tags are scoped per channel, deliveries must be acknowledged on the same channel they were received on.
• The difference is primarily in the semantics: positive acknowledgments (ack) assume a message was successfully processed while their negative counterpart (reject) suggests that delivery wasn’t processed but still should be deleted.
• if consumers' TCP connection or channel is closed before successful delivery, the message sent by the server will be lost. Therefore, automatic message acknowledgment should be considered unsafe and not suitable for all workloads.
• Automatic ack doesn’t have any prefetch limit.