This article provides performance numbers for both the Solace VMR and the 3560 message router appliance. It focuses on the Solace message router throughput and aims to help you understand the point-to-point and fan-out performance of Solace message routers.

Revision History

  • Sept 2017: Updated with results from VMR release 8.6.0
  • May 2017: Updated with results from ADB4 and 8x10GE NAB, appliance release 8.1.0
  • April 2017: Updated with results from VMR release 8.2.0
  • Sept. 2016: Updated with results from SolOS release 7.2
  • Nov. 2015: Posted with results from SolOS release 7.1.1

Table of Contents

Summary of Results

Detailed descriptions of test methodology, parameters and results are provided below, but here is a summary of results:

Table 1 – Summary of Results

   Virtual
Message
Router
3560
Message Router
Appliance
Messages Per Second
Non-persistent (Direct) point-to-point 1,540,000 9,290,000
Non-persistent (Direct) fan-out 5,200,000 28,400,000
Persistent message point-to-point 145,000 646,000
Persistent message fan-out  858,000 5,530,000

These results show that the Solace VMR offers very fast, scalable performance. In general, it behaves a lot like the 3560 appliances, just at lower message rates. That isn’t surprising since the VMR, which runs on commodity hardware, is based on the same underlying software, SolOS, that runs our appliances. Solace Message Router Appliances still offer the highest possible throughput and performance with our purpose-built hardware, so if you want to support massive messaging capacity or scale across many applications in a compact footprint, this hardware is for you.  If you need less performance, want to deploy messaging into small offices or IoT, or scale your system horizontally in a private or public cloud, then the VMR is for you. The beauty is you can mix & match or switch from one to another without application or management impact.

Methodology

These performance results look at message throughput for point-to-point and fan-out scenarios. The results measure the Solace message router in aggregate in terms of messages per second. These message exchange patterns are explained in more detail in our core concepts page but this section briefly describes these two patterns as they apply to the performance scenarios.

Point-to-Point

A point-to-point flow is a single publishing client sending messages to the Solace message router which are received by a single subscribing client.

Fan-Out

Fan-out is the typical publish/subscribe scenario where a message is published once to the Solace message router and forwarded to multiple clients who have expressed interest in receiving messages of that kind through topic subscriptions. So the message is then fanned out to the clients by the Solace message router, and each client receives a copy of the message.

SDKPerf Test Tool

SDKPerf-Icon-TransparentFor all scenarios, the Solace test tool SDKPerf was used as the traffic source and sink. Because we take performance very seriously, over many years we’ve built SDKPerf into a very capable tool that can be used to test many different messaging APIs. You can find out more about the tool on our SDKPerf overview and you can download the tool and try it yourself as well. For these tests, sdkperf_c was used.

For each scenario, SDKPerf measures the following:

  • The received message rate.
  • Message loss using message ordering information that publisher instances of SDKPerf embed in the messages they send.
  • Message payload integrity by confirming the received message payloads are unmodified by the messaging system.

This ensures that the messaging system is running fast and correctly delivering all messages to the correct consumers without corruption.

Message Delivery Modes

The scenarios in this article cover two types of messaging delivery modes:

  • Non-Persistent – Which is “at most once” messaging. Solace message routers provide two options for “at most once” messaging. Both Direct messaging and Non-Persistent messaging achieve this level of service. The results in this article use Solace Direct messaging which is the higher performance option.
  • Persistent – Which is “at least once” messaging. To achieve full persistence, producers must receive acknowledgements from the messaging system when the Solace message routers have fully persisted the message to non-volatile storage. And messages for consumers must not be removed from the message router until an acknowledgement is received. Fully persisted, means the messages is flushed to disk. Often this option is called failsafe messages in other message brokers. Simply caching the message in the filesystem is not equivalent and can lead to message loss in failure scenarios. Therefore in the article when results are listed for Persistent messages this is for fully failsafe messaging with no risk of message loss.

Performance Scenarios

In the performance scenarios below, the goal is to measure the maximum performance of the Solace message router in each scenario. To achieve this, we use groups of point-to-point or fan-out clients all running in parallel. Each group of clients is monitored to make sure it is running at a rate which it can handle to avoid an individual client becoming the bottleneck and skewing the performance results artificially.

For point-to-point flows, this means you end up with the scenario illustrated in the following figure.

In this figure, client groups are depicted using colors. The figure shows an orange, blue and grey group. Each group has a single publishing client sending to a single receiving client. The test uses enough groups of clients such that the Solace message router becomes the bottleneck, not the clients. When you monitor the Solace message router and observe the aggregate messages per second across all clients then you determine the maximum message rate a Solace message router can sustain for the point-to-point scenario.

Similarly for fan-out scenarios, you will end up with the scenario illustrated in the following figure.

Again in this figure, client groups are depicted using colors. The figure shows an orange, blue and grey group. And again, the number of client groups is scaled as required until the Solace message router maximum fan-out rate is observed. The maximum fan-out rate is the total output message rate for the Solace message routers. Within each group, the number of subscribing clients is dictated by the fan-out factor being measured. So a fan-out factor of 10 will require 10 subscribing clients per group. To figure out how many messages per second are being sent by clients to the Solace message router, simply take the output message rate and divide by the fan-out factor. This is the input rate.

At Solace we believe in full transparency for all aspects of performance results, so comprehensive details on the exact hardware, setup and methodology are described below in the section Appendix A: Test Setup.

Results

The following is a brief introduction to the performance scenarios to help you understand the results.

The first set of scenarios cover Non-Persistent messaging using the Solace Direct message delivery mode. The results cover point-to-point and fan-out message traffic.

The second set of scenarios cover Persistent messaging. Here again the results cover point-to-point and fan-out message traffic. Additionally, often for Persistent messaging, people are interested in how fast messages can be saved to disk and separately how fast messages can be retrieved from disk and delivered to clients. For the Solace VMR, the persistent store is also just a “disk”, but for our hardware appliance, it is a combination of custom hardware backed by a disk. Regardless of the technology, we call the persistent store the Message Spool and the results below include these two scenarios for various numbers of Persistent Queues in the system.

Point-to-Point Non-Persistent (Direct) Messaging

The following table shows the aggregate output message rate (messages/sec) for Solace Direct messages (non-persistent) as the message payload size is varied. In this test groups of clients are sending messages in a point-to-point fashion. Twenty parallel sets of clients are used to generate the required traffic.

Table 2 – Point-to-Point Non-Persistent Message Throughput (Output Msgs/Sec)

Message Size
100B 1KB 2KB 10KB 20KB 50KB
VMR 1.54M 784K 474K 104K* 53K* 21K*
Appliance 9.29M 4.05M 2.44M 885K* 443K* 170K*

* The results measured reached the bandwidth limit of the NAB on the 3560 Appliance, or the NIC on the VMR.

 

From these results you can see that as expected the performance is fastest for small message sizes, as these are easier for the Solace message routers to process efficiently.

It is also interesting to graph these results and add in calculated bandwidth in Gbps. From this you can see that the Solace VMR bandwidth utilization is always increasing as the message size increases, until it reaches the 10Gbps bandwidth limit of the NIC at the 10K byte message payload size.

For the Solace 3560 Appliance, the results for the bandwidth are similar. The hardware I/O card, called the Network Acceleration Blade or NAB, has a messaging capacity of around 80 Gbps. So as the message size increases, the NAB is able to saturate the network bandwidth and reaches the maximum bandwidth at around the 10K byte message payload size. From there the output message rate is governed more by the bandwidth capabilities of the NAB than other factors.

Fan-out Non-Persistent (Direct) Messaging

The following table shows aggregate output message rate (messages/sec) for Solace Direct messages (non-persistent) as the message fan-out is varied. At a fan-out of 1, this scenario is equivalent to the point-to-point scenario above. Then more clients are added to consume the published messages. This increases the fan-out for each message. The Solace message router maximum output is recorded for each fan-out of each message size as shown below. Again twenty parallel groups of fan-out clients are used to avoid any individual client becoming the bottleneck in the test.

The following table shows results for the Solace VMR and 3560 appliance. It also shows various message payload sizes in bytes for each Solace message router.

Table 3 – Fan-out Non-Persistent Message Throughput (Output Msgs/Sec)

Msg Size
(Bytes)
Endpoints/Subscribers
1 2 5 10 50 100
VMR 100 1.54M 2.62M 3.20M 4.27M 4.85M 5.2M
1K 784K 1.04K 1.09M* 1.10M* 1.10M* 1.08M*
2K 474K 542K* 553K* 546K* 552K* 545K*
Appliance 100 9.29M 15.90M 24.20M 27.10M 28.00M 28.40M
1K 4.05M 5.51M 7.93M 7.82M 8.00M 8.66M*
2K 2.44M 3.80M 4.00M 4.02M 4.06M 4.39M*

* The results measured reached the bandwidth limit of the NAB on the 3560 Appliance, or the NIC on the VMR.

Graphing these results lets you analyze the overall impact of message delivery fan-out across message payload sizes. For both the Solace VMR and the 3560 appliance, at low message sizes, the overhead of processing the small messages dominates. However, as message sizes increase the limiting factor shifts to handling the bandwidth of the messages. This is seen on the graph when the curves flatten.

In the case of the 3560 appliances, output increases to the point of eventually being able to saturate the 80 Gbps bandwidth capacity of the NAB.

Point-to-Point Persistent Messaging

The following table shows aggregate output message rate (messages/sec) for fully persistent messages as the message payload size is varied. In this test, groups of clients are sending messages in a point-to-point fashion. Twenty parallel sets of clients are used to generate the required traffic.

Table 4 – Point-to-Point Persistent Message Throughput (Output Msgs/Sec)

Message Size
512B 1KB 2KB 4KB 20KB
VMR
(w/o HA)
145K 130K 101K 76K 26K
VMR
(with HA)
97K 84K 62K 43K 10K
Appliance
(with or w/o HA)
646K 646K 502K 283K 60K

As with the non-persistent scenario, from these results you can see that the performance is fastest for small message sizes as these are easier for the Solace message routers to process efficiently. Each of these messages must be saved to non-volatile storage prior to acknowledging the publisher so the overhead is much higher compared to non-persistent messages.

It is also interesting to graph these results and add in calculated bandwidth in Gbps. From this you can see that the Solace VMR bandwidth utilization is always increasing as the message size increases.


For the Solace 3560 Appliance, the results for the bandwidth are a little different. The hardware guaranteed messaging card, called the Assured Delivery Blade or ADB, has a messaging capacity of around 9.5 Gbps. So as the message size increases, the ADB becomes the bottleneck and the system reaches the maximum bandwidth at approximately the 4K Byte message payload size. From there the output message rate is governed more by the bandwidth capabilities of the ADB card than other factors.

Fan-out Persistent Messaging

The following table shows aggregate output message rate (messages/sec) for fully persistent messages as the message fan-out is varied. At a fan-out of 1, this test is equivalent to the point-to-point test above. Then more clients are added to consume the published messages. This increases the fan-out for each message. The Solace message router maximum output is recorded for each fan-out and message size as shown below. Again twenty parallel groups of fan-out clients are used to avoid any individual client becoming the bottleneck in the test.

The following table shows results for the Solace VMR and 3560 appliance. It also shows various message payload sizes in bytes for each Solace message router.

Table 5 – Fan-out Persistent Message Throughput (Output Msgs/Sec)

Msg Size
(Bytes)
Endpoints/Subscribers
1 2 5 10 50
VMR
(w/o HA)
1K 130K 227K 453K 652K 858K
2K 101K 181K 365K 483K 541K*
20K 26K 46K 55K* 56K* 56K*
VMR
(with HA)
1K 84K 174K 361K 493K 695K
2K 62K 132K 284K 376K 462K
20K 10K 21K 44K 50K 53K*
Appliance
(with or w/o HA)
1K 645K 1.25M 2.69M 3.87M 5.53M
2K 502K 938K 1.85M 3.10M 4.18M*
20K 60K 120K 301K 316K 434K*

* The results measured reached the bandwidth limit of the NAB on the 3560 Appliance, or the NIC on the VMR.

Graphing these results lets you analyze the overall impact of message delivery fan-out across message payload sizes. For both the Solace VMR and the 3560 appliance the behaviour in this persistent scenario is different than the non-persistent case. In this scenario the work of fanning out the messages is handled by the Network Acceleration Blade (NAB) in the 3560 appliance or the I/O software in the Solace VMR. This offloads the processing from the guaranteed messaging components and enables the message rate to continue to increase as fan-out increases. In the case of the 3560 Appliance, output increases to the point of eventually being able to saturate the 80 Gbps bandwidth capacity of the NAB still all fully persistent.

  

Message Spool Read & Write Performance

These tests focus specifically on how fast a Solace message router can save messages to its non-volatile storage called the Message Spool. For the Spool scenarios, publishing clients are sending persistent messages to the Solace message router destined for durable queues. Since there are no consumers connected to the queues, the Solace message router must save these messages and the state of each queue in its Message Spool. This scenario highlights the Solace message router’s capacity to store messages for offline clients. Then for the Unspool scenario, all the publishers are stopped and this time consuming clients are connected to the queues containing the saved messages. These consuming clients try to receive the messages as fast as possible. This highlights the maximum rate at which Solace messages routers can retrieve messages from the Message Spool and deliver them to consumers. This is in essence the rate at which clients can recover following some application outage.

The following table shows the messages/sec observed in aggregate across the Solace message router. The messages all have a payload size of 2K bytes. The results are captured for a single queue, 10 queues, and 50 queues to give you an idea of how these rates vary by number of queues. In all cases messages are sent directly to individual queues in a point-to-point fashion.

Table 6 – Message Spool Read & Write Performance (Msgs/Sec)

Scenario Endpoints/Subscribers
1 10 50
VMR (without HA) Spool 123K 122K 123K
Unspool 221K 215K 167K
VMR (with HA) Spool 82K 79K 80K
Unspool 218K 208K 147K
Appliance
(with or w/o HA)
Spool 519K 515K 509K
Unspool 305K 232K 229K

One of the key differentiators of Solace message routers is their slow or offline consumer handling. Solace message routers are engineered for slow consumer handling (see demo here). They avoid impacting the publisher’s ability to send messages or online consumer’s ability to receive messages. That’s why you will see flat lines in the following graphs for message spool performance.

When unspooling messages from the message spool, the software will recover clients as fast as possible without impacting online clients. Therefore the unspool performance varies more but is still impressive.

 

Appendix A: Test Setup

Performance metrics are only useful if you understand how they were measured and what test equipment was used. Quoting numbers without this is meaningless.

The following two figures outline the lab setup when running the tests. Instances of the SDKPerf tool were run on test host servers which were running Linux. Depending on the test requirements, different numbers of servers were used such that the clients were never the bottleneck. For the Solace Message Router Appliance scenarios, a pair of 3560 message router appliances setup in the typical fault tolerant configuration with attached external storage. Up to 12 test hosts were required to generate the appropriate test load.

throughput-page_image-5

For the Solace VMR tests, a VMware ESXi server was used to host the VMR VM image. And for the VMR scenarios, less tests hosts were required for the traffic generation due to the difference in the performance capabilities.

throughput-page_image-6

In both cases, a Mellanox SX1016 10GigE Ethernet switch was used to provide the LAN connectivity.

The following tables give full details of each of the hardware used in testing.

Solace VMR

Platform Solace VMR Enterprise Edition
Hypervisor VMWare, Inc. VMware Virtual Platform
BIOS Version 6.0.0
CPUs Intel(R) Xeon(R) CPU E3-1270 v5 @ 3.60GHz with Hyperthreading
Memory 32 GB
Datastore 40 GB Intel SSD 750 Series
Network Interface Intel Corporation 82599EB 10-Gigabit SFI/SFP+
VM Settings 4 cores, 8 GB RAM
API All tests use Java clients (Solace JCSMP API)

Solace Message Router Appliances

The Solace message router appliances were 3560 appliances setup as a fault tolerant pair. The specific details of the hardware are included below including the details of the required attached external SAN storage.

Solace Message Router Appliances
Platform Solace 3560
NAB (I/O Card) NAB-0810EM-01-A (8x10GE)
ADB (Guaranteed Messaging Card) ADB-000004-01-A  (ADB-4)
HBA Connectivity HBA-0208FC-02-A (8Gb)
Product Keys GM650K Performance Key
Storage Area Network (SAN) Characteristics
Controller Redundant controllers and write-back cache
RAID RAID 10, 4 + 4 Stripe Size
Drives Serial Attached SCSI (SAS), 10K RPM
Connectivity 8 Gbps Fibre Channel

Required Test Equipment

Up to 12 performance test hosts were used to generate and sink the required message rate for the various scenarios. They all had characteristics that matched the following specifications:

Performance Test Hosts (Load Generation Hosts)
CPU Intel Core i7-3930K CPU (6 cores) @ 3.20GHz (HT Disabled)
Memory 16 GB
Hard Disk Western Digital WD2002FAEX (2 TB)
Network Interface Ethernet Converged Network Adapter X520-DA2 (10 GigE)
Host OS (Kernel) CentOS release 6.4 (2.6.32-358.el6.x86_64)

Connecting all the equipment together was a Mellanox 10GigE Ethernet Switch.

Ethernet Switch
Model Mellanox SX1016
Description 64-port non-blocking 10GigE Ethernet Switch