kafka-logo-wideThere is a lot of buzz about Kafka these days among developers, and we get asked how Solace and Kafka compare often enough that we wanted to document the difference between the two. You can download a PDF summarizing the differences here.


Kafka and Solace products have similar main functions:

  • Allow applications to produce and consume information independent of each other
  • Store data in a highly available fault tolerant manner
  • Allow data processing in real time or near real time

Even though the products perform similar functions, they were built for fundamentally different purposes and evolved through different architectural decisions that make them perform quite differently.  This comparison will explain these differences and why they affect how easy it will be to build and operate an application designed to address challenges in the areas of hybrid/multi-cloud, digital transformation, event-driven microservices and the Internet of Things.

Background and Architectural Differences

Kafka was built to solve a log ingestion problem.  There was a need to consume a large amount of data from many sources quickly without impeding the data sources, then store that data and deliver it to a few large consumers, namely things like HDFS writers for a big data solution.  There was a need for horizontally scaled, efficient data consumption and delivery of data at a macro level, but little requirements for fine-grained filtering or high fan-out.  Kafka was therefore architected to be a very simple efficient data store and replay broker, with complexities pushed out to the more complex application APIS.

Solace was built to solve demanding and diverse application-to-application communication problems at scale and with high performance. Whether it’s financial trading platforms, data distribution over the WAN for hybrid cloud, massive scale IoT connectivity, or event-driven microservices – Solace messaging was designed to solve these challenging use cases. Solving such problems requires high performance in terms of both throughput and latency, along with support for consumers that are heterogeneous in both their design and data interests.  To address these needs Solace offers fine grained filtering and support for many open APIs and protocols, programing languages and LAN/WAN topologies. And to facilitate the monitoring and management of these complex environments, Solace technology offers end-to-end visibility including client state, automatic protocol translation and WAN optimization.

For more info about the differences between the Solace and Kafka architecturally please read Product Architecture and Datapath Architecture.

Having looked at the backgrounds and architectural differences, let’s look at how these differences affect the application of each technology to some common use cases, namely hybrid/multi-cloud, digital transformation/bi-modal-IT, event-driven microservices and the Internet of Things.

Hybrid/Multi Cloud

Most modern applications are not deployed in a single executable in a single run-time environment.  The applications are divided into logical components, with some pieces not feasible to move to the cloud, while other components are are better suited for one cloud infrastructure over another.  Instead of trying to mold the application to fit the environment, it’s better run each piece of the application where it most naturally fit.   This is the main reason companies are implementing multi-cloud (either multiple public clouds, or mix of public/private clouds) or hybrid cloud (where legacy components remain native to a datacenter) architectures.  In order to take advantage of these deployments, the links between clouds needs to be secure, fast and cost effective.

Besides delivering the correct data to the correct virtual location in a secure and efficient manner, there will be a requirement to interoperate with both legacy ESB technologies and the latest cloud specific data movement technologies.

Below is a table of key messaging features required for this use case, and how Solace and Kafka differ.

Message Feature Solace Kafka
Legacy Application Integration Needs adapter or bridge, easily map messages and patterns Needs adapter or bridge, complex to map message headers and exchange patterns as Kafka does not support several standard message exchange patterns
WAN Distribution All the required security and compression functionality is built in. Complex with additional components required for fault tolerance, security and compression.  Harder to troubleshoot performance and message loss.  Takes additional layers and care to prevent looping, as Kafka doesn’t offer loop detection.
Topic Filtering Fine grained topics and wildcards make message routing over multiple zones easy. Difficult to overlay multi-zone hierarchy on non-hierarchical topics.    No fine grained control of exactly what messages transit over the WAN.
Security Fine grain authentication and authorization Cannot achieve fine grain authorization. Main point of connection for WAN is MirrorMaker, which needs to be tested for exploits
Performance  w/TLS Reduced hops, and optimized TLS 50-90% performance degradation with TLS.  Significant increase in store and forward hops as well as messaging hops.


Digital Transformation

Digital transformation or bi-modal development is a great application of hybrid cloud, but it is so important it stands as a use case in its own right.  To achieve digital transformation you need to maintain legacy systems like mainframes because it’s too expensive or disruptive to move, but it’s usually also expensive and/or disruptive to update these legacy systems to meet modern requirements.  The solution is to extract transaction logs, or the like, from systems of record; then move this data into an environment where additional value can be extracted.

Below is a table of key messaging features required for this use case, and how Solace compares to Kafka.

Message Feature Solace Kafka
Transaction Log Integration Require adapter or database plug-in. May require development work Require adapter or database plug-in. More likely to have community or data source provided plug-in.
Legacy Application Integration Needs adapter or bridge, easily map messages and patterns Needs adapter or bridge, complex to map message headers and exchange patterns as Kafka does not support several standard message exchange patterns
WAN Distribution All the required security and compression functionality is built in. Complex with additional components required for fault tolerance, security and compression.  Harder to troubleshoot performance and message loss.  Takes additional layers and care to prevent looping, as Kafka doesn’t offer loop detection.
Topic Filtering Message routing over multiple zones and into applications, with sophisticated slow consumer handline, is a core, mature feature. Difficult to overlay multi-zone hierarchy on non-hierarchical topics.    No fine grained control of exactly what messages transit over the WAN. Slow consumer core feature.
Security Fine grain authentication and authorization Cannot achieve fine grain authorization.
External Distribution Seamlessly move data from enterprise to open web mobile API Requires REST and other gateways add overall solution complexity.
Replay There is no opportunity to replay from source in these use cases.  Solace current provided persistent messages, message caching and will soon provide replay. Replay from broker is core feature of the Kafka broker.  This is a single efficient message delivery model to consumer.
Performance  w/TLS Reduced hops, and optimized TLS 50-90% performance degradation with TLS

Event Driven Micro Services

In an event-driven microservices architecture, a service publishes events when something notable happens, such as when it updates a business object, and other services subscribe to those events. In response to an event, a service typically updates its own state. It might also publish more events, which then get consumed by other services.

Loosely speaking, microservices are a pipeline of processes, where the input is frequently a customer interaction via web protocols through an API gateway in some form of request/reply.  From this point services work in concert to fill the request.  Each stage of the pipeline can be written in the language and runtime environment that best suites the task.  The communication between the stages should be open and via standard protocols.

Below is a table of key messaging features required for this use case and how Solace compares to Kafka.

Message Feature Solace Kafka
Topic Filtering Fine grain filtering in the broker reduces load on applications.  Wildcard subscriptions. Strong fan-out support in broker. Filtering in the app means excessive network traffic.

Regex subscription causes increased latency

Horizontal Scale Apps Solutions for context aware and context free applications with selective acks for out of order event processes without duplicates on reconnect. Statefull scale apps, only sequential acks.  Duplicate events occur on reconnect.

Very good streaming fanout.

Security Fine grain authentication and authorization. Cannot achieve fine grain authorization, so filtering must be handled by the application.
Transactions Mature session-based transactions in multiple protocols. Kafka is adding this feature now, but limited in that transactions can only exist within a cluster.
Replay There is little chance to replay from source in these use cases.  Solace currently supports persistent messaging and message caching and will soon provide replay functionality. Replay from broker is core feature of the Kafka broker.  This is a single efficient message delivery model to consumer.
Performance  w/TLS Reduced hops, and optimized TLS 50-90% performance degradation with TLS

Internet of Things

Massively scaled IP-addressed applications need the ability to not just send data to other devices along with applications and analytics engines running in clouds and datacenters, but to send and uniquely addressed command and control messages.  In order to enable the adoption of IoT, issues like identification, privacy and security and semantic interoperability have to be tackled.

Below is a table of key messaging features required for this use case and how Solace compares to Kafka.

Message Feature Solace Kafka
Legacy Application Integration Seamlessly move data from enterprise to open web mobile API. Requires external REST gateway and a MQTT broker on the connection layer.
Large connection counts per broker Terminate hundreds of thousand of Web or MQTT connection per broker. Require external bridge or broker to terminate Web and MQTT connections.  Thus no real connection layer support for this use case.
Topic Routing Address millions of unique devices. Topic too heavy weight to address devices.
Security Able to create producer-specific and consumer-specific access control lists. Security solutions do not extend to the client level, just course grain topics.
Scale solutions to millions of connections Ability to build hieratical broker solutions to allow single point of connection for core apps while scaling edge connection. No simple way to build complex topologies..