Systems Integration Buyers Guide

Automate the flow of data between your systems to power massive business growth.

This systems integration buyers guide will save you time, money and credibility. Avoid costly mistakes and let us help you find the most suitable systems integration tool for your business.

Why Does Your Business Need An Integration Tool?

Most businesses today have at least several mission critical systems, for example: an ecommerce systems, CRM, ERP, customer service, accounts package, business intelligence tools, and this is just the tip of the iceberg.

The ability to quickly and accurately share data between systems creates a competitive advantage.

It might mean that your workforce can spend less time executing repetitive tasks. Perhaps it will allow you to fulfill cusomter orders more quickly. Or maybe it will allow you to make real time, automated decisions such as serving personalised content to webste visitors, dynamically repricing products or automatically adjusting online advertising campaigns.

Since APIs are now everywhere, one solution to deliver integrated system is to start writing code that retrieves data from one system and writes it into another system. While this approach has its place, it is generally not a good idea when integrating the estate of enterprise systems. There a many reasons why this is the case but the main ones are:

1. It becomes increasingly difficult and costly to maintain and evolve a complex, spaghetti architecture of tightly coupled systems
2. Replacing one system can require re-integrating and re-testing the whole connected IT estate due to tight coupling
3. There is no single control panel to spot failures and drive remediation
4. Fault tolerance capabilities have to be implemented in code rather than simply configured
5. It is more difficult to operationalise and knowledge share in the absence of key, technical personnel

What Types of Integration Tools Exist?

Fortunately, there are many integration tools on the market that help avoid this integration spaghetti. The first step in selecting the best tool for our business is to understand the differences between the main types of integration tool.

1. iPaaS (Integration Platform As A Service)
2. ELT (Extract, Load, Transform)
3. ETL (Extract, Transform, Load)
4. Reverse ETL
5. Event Stream
6. Message Broker
7. ESB (Enterprise Service Bus)
8. Integration Framework

iPaaS (Integration Platform As A Service)

An iPaaS is a tool that provides building blocks to quickly integrate with commone systems through a library of connectors. There is typically a design interface, or canvas, that allows uses to build data flows, alos known as pipelines and recipes. These data flows allow users to implement logic such as conditional tests, loops, data transformations and error handling.

A good iPaaS solution should support the implementation of industry recognised EIPs, (enterprise integration patterns), that have their roots in established software development principles, as well as offering non-technical users an interface that they can use to operationalise integrtions across the IT estate, an even create new integrations without requiring the input of a technical team...provided they following the pre-agreed implementation principles!

ELT (Extract, Load, Transform)

ELT tools are the successor to ETL tools and they have become very popular in recent years. Their job is to extract raw data from one or more source systems, load that data into a target system in raw form, and then optionally transform the data for onward storage, but leaving the raw data unchanged.

Like the iPaaS tools, ELT tools will boast a large library of connectors to support quickly gaining access to data in source systems. They will typically have a smaller range of connectors for target systems, most frequently landing data into cloud a database, data warehouse or data lake.

The typical use case for ELT tools is to land data from multiple systems into a target that will be used for data analysis, rather than synchronising data into other operational IT systems. The reason for loading and storing the raw, unchanged data into the target is because if data requirements change in say a business intelligence tool like Power BI or Tableau then it is possible to simply update the transformations without needing to worry about extracting from source systems again, which can be resource and time intensive for large datasets.

ETL (Extract, Transform, Load)

ETL tools delivered one of the earlier, standardised approaches to moving data between systems. A library of connectors is used to connect to source systems, the data is transformed to meet the requirements of a target system and then the data is loaded into that target system. The target system could be a datastore such as a data warehouse, but it could also be another operational IT system, for example fetching customer records from an ERP system and batch loading them into a CRM system.

This approach is becoming less common, with data pipelines for data warehousing tending to use the ELT pattern and operational system integrations tending to use the more real time approach offered by iPaaS tools.

Reverse ETL

As businesses shifted away from ETL towards ELT, Revers ETL has arrived on the scene offering a tool that allows a business to extract data that has been loaded into a data warehouse or data lake from multiple source systems, transform that data and then load it into an operational IT system using a broad library of connector modules.

Message Broker

A message broker is a tool used by developers to send data between systems in a decoupled fashion. One system is able to send a message and one or more other systems are able to receive that message. A mature collection of enterprise integration patterns such as Pub/Sub & Request/Reply use the capability of message brokers to create highly performant, highly scalable, decoupled integrations. However, they are usually built by, owned and operated by technical teams.

Event Stream

Despite the scalability of message brokers, they have their limits. LinkedIn used message brokers extensively in their earlier years but all of the views, clicks, likes etc that their systems generated and consumed was becoming difficult to manage with too many integration endpoints to support and all at massive scale.

The LinkedIn team designed and built Kafka, an event streaming engine that they have since open sourced. One of the significant differences between message brokers and event streams is described by the phrase "dumb pipes and smart endpoints". This phrase is describing how an event stream is not interested in implementing routing rules or processing logic, it is just a dumb pipe. The event stream allows smart endpoints to connect, or subscribe to the stream of events.

The smart endpoints couple be system specific microservices, demonstrating an implemetation pattern that delivers true decoupling at scale.

ESB (Enterprise Service Bus)

ESB tools sought to operatinalise the message broker technology by creating a toolkit to design and build message based integration with integrated error handling and remediation tools.

Conceptually there are many similarities between ESB tools and iPaaS tools. The main differentiators tend to be that ESB tools sought to centralise routing logic, data transformations and business logic and were typically deployed as on-premise software tools. iPaaS on the other hand tends to decentralise logic and transformation and is deployed in the cloud.

Integration Framework

Integration frameworks are software libraries that act as accelerators for developers who are coding the integrations between their systems. The framework delivers efficiency and standardisation by offering a pre-built library of components that implement the most common EIPs.

What Type of Integration Tools Does Your Business Need?

An integration platform as a service (iPaaS) is a technology tool used to efficiently and consistently integrate applications and data sources across your IT estate.