Open Source Solutions and Other Strategies for Successful Interface Aggregation
Tim LeMasters | January 25, 2017
Many federal agencies rely on a diverse set of data sources to accomplish their mission. This spotlights data sharing initiatives as critical to government effectiveness. A key challenge in their execution is the heterogeneity of platforms, protocols, languages, and message exchange patterns used across information sources, making for a complex environment and a potential obstacle to mission success.
Having a well-designed interface aggregator, or “hub,” addresses this challenge by allowing an agency to conduct time-sensitive data communications with a wide variety of other organizations. Functioning invisibly to end users, the hub establishes the necessary connections and performs the needed message routing, protocol mediation, and message transformation in a secure manner. This enables the data exchange that is crucial to the success of multiple information sharing partners involved. Here are four essential recommendations to keep in mind when planning an interface hub solution.
1. Start with the Right Solution Platform
There are three general directions that your agency can take in constructing an interface hub, each with its own advantages and limitations.
Enterprise Service Bus (ESB) – ESBs are large software packages that aim to provide a comprehensive, ready-to-use set of capabilities that go beyond routing, mediation, and transformation. They include rich sets of connectors for various types of back ends (web services, message queues, databases, email, and many others). They also provide support for high-end features such as business process orchestration, complex event processing, service management, and governance.
Generally, this power does not come for free. ESBs are usually proprietary, and costs for initial licensing and ongoing operations can be significant. Even in the case of open source solutions, you may need to purchase a support package, and the expense of maintaining a large system may not make sense if you only use a fraction of it. Finally, for any ESB solution, expect that some complex configuration and/or custom development may be necessary.
Enterprise Integration Framework (EIF) – EIFs contain building blocks for implementing industry-recognized Enterprise Integration Patterns, providing the basis for a hub that supports routing, mediation, and transformation. Additionally, EIFs provide rich sets of connectors, similar to ESBs. Examples of EIFs include Apache Camel and Spring Integration, both of which are open source solutions with wide adoption and active communities. If you don’t need many of the premium features of an ESB, using an EIF can avoid much of the effort of custom development and many of the costs associated with an ESB.
Custom development – This option is potentially the most costly and risky of the three. In addition to increased labor costs, you miss out on the extensive exercise of the product by a broad user base and the enthusiastic input of an open source community. You should consider it only if your communication needs are extraordinarily unusual, or the protocols used to communicate are almost entirely proprietary in nature. More than likely, you could meet your requirements by supplementing an EIF with a smaller amount of custom development.
2. Plan for Performance
Start by determining the different types and frequencies of transactions your interface hub will need to support. For example, you might determine that the great majority of your users will be executing searches that involve relatively small amounts of data, while a small set of users will be exchanging complex documents that include large numbers of images—two very different loads that must be taken into account. Also, estimate the timing and size of peaks or surges in volume. When performing these analyses, use the best data available — that is, data gathered from operational scenarios that most closely match your own. These parameters are essential for planning capacity.
Also, be sure to plan your system with the ability to expand capacity as needed. Ideally, your hub will be horizontally scalable, meaning that you can increase capacity by adding more commodity platform units, such as servers, distributing load across them. This is especially important today, because cloud providers such as Amazon Web Services and Microsoft Azure make it faster and cheaper than ever to perform such an expansion. They even allow you to dynamically adjust the size of your infrastructure based on current usage, thus optimizing ROI.
3. Be Strategic in Your Interface Design
In general, you want to minimize the amount of back-end specific knowledge needed by developers using your hub. Making your API as consistent as possible will make it easier to integrate with your hub, producing a better one-stop shop. Ultimately, this increases your hub’s value proposition and adoption.
You also want to align the language used in your interface with the language of your user community or industry, including the names of objects and activities. This will help ease communication between developers, end users, and anyone involved in support of the hub.
Going one step further, try to use standard interface specifications, even if you have to customize or extend them a bit to fit your data model. Using a widely adopted standard will make integration with other systems using that standard much easier and cheaper. This is because you will reduce time spent on interface definition and eliminate many message transformations. For example, if any of your target sources are health care organizations, using the HL7 standard can reduce your development cost and promote adoption of your solution.
Last but not least, you must make sure your reference data is well-managed. To understand the issue, consider the example of eye color. One data source may include eye colors of black, brown, green, gray, and blue, while another service uses black, brown, green, hazel, blue, and violet. You will have to augment your set and/or map between sets as you incorporate various data sources. Also, reference data can change over time — consider the set of countries in the world as an example. You must plan to incorporate such changes while avoiding errors caused by the exchange new, changed, or removed values. The harmonizing of reference data can be more involved and impactful than one might think, so be sure to consider it in your planning. Note that if you use a standard interface definition, many of these reference data problems are solved ahead of time.
4. Monitor Your Hub’s Health
One of the important things to understand about an interface hub is that it has multiple sources of error — user error, bugs in applications consuming your hub’s services, network issues, problems in back-end data sources, and your hub itself.
Users will generally not be concerned with the cause of the problem — they’ll just know that the system is not working for them. Therefore, it is critical to monitor and detect problems before they affect users, then isolate the root cause and make sure it gets fixed. Not only does it make your users happier, it also makes your job less stressful, and increases the acceptance and adoption of your interface hub.
Keep an Eye on the Prize — While You Manage the Details
There’s no avoiding the need for agencies to share data with allied partners. The real question is: how can you do so most effectively? In many cases, a well-planned and implemented interface hub provides an optimum solution. Depending on your particular needs and constraints, you could opt for an ESB (including both commercial products and open source solutions), an EIF, or custom development. Regardless of which approach you take, it pays to carefully plan for expected loads, to thoughtfully design your hub’s interface, and to proactively monitor your hub and respond to possible issues.