Integration

Integration represents one of the difficult issues when creating a software architecture. You need to integrate your software in an existing environment and you typically also integrate external components into your software system. Last but not least you often even integrate different home-built parts to form a consistent whole. Thus, integration basically means to plug two or more pieces together where the pieces were built independently and the activity of plugging requires some previous efforts to make the pieces fit together. The provided interface of component A must conform to the required interface of component B.  If these components run remotely, interoperability and integration are close neighbors.

For complex or large systems, especially those distributed across different network nodes, integration becomes one of the core issues.

Integration necessities of external components or environments should be treated as high-priority requirements, while internal integration often is caused by the top-down architecture creation process. Examples for external component integration comprise using a specific UI control, running on a specific operating system, requiring a concrete database, or prescribing the usage of a specific SOA service. Internal integration comes into play when an architect partitions the software systems into different subsystems that eventually need to be integrated with each other.

Unfortunately, integration is a multi-dimensional problem. It is necessary to integrate services, but also to integrate all the heterogeneous document and data formats. It is required to integrate vertically such as connecting the application layer with the enterprise layer and the enterprise layer with the EIS layer. Likewise, we need to integrate horizontally such as connecting different application silos using middleware stacks such as SOAP.  In addition, shallow integration is not always the best solution.  What is shallow integration in this context? Suppose, you have developed a graph structure such as a directory tree. Should you rather create one fat integration interface with complex navigation functionality or better provide many simple integration interfaces for all nodes in the tree?

What about UI integration where you need to embed a control into a layout?

What about process integration where you integrate different workflows such as a clinical workflow that combines HIS functionality (Hospital Information System) with RIS functionality (Radiology Information System)? Or workflows that combine machines with humans? 

What if you have built a management infrastructure for a large scale system and now try to integrate an addition component that wasn't built with this management infrastructure in mind?

Think about data integration where multiple components or applications need to access the same data such as accessing a common RDBMS. How should you structure the data to meet their requirements? And how should you combine data from different sources to glue data pieces to a complete transfer object?

Semantic integration is another dimension where semantic information is used to drive the integration such as automatically generating adapter interfaces for matching the service consumer with the service provider.

And finally, what about all these NFRs (non-functional requirements)? Suppose, you have built a totally secure system and now need to connect with an external component?

It is needless to say that integration is not as simple as it often appears in the beginning. Even worse, integration is often not addressed with the necessary intensity early in the project. Late integration might require refactoring activities and sometimes even complete reengineering if there are no built-in means for supporting "deferred" integration - think of  plug-in-architectures as a good example for this.

Architects should explicitly address integration issues from day 1. As already mentioned, I recommend to place integration issues as high-priority requirements into the backlog. Use cases and sequence diagrams help determining functional integration (the interfaces) as well as document/data transformation necessities (the parameters within these interfaces). During the later cycles in architecture development, operational issues can also be targeted with respect to the integrated parts. Last but not least, architects and developers need to decide which integration technologies (SOA, EAI, middleware, DBMS, ...) are appropriate for their problem context. Integration is an aspect crosscutting through all your system!

Agile processes consider integration as a first-class citizen. In a paradigm that promotes piecemeal growth, continuous integration becomes the natural metaphor. From my perspective, all software developments with high integration efforts must inevitably fail when not leveraging an agile approach.

To integrate is agile!