Hitchhiker's Guide to Software Architecture and Everything Else - by Michael Stal

Trapped

2012-01-07T14:45:00.001+01:00

DSLs are currently being promoted by a large number of activists. If you ever read the seminal Pragmatic Programmers book, you'll also find such a recommendation there. So, will the software engineering universe soon turn into a paradise?Let me tell you two examples from industrial practice:Once upon a time, in the logistics domain, a smart and lazy software engineer introduced a new language just for himself, in a very ad-hoc manner. When they saw the result, all his colleagues got very excited about the concept and started extending the language for their own purposes. The language inevitably grew and grew. Soon, an increasing number of system parts were depending on the language. Unortunately, when our smart inventor reached a high age, he had to leave and let his colleagues alone. But now, there was no one left in the company who really understood the language concept and its realization. It is claimed that the language keeps still growing. Other theories assume the system had to be completely rewritten in the meantime.In another universe and another time, another software engineer was convincing his fellows to introduce DSLs. All the IT dwarfs went crazy inventing their own languages. Yes, they even made competitions who could come up with the most fasinating or useful language. After a while, the DWARF software system was nothing but an ocean of languages. Only wizards were able to generate a working solution. When a furious dragon (= customer) attacked their town, everything imploded. No one has a clue where the remains of the dwarf town are located.What can we learn from these failures?DSL design is (like) architecture design. An ad-hoc language will lead to design erosion, and cause more harm than good in the long run. DSLs represent strategic design tools that should only be in the hands of experienced designers.There are purposes for which DSLs are a perfect fit. But there are also circumstances where they shouldn't be applied. Overdoing it increases accidental complexity.It is like component-based re-use. Each DSL should only have one responsibility.Plan to grow the DSLs systematically, refactor it and verify its correctness. Communicate with all stakeholders that are affected by the DSL.Consider the design choices for the language syntax. Should it be a text-based language, a graphical language, or both? Mind the XML hell - long, long ago almost everyone was striving for XML based languages. But in some rare contexts XML made writing configurations and documents ineffective, uncomprehensible, and tedious. I've seen systems wirh thousands of XML files.Let experts build and grow languages.It is amazing how (often) our discipline gets addicted to all kinds of panacea. Once you are lost in the buzzword jungle, all systems and stakeholders will suffer fom DSL paranoia.DSLs are an awesome means for boosting productivity. However, used by the wrong persons or with the wrong motivations, it is easy to shoot yourself in the foot.

Architecture Governance

2011-10-12T23:22:00.001+02:00

In a large project a product line is developed that supports multiple smart phones. The platform development team puts all code in its super magical configuration management system. Whenever one of the phone teams starts with a product development, it extracts the platform code from the CM system, and adapts it to its own needs. With other words, even the core assets (i.e., the platform itself) are modified. Of course, the platform team is not aware of all these activities. C'est la vie.

This is an example for reuse, albeit one which rather serves as a war story. How come? Imagine, the next version of the mobile platform is going to be developed. How can the domain engineers ever evolve a system that was modified by various product development groups in different ways? The product line has been literally transformed in a bunch of one-off applications by unsystematic reuse. One approach to prevent such architectural war stories is what we call Architecture Governance.

Architecture Governance is a systematic approach for managing architectures and controlling all modifications in order to ensure quality and sustainability. This holds for all modifications, those for developing the system and those for evolving it.

But who is in control? On one hand, there should be an architecture control board in charge of the technical architecture, and on the other hand, another control board should be in charge of the business and business strategy - Luke Hohmann once coined the terms "tarchitecture" and "marchitecture" in this context.

It s important to think about governance in general. Architecture governance is no island. It must be balanced with IT Governance, SOA Governance, and other classes of governance. Governance is about preferring control and monitoring over trust.

WTF does "control" mean? In the product line example, it means we need to introduce an owner of the platform. And we need someone responsible for business & strategy. Only these two "persons" are allowed to decide about modifications of the platform in terms of business and architectural sustainability. Of course, the business control board is supervising the architecture control board. Eventually, it is the business that drives the development.

For architecture control, guiding principles should provide policies for modification and evolution. Obviously, the internal and external quality of the system as well as its expected behavior must never be compromised. That's why we need tools to check and enforce policies, tools to assess the architecture, and test suites to obtain respective information. This is where monitoring comes in. By the way, "tools" in this context also include reviews. And mind the gardening activities! How can a system be systematically prepared for modification. This is where reengineering and in particular refactoring become important.

But there are even more details to bother about:

We need to address legal and regulatory issues. Any change must not violate such standards. Think of safety features for medical products as an example.

It is important to care about quality of service. Assume, a modification leads to a breakdown of KPIs or SLAs.

Don't ignore or neglect quality attributes in general. Does a modification influence sensitivity or tradeoff points? Will it introduce new risks?

An issue might also be patent scanning. Is there any new code introduced such as an Open Source Software that violates intellectual property rights?

But it is not only about tools. It is also about a governance process as well as assignment of responsibilities to roles and persons. For example: Who is allowed to change, configure, update, or add what and when. What measures are necessary in order to guarantee quality and sustainability. How does information flow between the different actors. What happens in the case of policy violations.

Note: This does not only hold for product lines but also for one-off applications. Unsystematic modification almost always leads to potential problems, because of unwanted side affects, accidental complexity, and lack of transparency.

What does this imply for the design of new architectures? It is the same issue: we need a business owner, an architecture owner, a design and implementation strategy, test-driven development, refactoring, and so forth. I happen to have introduced all these topics already in my blog :-) Modifying and evolving a system is only a special case of designing it.

By the way: An agile development process supports architecture governance in that its iterative incremental approach already introduces control and monitoring points.

In practice, there are many ways to deal architecturally with architecture governance. I cannot describe all of them for the sake of brevity. But let me give you some examples:

Using the Layers patterns in a strict sense helps to protect subsystems. Think about Safety segregation.

Clean Coding represents a good way to make control and monitoring easier.

Well documented software architectures are easier to modify and control.

There are many architectural means to foster governance.

Keep in mind that architecture governance is not about controlling architects or developers. Instead, governance helps architects and developers to keep in control. It requires additional efforts but its RoI is very high. Ignoring governance leads to project failure, overcomplex systems, buggy implementations, and design erosion. And eventually to dissatisfied customers and users (and dissatisfied architects and developers). Don't let the software govern you. Govern the software!

Steve Jobs, 1955 - 2011

2011-10-06T23:02:00.001+02:00

Born in 63, I had the great opportunity to grow up with personal computers. Even around the Eighties Woz and Steve Jobs were already legends for their visionary minds and creations. I liked the first LISAs and Macs but unfortunately could't afford to buy one.

When Steve had to leave Apple in 1985 he created the NeXT which has been an extraordinarily successful machine, not so much from a business perspective but from a creative one.

But his popularity exploded eventually after Steve Jobs rejoined Apple as CEO. Steve Jobs stands for creativity, vision, courage, leadership, charisma. He honestly lived and worked literally day and night for Apple and its users. In contrast to many competitors he gave products not only functionality but personality and style.

What only few people know is that Steve also had a strong influence on software. Not only on UI topics but also on programming languages, operating systems, and many other aspects.

I personally have never been an Apple "fan boy" before I bought my first iPod, but now I love their products. However, I must confess that other mothers also have beautiful daughters. Nonetheless it is evident that Apple has always been the driver of amazing innovations that were frequently copied by competitors.

That's the reason, why I want to thank you, Steve. Many things we take for granted simply wouldn't exist without you. I fell deep respect for you and your work. I am sure, you'll always be unforgotten as one of the leading personalities in the IT Hall of Fame. And I really hope, your spirit will always persist within Apple.

The last thing I learned from you, although this has been a sad and bitter experience is: Carpe Diem! And that we should see the people behind and in front of all these nice products.

Send my greetings also to Doug Adams. So long, and thank you for all the apples!

Using War Stories

2011-09-20T17:38:00.001+02:00

For the education of software architects we are using war stories to emphasize important learnings. The whole curriculum is based on the mantra of learning from failure. Errare humanum est! Thus, it is important to see what can go wrong and how to deal with it in a better way. Everyone of us architects knows a whole bunch of war stories from the own career. I also caused failures, but learned from them. It is basically the same like children when they learn to walk. They'll fail but keep on trying until they eventually succeed. It is not a shame to think about own failures. Some cultures force people to always appear perfect which is a bad fundament. As we all encounter failure, it is better to learn what exactly went wrong and why instead of just hiding it from others and ourselves. When we educate architects we do not just teach them theory but also and even more practice. As Einstein once said, "in theory, theory and practice are the same. In practice they aren't."

Fractal Design

2011-09-16T22:13:00.001+02:00

If you are going to design a system, you basically need to identify prioritized main use cases and quality attribute scenarios among other forces to design the system. Together with a problem domain model that shows the inner constituents, and a context model that integrates the system into its environment, the design can evolve in a evolutionary way. As a result of the design activities, architects are able to introduce subsystems as well as their relations according to functional or qualitative drivers. And, of course, interfaces start to appear, each of them defining one specific and explicit role of the subsystem.

A subsystem is itself integrated into an environment – the enclosing system under development. So the subsystem can also act as a system. Thus the same principles apply for the subsystem acting as a system itself – you may even define use cases and quality scenarios for the subsystem with use cases and scenarios being derived as a subset of the “outer” use cases and scenarios. After this step, “subsubsystems” are created, often coined “components”.

What we do here is applying the same design principle in a top-down manner.

But is it useful or possible to apply the principle infinitely? No, because at some level the solution domain is shining through. Solution domains tend to introduce their own composition techniques such as assemblies, bundles, EJBs, services, classes and objects. If the top-down design approach reaches this level, designers must map the architecture artifacts to the solution domain. Maybe, we should call this level architectural twilight zone or the problem-solution-boundary

Side remark: To overcome the twilight zone, we could introduce DSLs and use Model-Driven Software Development.

As a rule of thumb, we typically obtain 2 sublevels (subsystems, subsubsystems) as architects. If less, the design is too abstract and vague. If more, we are introducing too many details.

One of the core challenges in this context is the fact that there might be different strategies to view a domain and thus different ways to cut a system into subsystems.

Partitioning a system into subsystems independent of the hierarchy level is influenced by functional aspects and the problem domain. Thus, subsystems should introduce meaningful subdomains of the surrounding problem domain. With other words: methods like Domain-Driven Design together with some extensions can help nicely.

No matter what you do, there will always be crosscutting qualities and topics. The same observation can be made when structuring an organization into divisions, departments, groups. Have you ever seen an organization without overlapping units? The introduction of crosscutting concerns may introduce new sub^ⁿsystems, add new interfaces or even change their implementation depending on the invasiveness of the concrete concern. Each concern can be considered a subordinate view mixed into the subsystem respectively its domain.

Architecture design is basically fractal design up to two levels of depth. The priorities of use cases and quality scenarios as well as their properties (strategic versus tactical) define how and in which order the functional model needs to be refined hierarchically by integrating scenario-based views.

Of course, one person’s solution domain can be another person’s problem domain which is why exceptions to the aforementioned rule might apply.

The Telephone Test for Software Architecture

2011-09-08T12:36:00.001+02:00

We all know that a software architecture should reveal two properties among many others for an adequate internal quality:

Simplicity implies that the software architecture only addresses inherent complexity without introducing accidental complexity. Since, there are typically several ways to solve a problem, there is no simplest architecture available. Instead, there rather are solution architectures that follow one specific solution path with a minimal number of artifacts. As some quotes propose, simplicity is achieved if you cannot take something away from your system without failing to meet its specification.

Expressiveness implies that the artifacts of your architecture are easy to understand. That is, artifacts should have expressive names, and each responsibility should be assigned to one artifact. Thus, components with a multitude of responsibilities are often a bad idea such as are responsibilities spread across multiple components. However, it is particularly difficult to achieve the latter goal due to cross-cutting concerns. An additional step to achieve expressiveness is having role-based, explicit interfaces with concrete contracts.

But how can you test simplicity and expressiveness? There is a good low-tech suggestion for doing this: Let a software architect explain the architecture to an engineer not involved in the project, for instance using a phone call. Limit the time to - let's say - 10 minutes. If the other engineer gets a good idea of the architecture, it is an indication that the architecture is simple and expressive. Of course, I am assuming that the software architect is a person good in communication as I expect from architects, anyway.

Some might argue that design metrics could also help in this context. Indeed, metrics provide some insights. But we shouldn't forget that metrics analyze the structure, not the semantics. Thus, they are not capable of deciding about expressiveness.

Location:Eduard-Schmid-Straße,Munich,Germany

Apples and Oranges

2011-09-07T13:39:00.001+02:00

How often do we hear statements like how immature software engineering is compared to other engineering disciplines? Of course, construction building is a very old craft where engineers could collect huge amounts of knowledge, methods, and experiences over thousands of years.

On the other hand, there is a huge gap between traditional engineering disciplines and software engineering:

While other engineering disciplines are focused on specific domains, software engineering is supposed to support countless problem domains.

This is why we came up with technologies that are more general in terms of problem domains such as:

UML
Architecture Definition and Specification Languages
Generic Design Patterns (GoF)
Components and Services

However, there is a huge trap using such approaches. Due to their general and generic nature, they are far away from the realities of the problem domain. Thus, communication between software engineers and non IT-knowledgeable stakeholders does suffer.

It is interesting how many experts still emphasize those general-purpose tools and technologies. So to speak, they are addicted to the One-Size-Fits-All drug.

For instance, look at architecture description languages that introduce general components and connections. The problem is that for a concrete problem domain, such generality simply does not work. It is a difference, whether you are dealing with a medical imaging modality or a VoIP platform. Yes, we can … call all building blocks of such systems components or subsystems. And, yes we can … call all interaction paths connections. But, here we are unifying different concepts under one common umbrella.

What software people often forget is that there are two universes:

the Solution Domain with all its supporting tools and technologies. In this domain, general-purpose approaches as the aforementioned make sense,
AND the Problem Domain.

In the Problem Domain we can not naively leverage concepts like UML, Components or General Design Patterns. Instead, we need to follow a Problem Domain First Approach. Thus, it is necessary to start with concepts of the problem domain. What does that mean in practice?

Introduce DSLs and Domain-Driven-Design to cover the problem domain instead of relying on UML. As a matter of fact we can use the underlying meta-model of UML as a base. It is possible to evolve such a language in an iterative approach.
Think about the basic building blocks not as components and subsystems, but use the terminology of the problem domain. You can define your own problem-specific components, subsystems, or services for this purpose. This is exactly what we did for a very large Enterprise Communications System.
Consider the availability of Analysis Patterns for your problem domain and subdomains. Use them if available.

Building the Software Architecture then consists of mapping the problem domain to the solution domain. How could we do that? I mentioned the Onion Model several times. So steps could be:

Understand the problem domain and build a model of it jointly with domain experts
Leverage use cases to understand what the system under development is supposed to deliver from a black-box view.
Use the problem domain model to map the use cases to the problem domain artifacts introducing further functional aspects.
Stepwise extend the architecture by using strategic quality-attribute scenarios with descending priorities.
Prepare the architecture for tactical requirements by using the tactical quality attribute scenarios with descending priorities.
Do all this just considering three abstraction levels to limit complexity.
Apply architecture patterns to structure the overall system.

Mind the Change! Unlike in many other disciplines, software is considered so soft that it should even support strategic late changes. This is like switching your project goal from creating a coffee machine to creating a power plant. Unfortunately, customers are rarely aware of this problem because they are being constantly told by sales and marketing that changes are no problem. Thus, make customers and maybe even more sales and marketing aware of this fact. And use an agile approach to embrace change. But also have courage to deny changes. This is another reason why the Problem-Domain-First approach is so important. How else could you know what changes or extensions are typical for a problem domain? Think of the tax legislation in your country. It is not helpful to only think of interception and extension hooks or Strategy pattern instantiations in this context.

All general approaches for software engineering have their value and can be used as the underpinnings of your systems. But they should not be used for covering the whole software development from problem domain to solution domain. This is also what we can substantially learn from other engineering disciplines.

Thus, don’t mix apples and oranges. Otherwise, the result may disappoint you.

Emergence

2011-09-04T13:36:00.001+02:00

As explained in my last posting, emergence is a powerful approach for solving some hard problems in a non-deterministic way. It's a kind of magic? Not really, it is more an architecture pattern.

What do we need for implementing emergence?

We require

a set of active agents,
a (set of) communication mechanism(s),
a common goal the agents implicitly or explicitly share,
an environment,
cooperation strategies of agent (optionally),
a method to determine when to stop (optionally).

The agents are active in that they execute some functionality in order to reach a goal or subgoal. They communicate with other agents to transfer information. And they recognize when their work is done or when they should stop.

There is a lot of possible variation here:

Agents might all share the same role (behavior) or they may have different roles.
Agents might be organized in a peer-to-peer fashion or hierarchically.
Agents might share the same goal or have individual or role-based goals.
Control might be distributed across all agents, or only be assigned to one or a multiple of them.
Agents might be preinstantiated at start-up or adapt their number or roles according to environmental conditions or achievement (being born, living, dying).
Agents might be able to give life to new agents. They may also be able to terminate other agents, even themselves.
There even could be some environmental conditions that result in death or birth of agents.
Communication might be one-to-one or one-to-many or abide to any other message exchange pattern.
Communication might be synchronous or asychronous.
Agents might adapt their behavior including communication due to environmental changes or interaction with other agents.
Which implies, the environment might change.
Agents might "move" in their environment.
There could also be diverse neighbor environments.

In a Map-Reduce system we have a controlling agent and subordinate agents. The controlling master might instantiate slaves depending on problem complexity. Goal of the system is to reach a common goal such as finding all specific entities in an environment. The master communicates with the slaves and provides a subgoal to each of them. The slaves solve the subgoal and may act as masters for their subgoal, thus recursively applying the pattern. Slaves send their findings back to their master which then recombines partial results to the common result.

In Leader-Followers, the leader is always the controlling agent. When it receives an information from its environment, it promotes a follower to become the leader, and switches its role to worker. A worker who is done with a job, switches its role into follower and enters the followers list.

All these systems reveal emergent behavior. And this also resembles swarms like ant populations. There are different roles such as queen, warrior, worker, etc. The population's goal is to sustain by ensuring the survival of the population and their offspring as well as giving birth to new populations. When searching and locating food sources, ants use pheromons for communication. They show warrior strategies for fighting enemies. And they always adapt to their environment in terms of food, war, weather conditions. Ants might even adapt distribution of their roles such as creating more warriors when necessary. The whole population appears as a kind of smart creature.

A human might also be considered as a whole consisting of smaller agents (cells) and this is how evolution actually developed complex life forms.

And if you think about it, the Web itself is just an emergent system.

When we are going to build such an emergent system, we need to consider all these aspects defined above such as roles, hierarchies, communication styles, goals, controls, environment, adaptation, cooperation strategies.

One promising way to implement such systems are Actor-based approaches.

If we had a toolkit for emergent system that allows to configure the different variation points, we could play around with the concept. Akka is one of the excellent frameworks that could serve as a base for such a toolkit.

So, eventually emergent systems are developed using emergent design approaches. Until then, there is a large journey ahead of us.

The Whole is more than the Sum of its Parts – Creating Complex Systems from Simple Entities

2011-08-31T12:47:00.001+02:00

The Internet represents an ubiquitous infrastructure that enables complex systems such as clouds, SOA-based systems, e-mail, or Web 2.0 applications. Despite of its overall complexity, the basic ingredients of the Internet are rather "simple" and easy to understand. Think of parts like TCP/IP, HTTP(S), HTML, URIs, DNS, IMAP, SMTP, REST, or RSS, to name just a few.

Another example though not software-related is Lego. Children (and adults :-) can build the most sophisticated systems from very simple Lego bricks. And even in biological systems like ant populations or in physical systems - like the whole universe itself - we can make similar observations.

Infrastructures such as the Internet with high inherent complexity consist of such simple constituents. Why are we still capable of providing new kinds of innovative software applications based on technologies that are more than 20 to 40 years old? What do all of these examples have in common?

Well, they basically combine simple building blocks with a set of interaction-, adaptation-, and composition-mechanisms. Moreover, they allow to compose higher level building blocks (from simpler parts) that support their own abstract set of interaction, adaptation, and composition. Think of Layered Architecture! And mind the fact that the building blocks in this context are not only static entities but (might) also include behavioral aspects making them active objects.

Interestingly, the composition or interaction of parts leads to cross-cutting functionality that wouldn't be possible otherwise. Most recently, there have been scientific papers claiming that the human personality is "just" a result of cross-cutting behavior - but that what I'd like to mention as an interesting side-remark.

In general, approaches are summarized under emergent behavior or emergence when we speak about (inter)active parts. In this context, I am considering the term "emergent" as a broader concept also including passive ingredients.

But why should we care? What can we software engineers learn from the principle of emergence, evolution or composition?

It is possible to build very complex systems based on simple building blocks. That is very obvious because eventually all physical systems are composed from simple elements. However, we need to address the challenge, how, why and when to apply such composition techniques to our own software. And we also need to address the issue of quality.

While it is theoretically possible to build every system using very small, atomic parts, this would lead to unacceptable development efforts, bad maintenance and low expressiveness. Think about using a Turing Machine for implementing a word processor. Hence, the gap between the behavior we are going to provide in resulting applications respectively artifacts and the basic building blocks used for their construction shouldn't be too wide.

On the other hand, we’d like to support the creation of a whole family of applications in a given domain. (If you are knowledgeable about Product Line Engineering, this should sound familiar.)

There are three fundamental possibilities to construct such systems:

Bottom Up, i.e., creating a base of entities that spans a whole domain. We can view the domain as a “language” with the base acting as a kind of “grammar”.
Top-Down: i.e., thinking about what kind of artifacts we’d like to create and trying to figure out a set a set of entities from which these artifacts can be composed via one or more abstraction layers.
Hybrid: maybe, the most pragmatic approach which combines Top-Down and Bottom-Up.

To make things even more complex, the “grammar” could be subject to evolution which might change the “grammar” and/or the language.

Very theoretical, so far. Agreed, so let me show a practical show case.

A prominent example is the Leader-Followers pattern. Roughly speaking, an active leader is listening to an event source. Whenever the leader detects an event, it turns into a worker and actively handles the event, but only after it selected one of the followers to be the next leader. As soon as the “new born” worker has completed its event handling activity, it turns into a follower. And then the whole story might repeat again. This pattern works nicely for applications such as a logging server. Its beauty stems from the fact that active agents comprise a self-organizing system, leading to non-deterministic behavior.

Engineers often prefer centralized approaches with one or more central hubs or mediators. That is, we like to introduce a central control. There must be someone or something in control to achieve a common goal, right?

In fact, this assumption is wrong. In some cases, it is necessary to have a control, but this control does not need to be a central component but can be distributed across several entities. Let me give you a real life example:

Some day, you are visiting a shopping mall. After having a nice time, you are leaving the mall. But you forgot where you left the car. Thus, all members of the family start searching in the parking lot until someone locates the car and notifies everyone else. Normally, at least in some families, this search activity would be rather unorganized. Or each family members would agree to search in a specific area, but no one would prescribe how the search pattern looks like. Nonetheless, all share a common goal and will finally succeed. (Let me just assume, the car hasn’t been stolen :-)

An architecture concept that resembles this situation could be active agents that communicate with each other using a blackboard. Just assume, we are building a naive web crawler. On the blackboard you could write down the link where you’d like to start the web crawling. An agent takes the token (i.e., the link) and starts to search the Web page. On the Web page it finds further links which it writes to the blackboard, each link representing a separate token. Now, all active agents can access the blackboard, grab a token, analyze the corresponding page and write the links they find on the blackboard. But, wait a moment, when does the search end? The problem boils down to the question how we handle duplicates (Web sites the system already parsed). For this purpose, we could introduce a hash table where we store pages the agents have already visited. If an agent reads a token which represents an already visited URL, it just throws the token away and tries to grab another one. As soon as there are no more tokens available, the crawling is completed.

Sometimes, we also need (single or multiple) controls. For instance, in most Peer-to-Peer networks, the search for a resource starts using an initial set of controls that aggregate information about their environment. But the search or the registration of resources are conducted in a self-organizing and decentralized way. Note, that the introduction of controls or hubs is not necessary for functional reasons, but helps to increase scalability and performance.

A Sensor (or Robot) Network is an example, where we might even need no control at all. Just consider the case, that we distribute sensors across a whole area to measure environmental data. Even if some of these sensors fail, we are still able to gather sufficient data, given a certain amount of sensors is available. But, of course, one could argue, that there must be someone central who monitors al the sensors. By the way, the network of weather stations is an excellent show case for this strategy.

To summarize this part, in such scenarios with active components, we can have the full range of centralized control, decentralized control, or no control at all. Anyway, the important thing is that each active entity has a predefined goal and there is communication between these entities either using a Peer-to-Peer communication style or one (or more) information hub(s).

As already mentioned, we could add evolutionary elements by letting the agents adapt to changing environments using strategies like survival of the fittest and uncontrolled modification. Genetic algorithms are the best example for this architectural style. Like in neuronal networks, the main challenge for a software engineer is the fact that such systems mostly reveal their functionality using cross cutting and non-deterministic behavior. This makes it difficult to create such systems in a top-down way. Most of the time, such systems are composed bottom-up leveraging a “configurable” evaluation function. With the evaluation function, systems can measure how good the results are and adapt dynamically until the results reach a required level of quality.

Now, I’d like to move to more “conventional” software architectures. As we know, all software architectures are idiomatic, usually providing a stack of idiomatic layers and composition of idiomatic subsystems and components. (Whatever, “subsystem” and “component” mean in this context :-) Note, that this also holds for patterns and especially for pattern languages.

By “idiomatic” I am referring to the point that architectural artifacts offer a kind of API which define syntax and semantics from their consumers’ perspective. By composing these artifacts, higher level functionality can be addressed using lower level functionality. Artifacts at the same level cooperate to achieve composite behavior. Again, we are introducing entities, composition, and communication. Cross cutting behavior addresses composites that reveal specific functionalities or quality attributes. To this end, even quality attributes could be considered functional: they may imply additional functionality or comprise cross cutting behavior.

With other words, in software engineering we face the challenge of hierarchies and composition respectively integration of languages.

The more complex these languages are, the more complex it is to use them.
The more complex the compositions of languages are, the more complex it is to integrate them into a whole.

Thus, we need to balance between complexity of these languages and the complexity of their composition. This balance of forces can be achieved by building an appropriate hierarchy of he languages.

Thus, yet another definition of “Software Architecture” could be:

Software architecture is about the hierarchical integration of artifacts that balances the idiomatic complexity of its constituents and the complexity of their composition. In order to meet the given forces, it addresses strategic design decisions using different viewpoints, prepares mandatory tactical aspects, and abides to common guiding principles.

The secret of good design is appropriate partitioning in core parts which are then composed to build a whole such that it can easily address inherent complexity, without introducing accidental complexity. In a platform or product line, this can be really challenging, because we must enable application engineering to create each member of the product family by using the common parts, while allowing to deal with variability. As we are dealing with hierarchies of artifacts, this turns out to be even more complex. When extending the scope of a product line, there might be implications on all aspects of the reference architecture.

The goal of architecture design should be to design complex functionality by composing simple artifacts in a simple way. The tradeoff is between complexity of artifacts and complexity of composition as well as finding an appropriate hierarchical composition. All approaches such as AOP, model-driven design, Component-Based Development, or SOA try to achieve this nirvana. However, introducing basic artifacts and composition techniques is beneficial not sufficient. Instead, it is crucial to cover the problem domain and its mapping to the solution domain.

This is what all the well-known definitions of Software Architecture typically forget to mention :-)

Make it explicit

2011-08-25T16:29:00.001+02:00

In development projects one of the main challenges comprises clarifying the requirements and assumptions. Knowledge about the problem (and solution) domains proves to be critical in this context. How else would a software engineer understand the requirements? I claim that one of the most critical problems is implicit knowledge which is knowledge taken for granted by some stakeholders, e.g., the customers, but not known to others.

An appropriate approach to address at least parts of this issue is to conduct a KANO analysis where engineers will define features all customers would expect, but also excitement features that may surprise the users in a positive way. Take the iPhone as an example: Expected features include the ability to make phone calls, to maintain a contact list, or to send SMS messages. If a product lacks one of these, customers will be disappointed, but even the coverage of all expected features won't make a customer excited. Excitement features of the iPhone include the Touch Screen and the possibility to add functionality using apps. It is important to mention that today's excitement features will become expected features in the future. Obviously, developers of a mobile phone need to know the expected features in order to prevent any disappointment. However, in many cases these expected features are only known implicitly which works fine if you are familiar with the problem domain, but leads to trouble if this is not the case. Thus, the first conclusion is to make these requirements explicit. The introduction of a problem domain model and of a feature tree prove to be excellent ways to understand the domain and the features as well as their dependence and interaction.

So far I mentioned functional aspects, but implicit assumptions about quality attributes should not be neglected either. Suppose, it would take minutes to find a person in the contact list or to initiate a phone call. Such cross-cutting concerns may not appear in the requirements specification, but nonetheless are essential for successful development. Thus, also the quality attributes must be made explicit which does not only turn out to be important for design but also for the establishment of a test strategy and test plans.

Besides requirements the business strategy and business case often make implicit assumptions such as the kinds of users or market segments or features or future evolution. Whenever architects are not informed about these assumptions, they can neither check their feasibility or validity nor can they provide the right solution. How could someone provide a solution for a vaguely understood problem anyway? Conclusion: always make sure that you obtain sufficient information from management. And mind the ambiguity problem! For instance, I once asked several product managers about their usability requirement. It turned out that every single one of them had a completely different understanding of usability. This is the reason, why a common model and effective communication are so important.

Another problem area is the mapping from the problem domain to the solution domain. Developers may use particular technologies, because they find these the right ones for addressing the problem. Or managers may prescribe technologies without knowing their implications. Architects may enforce decisions without giving a rationale. And developers may make implicit but wrong assumptions about the architecture.Thus, making decisions based on explicit assumptions is inevitable for successful design. For example, architects and developers should verify assumptions and document decisions explicitly. If you don't believe this, think about gifts you gave other persons based on your assumptions.

A benefit of explicit information is that it can be cross-checked and allows traceability. In a development project all the groups of stakeholders should not consider themselves as disconnected islands. Instead, they should share knowledge by using effective communication strategies. Also, they should make their decision process and decisions transparent to the other groups. Basically, it is like having one common information repository where stakeholders may have different views. But these different views must map consistently to each other. This is the reason why architects are the right persons to enforce making everything explicit. In particular, the development of platforms and product lines is very sensitive to implicit information, because failure to consider this information has an impact on many applications.

- Posted using BlogPress from my iPad

SEACON 2011 Architecture Day in Hamburg

2011-04-21T12:25:00.001+02:00

This year the German event SEACON 2011 is taking place from 27th to 29th June. The venue is the Atlantik-Hotel in Hamburg, a very nice location famous for its permanent guest, German Pop/Rock Tycoon Udo Lindenberg.

I was responsible to organize the architecture day on 29th June. And we managed to get excellent speakers covering a lot of interesting architecture topics. Jim Webber from Thoughtworks is going to give the keynote address: “Lessons learned in large HTTP-centric Systems”. Attendees will also have the opportunity to visit half-day tutorials on Dynamic Software Analysis (Johannes Siedersleben) and Software Quality (Erik Dörnenburg). In addition, the agenda is filled with excellent talks by renowned speakers and practitioners.

For software architects this will be a highlight in 2011. The event SEACON Architecture Day might become an established event on Software Architecture. It is up to you. Attend and enjoy!

The hard way to Distribution and Concurrency

2011-02-05T00:17:00.001+01:00

The Web is hot. So is Cloud Computing. And Multicores might even become hotter. Whenever there are additional capabilities, software developers will immediately start to exploit them. Why else do we always require new hardware for new operating system versions?

In particular, dealing with distribution and concurrency is pretty simple. Just start a thread in Java or .NET, and the force will be instantaneously with you. Or click on some menu items of your favorite tools to distribute functionality elegantly across the network.

Unfortunately, many projects fail in providing distributed or concurrent functionality. How comes all these nice programming platforms and libraries don’t suffice?

And the answer of course is software architecture. Especially for cross-cutting concerns such as distribution and concurrency the programming level offers the wrong abstraction layer. It is not sufficient to address concurrency programmatically. Foremost, we must address the architectural layer, then the design aspects before thinking about programming idioms.

Let me investigate concurrency as an example. For this purpose I’ll consider different domains with respect to concurrency:

In home automation systems there is a large variability of different services with various durations. Think about switching lights on/off, or controlling the climate system. Typical home automation systems are reactive systems that use a central component for event notification and event handling. As many events might occur in parallel, the Concurrent Reactor Pattern offers the right architectural approach.
For an image processing system we need a completely different architecture. Here, image data is created, processed and transmitted between subsequent components. Hence, the Pipes & Filters architecture pattern with concurrent filters fits nicely into the domain.
When dealing with control systems or Web Servers with asynchronous events arriving all the time that must be handled in a synchronous way, a Half Sync / Half Async architecture provides a message queue which decouples the synchronous from the asynchronous world.
In Servers that process single messages such as Log Servers, a Leader/Followers patterns applies perfectly. The leader is in charge of receiving the next event. If it receives an event, it promotes one thread in the followers chain to the next leader, while becoming itself a worker that after work completion will become a follower. A nice example for a self-organizing infrastructure.
In a warehouse system we’d like various actors to store or load items at the same time. Thus, the warehouse core cannot only provide one single interface. Instead, multiple interfaces operate on the warehouse core logic. They can synchronize their data by using a Half Object plus Protocol pattern.
In a mobile network or an AI system multiple, concurrent agents may need to coordinate their activities to achieve a common goal. For such systems, patterns like the Blackboard pattern are applicable.
For divide-and-conquer processing the Master/Slave pattern helps structuring our Map/Reduce problem to sets of interacting components, some of them responsible for the Map, most of them responsible for the Reduce functionality.

We got various example domains, all of which had to use a different concurrency architecture approach due to their diversity of forces such as existence of sessions, variety of services, synchronization necessities, and many others.

Our first conclusion is: different problem contexts require different concurrency architectures. There can’t be a single concurrency architecture for all problems. In addition, we need to deal with architectural issues before introducing finer grained design aspects.

For the next step of refinement it is essential to solve issues at the design level. Think about Half Sync / Half Async where we need to determine how to organize the synchronous worker threads of the Half Sync layer and how to let them access the shared queue of incoming events in a coordinated way, to address only a few aspects.

Therefore, we need to ask questions like:

How will we manage shared resources such as threads or connections? Thread pools and other design tactics come into my mind.
How do we efficiently deal with sharing and locking of data? We might use monitor objects or Software Transactional Memory approaches.
How can concurrent entities communicate with each other and coordinate each other? Actor-based programming patterns are one option here.
How can we efficiently access data? Think about caching, eager evaluation and lazy evaluation?
Is there a way to efficiently handle I/O? Asynchronous processing such as Proactors, Active Objects, or Asynchronous Completion Tokens come to our rescue.

Basically, all of those design aspects are on the component level. Fortunately, there are many design patterns that help us in implementing this abstraction level.

Our second conclusion: whenever we conduct fine grained design (i.e, tactical design), design patterns and design tactics offer guidance.

Only in the final step, we are going to leverage all programming language and library features. Eventually, we end up using methods for thread creation and management, mutexes, agents, semaphores, and all those other idiomatic features modern programming platforms provide.

In the best case, a DSL and/or a library could help increasing the abstraction layer of the implementation by hiding details of the underlying hardware system. Think about OpenMP, PLINQ, Akka STM, and many other solutions.

What we learn from this example is that cross-cutting concerns like infrastructure or quality attributes require a sound fundamental architectural approach which is refined using design tactics and design patterns, and then implemented with the available tools offered by the programming platform. For instance, creating a concurrent solution requires an architecture-first instead of an implementation-first approach. For the same reason, SOA approaches favor interface-first over implementation-first styles.

Sure, programming might be much more enjoyable and exciting than drawing architectural boxes and lines, but at the end ad-hoc implementation of cross-cutting concerns causes more harm than fun. And don’t forget: creating a sound software architecture is sexy – at least for software engineers.

Where we can meet

2011-01-23T16:35:00.001+01:00

I will participate in the OOP Conference 2011 in Munich which starts on Monday, the 24th January.

Tuesday morning I will talk on Design Tactics
Tuesday evening I will present Actor-based Programming
Wednesday evening my Scala tutorial will take place
Tuesday and Thursday I will also have the Meet-the-Editor events each 30 minutes where you might talk to me in my role as editor-in-chief of JavaSPEKTRUM
Tuesday evening I will act in the IT Stammtisch organized by the unique and nice Nikolai Josuttis. So, if you would like to see me very uncomfortable, you should definitely attend this panel show.

I will also attend the QCon 2011 in London (7th to 11th March) where I will

have a tutorial on functional programming with Scala and F# on monday
host a track on Software Architecture Improvement at wednesday (talking myself in the introductory session)
present the Onion model in Floyd’s track about models

Hope to meet you!

The Beauty and Quality of Software

2011-01-16T00:46:00.001+01:00

In my spare time I often enjoy digital photography. Capturing people, streets, buildings, wildlife or landscapes involves a lot of fun and is a relaxing experience. After taking photos and when processing images with Adobe Lightroom, I am used to differentiate those pictures I consider decent from those I am going to throw away. The obvious question for a photographer is whether there are some guidelines or indicators that help taking better photos and also help rating photos. And, indeed, there actually are such quality indicators.

Independent of the content of a picture, there exist some common properties I consider important when rating a photo e.g.,

Does the photo clearly focus on one specific object or does it rather confuse by showing various things without any focus?
Is the horizon depicted horizontally? Even if there is only 1 degree deviation the human watcher won’t feel comfortable.
Are there some interesting kinds of symmetries in the image?
Does the photo contain a chaos of many different things or does it concentrate on one specific detail?

There are many other indicators for the internal quality such as the rule of thirds. Interestingly these rules are really helpful. However, they are not proofs for good or bad quality of a photo, but mere indicators. For example, sometimes deliberately breaking symmetry or violating the rule of thirds may lead to much more expressive photos. No rule without exception.

How does all of that relate to software architecture? The question is whether we can apply some indicators to get a first impression about the internal quality of an architecture, something I consider as architectural beauty. When I once read Dave Cutler’s book on the design of Windows NT, it really opened my eyes. All parts of the architectural design were easy to understand. The architecture was expressed in a kind of idiomatic way, where the same principles have been applied to different parts. The partitioning of the overall operating system in different layers and components with clear responsibilities helped me grasping the details easily. On the other hand, I have also experienced bad architectures with over-generic designs, where it took me weeks to get the slightest clue what they were trying to implement. So are there quality indicators for good or bad design?

Obviously, there must be such indicators. Otherwise, the widespread addiction to metrics wouldn’t make too much sense. Metrics combined with CQM tools (CQM = Code Quality Management) and Architecture Analysis tools can reveal issues such as insufficient cohesion or coupling. Metrics however must always be considered relative! For instance, there is no rules of thumb whether 5k LOCs are good or bad. It might be good in one case, but bad in another. Applying the McCabe Cyclomatic Complexity can lead to wrong conclusions (If you don’t believe me, calculate the cyclomatic complexity of an Observer scenario with 80 observers). Why is that? Metrics operate on syntactic level. Thus, they can only measure syntactic properties. They are not capable of dealing with semantics. Hence, all metrics must be set into the right context by human engineers. While a coupling of 5 itself might not be too valuable, the increase of the coupling from 5 to 10 after one iteration reveals a potential problem.

One of the more absolute indicators are architecture smells which also help figuring out the necessity of architecture refactoring. Let me give you a few examples:

dependency cycles between architectural components imply that you cannot understand, test, change one component without addressing the other component in the cycle.
Inexpressive component names prevent engineers to understand the architecture without digging deeper into more details.
component responsibility overload means that a component is implementing too many different responsibilities preventing clear separation of concerns.
Unnecessary indirection layers do not only negatively affect developmental qualities such as maintainability or extensibility but also operational quality attributes such as performance.
Implicit dependencies often lead to a shift between desired architecture and implemented architecture. One notable example is violating strict layering, thus introducing unnecessary und unknown dependencies.
Over-generic design: When there are dozens of Strategy patterns (or proxies, visitors, etc.) in your design, this often denotes a clue for a potential problem lurking in your design. A Strategy pattern basically means “I have no clue but want to defer the decision to another place”. Applying many Strategy patterns mean, the architects had absolutely no clue. Instead of opening the system according to the Open/Close principle for some variability points, they applied Strategy unsystematically. This is the best way to achieve less flexibility and less performance.

Architecture smells can often be detected using architecture analysis (tools). If we think about these smells, we recognize that there must be something more that gives us indication about the internal quality/beauty of an architecture. Something which resembles which I addressed in the introduction on digital photography.

I can not speak for the whole community of software architects in general. However, here are the quality indicators that help me assessing a software architecture:

Simplicity: A software architecture is simple when it addresses the requirements with the least possible number of design artifacts (KiSS) and does not introduce accidental complexity by additional entities (aka design pearls) that are not required. There is the old rule: “an architecture is simple when you cannot remove anything without failing to meet some requirements”. Hence, a simple architecture is not a simplistic architecture. To achieve simplicity, one guideline is to root all architecture decisions in the architecturally relevant requirements. You can measure simplicity of an architecture using a simple test: ask the architect to introduce the architecture within at most 30 minutes only using a flip chart to draw the architectural baseline. I know, this also depends on the presentation skills of the architect but that’s another story. Nonetheless, a listener should be able to repeat the core architectural decisions. There is one caveat: Some “smart” software developers try getting rid of complexity by just hiding it in the guts of the implementation. For example, a system could theoretically consist of one single object with a doIt method. This is like some of the ESB and EAI products that just hide the mess behind some wrappers. This, however, does not introduce simplicity, but rather moves complexity to lower layers. The same simplicity test would definitely reveal that kind of bad trick.
Expressiveness: An architecture is expressive when you can easily grasp the purpose of all entities and the whole architecture by looking at the architecture baseline. I don’t claim an architecture needs to be self-explanatory such as understanding it by just looking at the UML diagrams. This can’t work because diagrams fail to reveal leading design principles and the rationale of design decisions. What I mean: Just looking at the architecture design should give you enough understanding to grasp the whole architecture vision by diving into some additional details. How can you achieve expressiveness? First of all, give expressive names to all entities, so that another engineer gets the idea. For all components make sure: It should do one thing but it should do it right. And each component should only have one responsibility. Assigning lots of responsibilities to the same component leads to inexpressive architecture as do all responsibilities crosscutting through your design. Role-based design is a perfect tool to assign role-specific interfaces to components. Speaking about interfaces: Also add a contract to each interface that specified what (kind of protocol) the interface expects and what it provides. Aspect-oriented development may help dealing with crosscutting concerns. Another important issue in this context is to make all dependencies explicit in the architecture. That’s a no-brainer because implicit dependencies are simply not visible in the design so that an architecture reviewer won’t recognize them easily. Expressiveness can be verified by a simple telephone test. Ask an architect to explain his/her architecture via a 10 min telephone call. After the call you should be able to understand at least the coarse grained architecture.
Behavioral Symmetry: Suppose, you got are viewing sequence diagrams of a transaction-based enterprise system. Although there is a method invocation called beginTransaction, you’ll recognize no endTransaction, commit or rollback. Won’t you feel uncomfortable with such an observation? Symmetry of behavior is a good indication, something might be wrong with the design. As it holds for all quality indicators, it is not a proof for bad quality. For example, in a distributed system the remote server might allocate memory for a new result object, while the client is supposed to free it after use.
Orthogonality or Structural Symmetry: I once had to review a software architecture where they used several solutions for the same problem. Hence, I required to understand all these various solutions to understand the architecture. Structural Symmetry means, you are leveraging the same solution for all instances of the same problem. Even if MFC provided a handful of string classes, you should not use all of them in the same application. This is particularly essential when dealing with crosscutting concerns such as error handling, tracing, logging. Imagine, each developer would introduce his/her own error management strategy. In this case you need to cope with broken structural symmetry. To avoid structural asymmetry, you may introduce guidelines and conventions for crosscutting concerns or recurring problems. It is not sufficient to provide documents, though, but you should also enforce these guidelines actively in the project. When reviewing software architecture, ask for such guidelines and ask how they have been enforced.
Pattern Density: We all have heard about the “not invented here” syndrome and may even have experienced it ourselves. Patterns capture the knowledge of experts how to solve recurring problems. Thus, a wise engineer would rather apply a pattern instead of coming up with a homegrown solution. The more patterns have been used, the better. Wait a second! That is not quite true. In this context, by using patterns I mean using them whenever they solve a problem you really have. Not like in the “hammer-and-nail” syndrome with an attitude like the following one: even I got no need for Observer in my system, I will change the system so that I can apply the only pattern I know. Applying patterns is not trivial. Patterns are not Lego building blocks you can add to your architecture. Rather, you need to adapt your design, map the pattern roles to your architecture and integrate all the patterns properly. In some hotspot areas of an architecture design, the same components may contribute to several patterns. Thus, even high pattern density could be malicious if the patterns have not been integrated properly.
Emergence: In an ant colony all individual elements reveal very “simple” behavior, but the whole colony seems to act smartly. Another example for the whole being more than the sum of its parts (Aristoteles). The Internet and the Web are also good examples for this kind of emergence. They consist of simple elements such as Ethernet, TCP/IP, DNS, HTTP, SMTP, RSS which serve as the building blocks for a complex ecosystem. Emergence also means decentralization and self-organization. Many architects are completely addicted to centralization. Often when quality attributes such as scalability, availability or fault-tolerance have high priority such as in Cloud or P2P Computing, decentralized approaches are much more effective. Think of the Leader/Followers patterns as an example which introduces a self-organizing pool of threads that reduces resource competition by only allowing the leader to access the joint event source. Another indirect consequence of emergence is that you shouldn’t strive for fat APIs that implement all methods, but just simple APIs providing methods with which you can easily implement more advanced functionality.
Partitioning and Spacing: The responsibilities in your design should be mapped systematically to components and subsystems. For example, subsystems should only contain functionality that reveals some semantic coherence. That is, the closer two different kinds of functionality are semantically related, the closer should they be aggregated in the design. Layering is a good way for separating different levels of abstractions with top layers being more abstract (close to the problem domain) and low layers being less abstract (close to the solution domain). Basically, you need some kind of systematic top-down design and problem decomposition so that the resulting subsystems and components will offer an adequate partitioning of responsibility and a sufficient spacing (different kinds of concerns should be mapped to different subsystems, layers, components). In addition, architecture entities should offer a clean partitioning into role-based interfaces instead of relying on one bloated interface per component or subsystem. An example could be an additional interface that enhances the testability of the system. Or an interface for configuring a component. If possible commonly used interfaces such as configuration interfaces, should be defined uniformly for the whole system. Otherwise, configuration would need to deal with many different configuration approaches (see the Component Configurator Pattern for more details).

There are definitely some more qualities I could add to this list of indicators. However, in my experience these aforementioned quality indicators already offer you an excellent mental tool for assessing the internal quality and thus the beauty of a software architecture.

Mind the gap: As mentioned several times, each of these quality indicators just serves as an indicator, not as a proof. But if the design under consideration does not provide one of these internal quality properties, the responsible architects should at least offer you a good and convincing rationale why this property is missing.

Needless to say, that quality indicators should not only be leveraged for assessing existing software designs. Likewise, you should consider them valuable when designing a new software architecture.

Qcon 2011 in London: 100 British Pounds discount

2011-01-07T23:39:00.001+01:00

I will host a track on Architecture Improvements at QCon 2011 with a row of excellent speakers. In addition, I will give a talk on Software Architecture Design as well as a tutorial on Functional Programming with Scala and F#.

If you’re interested in attending the conference just use the promo code STAL100 for receiving a 100 BP discount.

Here’s the URL for the QCon.

Small is better - really?

2011-01-03T06:52:00.000+01:00

In different panels and discussions I often hear programming languages such as Scala are not a good idea due to their "language size". Less syntax is better to learn a language which is why, for example, Clojure is superior to Scala. Often, the proponents of this argument point to languages such as C++, C# or Java as negative examples. Resembles software architecture a lot, doesn't it? Honestly, I object to this conclusion. Take languages such as brainf*ckr or the Turing Machine. Very simple syntax indeed, but would you consider them powerful languages? Definitely not! In my viewpoint it is not the size that matters but the noise-to-signal-ratio. If you need to express a solution for a given problem, how much efforts do you need to find and articulate the solution using the language idioms. The more appropriate and concise idioms are available that support your solution, the better. As in software architecture some qualities are important when judging languages:

appropriateness to solve a specific class of problems
simplicity
expressiveness
orthogonality of language structures
approach of least surprise
emergence of features
avoiding implcit dependencies and influences (such as temporaries in C++)
availability of powerful language idioms

By the way, these qualities are also applicable to software architectures. This is no surprise because we are speaking about architecture of programming languages here.

There are several conclusions we could draw from that qualities:

Even if a language offers most of these qualities at the beginning, it might erode over time when it is evolved in an unsystematic or improper way
Several problem classes might imply various languages. So, we have to choice between a best of breed approach for each problem class as a polyglot programmer would prefer, or we could try to indentify some multi-paradigm language. Note, however, that multi-paradigm languages are those which are often very likely to erode, especially when more and more paradigms are added as an afterthought. However, this is a risk, not a predefined fact. For instance, multiple paradigms were cleanly integrated in Scala from day one, while C++ started as a structured language with additional classes.
The language core might be excellent but it won't be helpful if the support libraries and tools do not reveal the same qualities mentioned above. A fool with a tool is still a fool. So, even for the best languages, APIs of libraries need to integrate tightly.
If you follow the Pragmatic Programmers advice to learn a new language every year, you will also be able to extend your knowledge about new solution paradigms and idioms. This will drastically improve your skills as programmer and architect. Mind the gap: it is not sufficient to just learn a language, you also need to practice it for a while.

Unfortunately, using multiple languages at the same time might be a bad idea. I experienced a project where people invented their own language. After a while, only one expert was left who did understand the language which was not that comfortable for the organization. If you plan to use multiple languages, think about the problems at hand. If they can be better solved using multiple languages, go for it, but make sure, several people in your team know these languages. If possible, use languages that run on either the DLR/CLR or the JVM, because then they will share the same SDKs and (often) even the same tools. Make yourself aware that you ARE already using several languages such as HTML5, XML, SQL, DSLs and so forth. We already have become polyglot programmers. But that doesn't mean you should strive for large numbers of languages in your projects.

Personally, I have learned a lot of languages in my career: x86-ASM, Pascal, Modula2, VB, Java, C, C++, Ruby, Lisp, Clojure, Scala, C#, F#, CIP, Haskell, D, Axum, ... All of these languages have their purpose and their strengths, but also their weaknesses and problem areas they cannot address well. Some of them are large and easy to learn such as Scala, while others are small and difficult to learn such as Lisp which is why they invented Scheme :-)

Size does not necessarily matter, the problem you are going to solve does.

Books, books, books and tools

2010-12-23T12:49:00.001+01:00

In vacations I often dedicate some time to reading new interesting books as well as experimenting with programming languages. For a software architect I consider it essential to stay in sync with technology evolution. Honestly, I do not believe in ivory tower architects who just know UML and Powerpoint. So, what are my plans for this christmas season?

One of my favorites is Martin Fowler’s and Rebecca Parson’s book “Domain-Specific Languages”! This book explains DSLs but also educates how to practically leverage internal and external DSLs.
From the Pragmatic Programmers I received “Seven Languages in Seven Weeks” by Bruce A. who provides a hands-on tour on Clojure, Haskell, Io, Prolog, Scala, Erlang, and Ruby. If you are a fan of programming languages, this is definitely a good read.
Of course, I am also enjoying fiction books as well such as Michael Crichton’s “The Great Train Robbery”. I recently started to read Crichton’s novels and really like his work. In addition, I am reading some layman books on the universe and quantum physics. I am currently too lazy to reread Feynman’s Lecture Notes on Physics.
For my preparation for OOP 2011 where I will give three talks, one on actor-based programming, one on Scala, and one on design tactics, I have programmed a lot with Scala 2.8.1, sbt, and Akka. But I am also currently trying to dive more into Clojure. I used Lisp in the university, recovered my knowledge a few years ago, and now starting to get familiar with Clojure. This is where monads come in. Interestingly, there are many definitions and explanations of monads but none of the is really decent. Maybe, I will give a try in the future, and also fail

So many new things to learn in such little time. Unfortunately, the times are over where I could learn the whole JDK or .NET in a weekend. Software engineers have to get more and more focused. It is even impossible to keep yourself up-to-date with enterprise technologies alone, or concurrency techniques, or mobile computing, you name it. Nonetheless, systems are gaining more and more complexity, becoming bigger, using more technologies. Interestingly, we are always testing our limits. No matter how excellent our tools are, we try building systems that are more complex than our skills allow. With other words, technology and complexity are always ahead of us. Software architectures do not only fail to master inherent complexity but become beasts filled with accidental complexity. Our capabilities do not scale with our problems. Did I mention Henri Petroski’s book “Learning from Failure”. The essence of this book is that we always will go over the limits, but the best thing to survive is to learn from failure. This basically resembles what Thomas Kuhn once said about technology evolution and revolution.

Fortunately, it also means a lot of fun to learn new things or technologies, try them out, learn their strengths and weaknesses, increase your mental toolset. It opens the horizon to some new ideas and ways. This is why learning is one of the most important skills for software engineers.

So, what are your plans for the rest of this year?

Understanding software architecture

2010-11-19T17:46:00.001+01:00

While creating software architecture is a very creative activity which I really enjoy, analyzing existing software systems is much harder and happening frequently.

There are several reasons for this observation:

Often, the designed and the implemented architecture differ dramatically. In these cases, the documents are nothing than fairy tales.
In some cases, I am encountering a kind of documentation hell. The information surely can be found somewhere, but it is almost impossible to figure out what could be the right document. I am here to find not to search.
Another “highlight” are insufficient abstraction levels in the documentation. No one can read end even less persons can understand an architecture document that covers 1000 pages. Likewise, the other extreme isn’t helpful. For example, I once received a software architecture document consisting of one page filled with UML diagrams.
What I really dislike are documents that only show the WHAT but not the WHY. However, I definitely need to know the rationale of all those decisions. All design pears that pop up from the void of the universe are worthless to all those mediocre reviewers who must understand them.

So how you could approach such an endeavor.

Firstly, get an idea about the domain, the development organization and the business, that is, the ecosystem, in which the software system lives.
Get an overview of the coarse grained architecture, from the user view and the developer view. I prefer strategic architecture design documents, guidelines, programming conventions, presentation slides, and even marketing material for this purpose.
Get a knowledge about the tools and technologies of the solution domains that have been used.
Interview some designers and developers in order to understand the implemented architecture (and the designed architecture) as well as its weaknesses and strengths. The more documents additionally capture the design essence the better. Both information sources will provide you with a good idea about the architecture.
Use architecture analysis tools to understand the internal quality of the systems (and also its pain points).
Try to change or implement yourself in the codebase of the system to stay close to the ground and thus to reality.
At the end, you should yourself be able to explain the architecture to other persons.

Basically, what you need to do to understand an existing software system is similar to conducting the first phases of an architecture review.

Don’t be shy and ask any questions that come to your mind. Often, these Q&A settings help architects figure out potential problems in their design, and help you understand the architecture. So, it can lead to a Win-Win situation.

Software Architecture Reviews

2010-08-29T11:58:00.001+02:00

Software Architecture Reviews represent one of the most important approaches for introspecting software design. The best way for introducing such assessments is to establish regular evaluation workshops in all iterations. Architects and designers will review the software architecture focusing on internal qualities such as unavailability of dependency cycles and quality attributes such as performance or modifiability. If they find any issues, they will define action items and means for getting rid of the challenges (also known as problems). My own experience shows that this is definitely the recommended way, because regular reviews will detect any issues as early as possible before they cause further harm such as accidental complexity or design erosion. If introspection tools such as Odasa or SotoArc are available, detecting internal quality issues is often surprisingly easy. And if architects compare quality attributes with architecture decisions in early stages, they will reduce the probability that an important nonfunctional requirement was neglected or designed in a wrong way.

Unfortunately, many architecture reviews are initiated very late. This makes problem detection and architecture refactoring much more complex. Nonetheless, such architecture reviews are always a good solution to get rid of architectural problems, especially when the organization could not handle such issues successfully in the development project. These reviews should have a clear scope. Otherwise, the review will take place for large time frames which is not acceptable in most cases. Scoping in this context means to define a main objective of the review as well as 2-3 key areas where potential problems supposedly lurk in the guts of the design. The result of such a "large" review should be a document with key findings, e.g., weaknesses and strength of the architecture. However, it should not only contain the weaknesses but also appropriate solutions for addressing these weaknesses. Some review methods don't consider solutions mandatory in such a report. I definitely do. Even more, I consider ways to get rid of the weaknesses the most important result of a review.

Such a report should reveal the following structure:

Scope of the review and review method
Documents (project documents) and information used (for example: interviews, test plans, source code, demos)
Executive Summary
Overview of the software architecture under review
Findings and Recommended Solutions
Summary

While regular, iterative reviews can be conducted by project members such as architects and developers, larger review projects should be done by "external" reviewers. There are two main reasons for this recommendation: First of all, external reviewers often see more. And secondly, higher management representatives are often inclined to accept external recommendations more than the ones from their own project members.

Interestingly, an architecture review is not constrained to software architecture challenges. It might also reveal problems in the organization, development processes, roles and responsibilities, communication, technologies, tools, business. To be honest, only rarely the results will address design problems only. That is another reason why architecture reviews should have external reviewers.

There is a whole selection of different review methods such as CBAM, SAAM, ATAM documented in literature. We at Siemens have developed our own method called Experience-based Reviews. While the former 3 methods are scenario-based (which I will explain in a later blog post), experience based methods are driven by the experience of the reviewers and are less formalized.

Such a review normally consists of the following phases:

A Kickoff Workshop where reviewers and project stakeholders meet to introduce the review method, illustrate the software architecture, and define the review scope
In the Information Collection phase, all available information is collected by the reviewers such as documents, test plans, source code, demos. Further information is collected by conducting interviews with the project's stakeholders, 1 hour for each and constrained to one interviewee. Information is kept anonymous in order to establish a relationship of trust between reviewers and interviewees.
In the Evaluation phase all information is evaluated. Strengths, weaknesses, opportunities, and risks are defined by the reviewers. So are all the possible solutions for the weaknesses and risks. At the end the reviewers create a review report draft structured as mentioned above. They then send this report to all stakeholders, leverage their feedback, and disseminate the final report.
A Final Workshop helps to summarize the key findings and discuss open issues.

This approach works extremely well if the reviewers are experienced. We normally will use the Master/Apprentice pattern to educate architects how to conduct an experience-based review. Senior reviewers will lead the reviews and teach junior reviewers by training-on-the-job.

All of these approaches define qualitative assessment methods. In addition, architects and developers should also leverage quantitative methods when actually creating design and implementation. Tools for quantitative assessment consist of but are not limited to feasibility prototypes, simulations and metrics.

Architecture assessment methods are a bit like testing or refactoring. Many engineers think they can survive a project without these disciplines. But in all but trivial projects this proves to be a wrong assumption. The more you invest in early and regular testing, refactoring, architecture assessment efforts, the better your RoI will be. Do less in the beginning, pay more in the end!

I have been upgraded

2010-06-18T00:06:00.001+02:00

Because I got so many questions on this: Most recently, my name got a little bit longer. The University of Groningen in the Netherlands (RuG, Rijksuniversiteit Groningen) has appointed me Professor Extraordinarius. Thus, I am now member of the Bernoulli Institute of Mathematics and Natural Sciences where I will give lectures and more.

If you now ask: this is in addition to my main profession at Siemens which by the way supported me in achieving the position. Of course, I am very glad to cooperate with Paris Avgeriou who is a professor himself at the RUG. Another personal friend and promoter, Jan Bosch, is member of the same institute. Needless to say, that Paris and Jan are internationally renowned software architecture experts with whom it is a lot of fun to work together. I am quite impressed by their knowledge and expertise, they also use to distribute to their students.

My appointment was greatly supported by Nicolai Petkov, another professor at RuG, whom I really like and respect. After the deans of all Computer Science Instititutes in other Dutch Universities had agreed to my appointment after checking my C/V and my References I finally received the appointment letter.

The RuG is quite famous for its research on software engineering, in particular in Product Line Engineering. Not to forget all the RuG offspring such as Jan Ommering and a lot of other great Product Line gurus. I do not only enjoy that but also the Dutch mentality and great hospitality whenever I am in Groningen. Only a few weeks ago I’ve been there to give lectures to students as I already mentioned in my last blog post.

Thus, I will be able in the future to support Ph.D students, master students as well as postdocs. They may learn from my experiences, but I will also learn a lot from them. You could call that a Win-Win situation.

So that’s the big news.

Teaching Students in Software Architecture

2010-06-11T10:02:00.002+02:00

This week I have been giving lectures on software architecture for master and PhD students at the Rijksuniversiteit in Groningen (RuG). It worked excellent and it was a lot of fun. I teached them systematic software architecture design, design tactics and introduced a pattern system for distributed and concurrent systems. What worked particularly well was the split into a presentation part and a tutorial part - each lasting 2 hours per day - where participants were designing example projects in groups of up to 5 people. This way, they would not only learn about architecture design, but also about group interaction. In fact, many problems in software architecture are caused by people issues such as lack of (efficient) communication.

They will also get a group assignment and an individual assignment for the time until end of June.The former one will comprise completing the architecture design and providing an architecture description while the individual assigment requires students to create a design essay on an architectural quality (in the context of their example project).

I received positive feedback that I did not introduce just a bunch of unrelated patterns but narrated a pattern story of how to apply patterns for the design of middleware and distributed systems. From my viewpoint, this is the best method to educate students that patterns are not just disconnected islands of code, at least they shouldn't be. Most of the power of patterns comes from using systems of patterns. Pattern languages would be even more attractive, but unfortunately there is only a bunch of them and the ones existing are really complex.

For the lecturer it is important to be constantly available for the exercise groups during the tutorial parts. I got a lot of questions about architecture design, modeling and documentation. It is pretty close to architecture enforcement. You cannot simply throw an (exercise) specification over the fence and expect attendees to exactly understand all issues. Management by walking around is much more effective.

As material for the course I relied on the excellent Software Architecture in Practice book, 2nd edition, by Len Bass, Paul Clements and Rick Kazman and of course - needless to say - on our POSA book series (Pattern-oriented Software Architecture). Using POSA patterns is beneficial in that it shows that the pattern community does not only depend on the GoF pattern book.

In summary, the mixture of conceptual parts and exercise parts works nicely to educate students about software architecture.
Looking forward to providing more lectures in ths future and keeping in touch with the students.

Functional Programming

2010-04-12T11:28:00.001+02:00

In the last months I have intensified my skills in functional programming. My favourite programming languages have been Scala and F# so far. Only a few years ago I recovered my Lisp abilities when I had been ill for some weeks. It is interesting how many functional languages currently evolve. Guess, it is closely related to the availabilty of virtual platforms such as the JVM and the CLR. No need to develop a whole bunch of APIs or IDEs anymore. just use the Java SDK or the .NET Framework Classes, and provide some plug-ins for Visual Studio, Eclipse, NetBeans, or any other IDE of your choice. A real paradise for language developers. Reminds me of the Eighties when all those OO languages appeared such as C++, Smalltalk, Objective C, or Eiffel. I anticipated several years ago that this would happen. Interesting, you might ask, but how is all this related to architecture? A lot if you ask me. Remember the book on object-oriented design patterns by Erich Gamma, Ralph Johnson, Richard Helm and the unforgettable John Vlissides. This book (as well as hopefully our POSA books) have changed the world in that they brought software architecture in people’s minds. The same will happen through functional programming.

Functional programming enforces another way of thinking. Functions are first class entitites that can be assigned to values, passed as arguments to higher-order functions, or just created anonymously (called: closures). In pure functional programming there are no variables anymore, just immutable values. All of this was already specified by Church & all in the Thirties. While the first functional languages such as Lisp, ML were difficult to understand for everyday programmers, new functional languages deviate from the academic approach and turn into hybrid languages that integrate object-oriented features, and permit mutability by supporting imperative programming styles.

Sometimes mutability is important such as for I/O and GUI operations or logging/tracing. But there are many situations where mutability is like shooting yourself in the foot. Think of race conditions in multithreaded applications. Most accidental complexity is caused by mutable state. No mutable state implies no race conditions. Or think of side-effects that make your application hard to understand and debug. These side-effects are simply missing in functional languages. Think of testing or validating your functionality. Guess why Joshua Bloch spends so much space in his book on Effective Java illustrating how to make Java classes immutable.

In contrast to object-oriented approaches functions are used to abstract functionality and function composition is the basic means of composing artifacts. That’s a perfect extension to object oriented abstraction and composition. Hybrid languages such as Scala use this combination to make building internal DSLs surprisingly easy. For instance, the Actor functionality looks like an integral constituent of the language, while in fact it is just a library. If you wonder what actors are: they can be considered abstract tasks or objects running on their own threads. They do not provide any possibility for reading or changing their internal state. The only way to communicate with actors is by exchanging messages´with them. If these messages are immutable values, there is no chance of side-effects. What a perfect way for implementing multithreaded applications.

Another famous example for functional features used to provide an internal DSL is LINQ (Language Integrated Query) in Microsoft .NET. Using extension functions, closures, anonymous types, and type inference helped to seamlessly integrate LINQ into languages such as C#.

Sometimes you need to deal with state. Languages like Clojure also support Shared Transactional Memory, a good way to encapsulate all state changes within transactions.

What makes the functional paradigma so effective, is it’s succinctness. As an example we’ll use F#:

let rec fib x =

if x < 2 then 1

else fib (x–1) + fib (x-2)

let results = Array.map fib [| 1 .. 40 |]

The code above introduces a recursive function for calculating fibonacci numbers, then builds an array containing the first fourty fibonacci numbers. Try this with Java, C# or C++.

Many problems can be described in a much more natural way using functional features.

Thus we already know several aspects where functional languages are beneficial for the architect:

introducing internal DSLs
Parallel and Asynchronous Programming (e.g, targeting Multicores)
Implementing some parts of an architecture in a much more succinct way
Improving testability
Understandability of design (expressiveness, simplicity)

As mentioned previously, functional approaches do not represent a panacea. Neither does Object-Oriented Programming. But they surely offer an excellent extension to imperative or object-oriented design and implementation. Consider them a perfect extension of Object-Orientation.

It is time to follow the advice of the Pragmatic Programmers and learn at least one additional language per year. My recommendation: focus on functional languages for this purpose.

Implicit versus explicit

2010-03-29T21:13:00.001+02:00

In a very larg telecommunications application I was supposed to review, the software architects had introduced a strict layering to shield different abstraction of functionality from each other. One of the lower layers comprised the database system, while the topmost layer consisted of the application UIs. Whenever something in the DBMS layer changed, the UIs broke. This shouldn’t happen according to the software architecture document. Thus, I asked the UI developer whether he abided by the strict layering. His answer was no, because he got the order to optimize performance which required him to access the data directly. What we see, is a typical example of implicit design decisions. When designers or developers break the architecture or use it in an unanticipated way without communicating the issue to software architects, this will inevitably lead to severe problems which are typically hard or even impossible to detect.

In another project the development organization built a GUI for a monitoring framework. After the application had been relaesed, customers started to complain about a feature they were expecting but which was missing. This illustrates what happens if requirements engineering or customers have implicit expectations. Unfortunately, these requirements change over the years. While 20 years ago almost every developer was satisfied with command line compilers and had to explicitly specify she would prefer more IDE-like environments, today this feature is considered an implicit requirement. You don’t need to mention it explicitely anymore. KANO-Analysis is a perfect tool for such investigations. For programming environments this is very obvious bút what if you’re working in a domain where you don’t know about all these hidden requirements. Thus, mind all invisiple traps.

There are two conclusions we can draw from these examples:

In a software development project, any information should be made explicit, even facts we normally believe every stakeholder should know. If communication among stakeholders is the most important asset in such projects, nothing shall remain implicit. Implicit information is unavailable information!
In every project we need enforcement. Requirements engineering must enforce that the development team really knows and uses all requirements. In a specification all implicit requirements must be turned into explicit requirements, because implicit requirements are typically swallowed by black holes. Software architects must enforce the architecture by closely cooperating with developers and must ensure quality by closely cooperating with testers.

Work based on implicit information can only possibly work in small teams that use an agile approach and where every team member has an explicit knowledge of implicit information. But do you dare to take the risk that all information gaps are really closed?

Always be an explicit software architect without a hidden agenda!

Over the Fence

2010-03-12T23:24:00.001+01:00

It is surprising how many software architects still create their design, throw it over the fence, and then expect others to follow the divine strategies implicitely hidden behind the lines and diagrams.

This behavior also holds for different teams with key developers or subsystem architects that are ought to coordinate design activities, while in practice they are happily inhabiting their isolated islands.

I’ll never forget a telecommunications project where the architects had introduced excellent guidelines and a decent strategic design. However, I was called for an architecture review, because the implementation had turned out to be not quite decent, to say the least. When I searched for the root of all problems, it became clear very soon that no one in the project had really cared about any of the architecture guidelines. And even worse, the strictly layered architecture design had been seriously violated. For example, whenever the database schemas were evolved, the clients broke, although they were totally isolated from each other by a “firewall” of 3 layers. The key developer of the GUI told me in an interview, that he considered performance optimization as his primary directive so that it semed to be totally acceptable to directly access the database.

We can learn from these war stories that good communication is the (most) critical aspect in any project. Without good architecture enforcement, there is no good architecture. But how can architects pragmatically ensure high quality as well as conformance with the architecture design? Management by walking around does the trick. You should frequently visit the different teams and make sure, they really understand the infinite wisdom and essence in your architecture design and architecture guidelines. This is the best way to get their buy-in and support. In addition, you ‘ll detect any architectural shift early. In particular, when developers add accidental complexity by introducing design pearls that are not motivated by requirements. I wholeheartedly believe, agility implicitely requires architects to work this way. But it is not the case that architects are inerrable, (although we’d really like to see the universe from this perspective). Sometimes, even clever wizards fail. As an architect I am definitely better off when developers show me where I was plain wrong. Believe me, this is much better than testers, reviewers or customers happily finger pointing to your design flaws. Thus, architecture enforcement is a bidirectional interaction between architects and implementation, with architects being the drivers.

As Lenin once said: trust is good but control is better. Think about this before throwing your design over the fence in the next project. Some things tend to backfire heavily.

The Extensibility Syndrome

2010-02-21T16:55:00.001+01:00

In one of the Facebook groups I am member of there is currently a discussion going on how to deal with extensible design, especially how to prove to a client that a system meets its extensibility expectations.

Focus on extensibility is important but can also be a big hazard, in particular, when engineers overdo it. For example, consider the application of the Strategy pattern. Basically, applying the Strategy patterns implies some kind of uncertainty which algorithm actually should be used. If many Strategy pattern instances are used, this almost always means that the engineers have no clue in which direction the system should be extensible. Of course, the same holds for overuse of other extensibility patterns as well. Such systems become a nightmare for developers and users.

Thus, make clear with requirements engineering or customers what kind of extensibility they really need.

What needs to be extended:

Should an algorithm be exchanged?
Is it a whole subsystem or layer that is subject to change or extensibility?
Is it a component that needs to be changed or extended?

When will it be extended?

Now,
Mandatory in the future,
Optionally in the future?

Which binding time?

source code,
compile time,
link time,
runtime,
maintenance time?

Who will be in charge of extending?

operator,
user,
developer

You should only consider those extensions that are mandatory, none of the possible or optional ones that may or may not appear in the future.

In the beginning of software design introduce change scenarios as proposed by the book Software Architecture in Practice, 2nd edition (Bass, Clements, Kazman). Here you’ll learn how to describe modifiability scenarios using scenario diagrams and rating them using so-called utility trees.

You may also use design tactics diagrams that deal with how to implement the different kinds of extensibility mentioned above.

If you follow the principles of my Onion Model, then each architecture design step will only be driven by requirements and their priorities. This way, you can establish requirements traceability within your architectural design.

For extensibility and change scenarios, a Commonality/Variability analysis can be leveraged, especially when dealing with a wide variety of extensions such as in platform or software product line development.

In contrast to common believe extensibility or change is only applicable after you’ve designed those parts of your architecture that are subject to extension or change. You can either extend some functional entities or operational qualities such as scalability mechanisms. This implies that all extensibility and change design activities follow after functional and operational design and not before.

When opening your functional & operational design for extensions or change always use the Open/Close principle to prevent unwanted side effects.

To prove that a software architecture is extensible such as expected by the customer you got different means.

An end-user or operator might like to see the implemented change scenarios and how they actually work in the implementation.
Another engineer might like to see how the architecture and implementation support extensions.
A requirements engineer may like to see the mapping from requirements to architecture decisions.

Modifiability aspects (e.g., extensibility and changeability) are much more difficult than they seem at first place. When the amount of extensibility is overwhelming, you’ll often find wizards or configuration automatisms to hide the mess underneath (which does not imply that wizards or configurators are indicators for bad design).

As Heraklitus once said: panta rhei – everything changes. Thus, be prepared!

Does Cloud Computing have an Architectural Impact?

2010-02-12T14:42:00.001+01:00

Let’s suppose you have good reasons to leverage a Cloud infrastructure. Will this decision hava any impact on your architecture? The answer is quite easy: It depends :-)

If you just use virtualization to exactly simulate a physical environment like in Amazon EC2, then your applications will be oblivious to the Cloud infrastructure.
If you leverage SaaS (Software as a Service) like Salesforce.com then it’s a SEP (Somebody Else’s Problem).

However, if you use PaaS (Platform as a Service) or IaaS (Infrastructure as a Service) for your applications, then things definitely will change.

It is not so much the domain-specific logic that is influenced, but infrastructural and non-functional requirements will experience an impact.

In a Cloud environment different storage means are supported. In general, you’ll find Blobs and NoSQL databases. Designing the persistence infrastructure must take these issues into account unless you are still relying on a mainstream DBMS running somewhere in the Cloud.
For messaging Message Queues are available. Thus, message-oriented communication between Cloud-aware subsystems is a reasonable approach for connecting islands with each other. Often, SOA style communication protocols such as RESTful communication or SOAP/WS-* are supported. These protocols alone imply different architecture paradigms - designing the communication and distribution infrastructure can be a challenge.
All operational qualities such as performance, availability, scalability, fault-tolerance need to be implemented with the SLAs in mind the Cloud provider offers. If, for example, availability zones are supported in the Cloud you better plan how to leverage them according to your needs. In a public cloud security is even a more critical issue. You cannot naively trust the Cloud provider to keep your mission-critical data secret. Thus, you have to provide means like encryption almost everywhere. For performance reasons, you might want to consider the Cloud as a Grid such as in Apache HADOOP. But how can you partition your system in a suitable way. Is Mapreduce what you need or, otherwise, does your application require a different kind of approach? There are many different concurrency architectures like Master/Slave, Pipes & Filters, Concurrent Reactor that could be appropriate. In contrast to typical application development you are much more constrained when addressing these operational qualities, because many decisions the Clound infrastructure has already made for you. It is YOU who needs to integrate with the Cloud, not vice-versa.
Developmental qualities such as modularity, modifiability, maintainability can be hard to achieve in such an environment. First of all, you should introduce some kind of governance approach. Secondly, you need to modularize in such a way that governance issues live in harmony with the kind of modularization propagated by the Cloud, in particular by the PaaS APIs. For command & control purposes you need to think about how to administer the Cloud applications. There might be 3rd party products for this purpose. It, however, might also be the case you have to implement your own tooling.
If, for some reason, you need to use different cloud infrastructures, interoperability might be a big issue given that today’s approaches almost inevitably lead to vendor-lock-in.

Not always, you will be able to start with a green field project, but rather need to deploy an existing application ecosystem or parts of it to the Cloud infrastructure. Migration is not straightforward as we already saw earlier. You may need to combine reengineering and refactoring activities for this purpose.

This posting describes only the tip of the iceberg. I hope, people will develop some best practices (patterns) for how to leverage existing Cloud platforms. I’d be rather sceptical if someone would come up with such patterns soon. You surely remember, patterns must be independently used in three different applications to make them patterns. I cannot imagine, we already obtained that much experience in practice.

So, if you are going to design for Amazon EC2, Google Web Engine or Microsoft Azure don’t forget: Mind the Cloud!

Requirements traceability – The Holy Grail

2010-02-11T19:54:00.001+01:00

It often occurs that I am supposed to review an architecture document – that’s one important part of my job. After having read the document, I always try to summarize what I just read. And in many cases, this proves to be rather difficult. One of the reasons is that the subsequent parts of the document seem to be only loosely connected such as graphical models popping up without textual explanation or architectural & technical decisions appearing without any hint what the design rationale could be. However, a reader who is dependent on rumors, insider knowledge, and vague assumptions will see the hodgepodge of syntax but never fully understand the semantics behind this syntax. If the reader is a tester, requirements engineer, fellow architect, or developer such situations need to be considered harmful. Yes I can see the answer is 42, but was the question respectively problem that lead to this result?

The big secret is requirements traceability. Each and every architectural decision must be strongly derived from forces (i.e., requirements and risks). This also helps to keep the architecture simple, expressive and balanced, because this way we can get rid of all these design pearls smart developers and architects typically invent.

From the forces elicitated at the beginning of a project architects derive architecturally relevant requirements. This includes requirements clarification and priorization. Remember: Garbage-in-Garbage-out. If the initial assumptions are wrong, the resulting implementation can under no circumstances meet the expectations. Of course, we do not expect requirements to be fully available at project startup. We have become extraordinary agile wizards, anyway, and can also handle requirements dropping in. Needless to say, we can not gurantee the same perfect software & system architecture we would have come up with if all these requirements had been fully available from day 1.

As we already got those long-term experience, we are using the onion model.

For all use cases (part of user requirements) we are deriving a black box view of our system. From this descriptions we also obtain actors and required/provided interfaces. As we see, this step is driven by requirements.
The domain model is another force. It is a means to understand typical entities in the underlying domain as well as their relationships and interactions. They will turn the black-box view into a white box view. Now you can start to map the use cases to sequence diagrams which is also requirements driven. All of a sudden the domain-specific subsystems, components, interactions, interfaces are disclosed refining our external model. Eric Evans is an excellent source how to deal with such domain-driven design.
So far we have focused on the domain entities. It is time to add infrastructural requirements such as communication, persistence and concurrency infrastructures as well as operational requirements such as performance, safety, availability. This we do by starting with the highest priorities- that’s why priorization is so essential – and ending with the lowest priorities. Each architecture decision is strongly rooted in requirements.
The same is done with the mandatory developmental requirements like modifiability, maintainability and then with the optional developmental requirements.

All these decisions are depicted in diagrams, but also explained in the textual descriptions.

What are the benefits of such an approach:

We get requirements traceability: Firstly, we know which requirements lead to which architectural decisions, thus also obtaining a mechanism to check whether we have already covered all the necessary requirements in our design. Secondly, we know which effects an architectural change such as a refactoring will have on requirements, thus preventing or at least lowering the risk of accidental complexity. Thirdly, all decisions are made explicit. Implicit parts lurk like invisble ghosts deep in the guts of the architecture hurting us when we are least prepared.
We are driven by priorities. If we miss our milestones, we skip the less important requirements instead of the really important things.
Readers of our documents will not only understand the “what” but also the “why” making it much easier to get their buy-in. This is what we call design rationale.
If we additionally come up with leading guidelines and principles for cross-cutting concerns in the beginning we will increase internal qualities such as symmetry, orthogonality, simplicity.

The journey is the reward in software architecture. Don’t tell readers about the target to aim for, but also about the way how you get there. On each junction, explain why you prefer one over the other road.

Writing of such documents can be such a fun. And reading them even more. For example, read Dave Cutler’s book about the creation of Windows NT.

A architecture document shall contain guidelines and design rationale, but no labyrinths.

Back from the OOP 2010

2010-01-30T14:31:00.001+01:00

This year’s OOP conference in Munich has been a lot of fun. I met Markus Völter, Stefan Tilkov, Floyd Marinescu, Jan Bosch, Eric Evans, Philippe Kruchten, Roman Pichler, Peter Hruschka, Kersten Auel, Rene Schönfeldt, Klaus Rohe, Matthias Bohlen, Gernot Starke, Jutta Eckstein, Nico Josuttis, Gregor Hohpe, Robert C. Martin, Matthias Bohlen, Kevlin Henney and my good ole friend Frank Buschmann. Especially the networking part was overwhelming. So many speakers and participants you can learn from.The talks and keynotes are definitely helpful to hear about new trends, perspectives and experiences. But the personal communication is the essence of such conferences. That’s the reason why I don’t believe in online events. Sure, it’s a nice add-on but it cannot substitute the “human” factor. In the middle of the week we had a meeting of the Siemens Senior Software Architects where I enjoyed meeting some of the aforementioned celebrities but also the smart senior architects within Siemens who do excellent work day by day. They might not be that much in the limelight, but those people are the real heroes of software engineering among all those “nameless” other heroes such as development heads, team leads, testers or developers. As Jan mentioned: there are two types of architects. Those talking about software architecture and those practicing it. Fortunately, I am a hybrid belonging to both types.

This year’s keynotes offered excellent quality. I’ll never forget Uncle Bob’s entertaining talk on the polyglot programmer where he dived into the history of programming languages in a very entertaining way. Or two speakers from Zühlke Engineering who presented a great keynote on functional programming. Not to forget Gernot Starke with his talk on software architects and stealing in your neighbor’s garden. Unfortunately, I could not make it to his keynote but from what I heard it was one of the highlights.

Most of the topics presented in talks and tutorials covered the typical topics you’d expect. Cloud Computing and Functional Programming as well as Agility represented the hot topics. What I liked most were all these more practice-oriented talks offering some insights for practioners and also revealing some real life show cases. For instance, I enjoyed Kurt Höpli’s talk on building and controlling environmental sensors in a project in Switzerland. And I also experienced a lot of fun in Markus Völter’s talk on how to apply Model-Driven-Software-Development within an embedded systems context. Never I will forget how Eric Evans and Hans Dockter presented Domain-Driven-Design convincing some attendees to participate as actors. And I’ll remember also Lothar Mieske who presented the real look-and-feel of all those Cloud Computing platforms. I could add much more but this should give you an impression and suffice as a pars pro toto.

Personally, I have been very busy the week. First, I gave a full-day tutorial on Software Architecture – From Requirements to Architecture. It was the tutorial with the largest numbers of attendees. This makes interaction somewhat difficult but nonetheless worked much better than expected. My talk on Hitchhiker’s Guide to Cloud Computing had about 80-100 attendees. To my pleasure the Global Technology Field Cluster Lead of my organization also attended the talk. For me, it was the biggest surprise that the talk “Introduction to Scala” had attracted so many people. I was totally electrified and I mean this literally cos’ static charge hit me whenever I pressed the page-down key. When people are interested, I will record this talk with Camtasia and publish the video clip in YouTube and other media. All in all, I got really overwhelming feedback. Thank you to all attendees for their nice evaluation sheets :-)

Next year the OOP will celebrate its 20th anniversary. Hope that many of you will join the event. It is really an extraordinary experience.

Stakeholder-specific Communication

2010-01-21T07:11:00.002+01:00

Communication is one of the most importants aspects of software architecture design. The architecture itself is a means of communicating design decisions to customers, product managers, project managers, testers, integrators, and developers. However, these stakeholders have different expectations regarding the software architecture. While developers need a clear guideline how to implement the architecture design, customers may be more interested how (well) the software meets their requirements. The software architecture documentation and all other means of communication such as presentations must cover these various perspectives. To address such broad spectrum of interests, architects could provide a specific document for each particular kind of stakeholder which leads to an unacceptable amount of effort and time. This is why documents should rather explain the architecture decisions - the "what", the "how" and the "why" - in a top-down approach and add a guide-to-the-reader motivating who (i.e. which stakeholder) should read what in which order. For presentations or other forms of communication it might be worthwhile to introduce a specific variant for each stakeholder. A project manager or customer is typically not that interested in all those strategic and tactical design destails, while developers need all the information they can get. Thus, communication between architects and other stakeholders should take the stakeholder-specific interests into account. This implies, that architects need to have a clear understanding of these interests before starting documenting or communicating. Otherwise, they won't get buy-in for their software architecture by these different stakeholders. Personally, I have experienced so many projects where managers couldn't follow and understand what the architects told them overwhelmed by details they did not (have to) care about. And I also have been in so many meetings where software architects introduced high-level abstractions that could not provide the details developers and testers expected.
As a prerequisite it is necessary to ensure a common understanding of terminology. Did you ever attend a meeting where different participants had a different understanding of terms? Communication will not work and succeed when one person is speaking english, while the communication's target can only understand spanish. This might be obvious for real languages, but it is often underestimated for domain-specific languages. Did you once try to read and understand (!) a treatment record of your physician? He speaks english, you speak english, but this won't help you much. As a consequence, introducing a domain-specific language is valuable and essential. It does not have to be a full-blown language in the strict formal sense, but often it makes sense to offer more than just a glossary. The core functional architecture should represent the problem domain.
As a consequence, all communication between architects and stakeholders must happen in a stakeholder-specific way. When giving a presentation on the software architecture, make sure you know the target audience and their interests. Don't assume, other types of stakeholders understand software architecture the way you do. You are the doctor and they are the patients, so-to-speak. Communication may succeed or fail. The risk of failure can only be minimized if you can take and understand the perspective of your communication partners.
Obviously, these statements also hold for other kinds of communication. Have you ever tried to explain to your spouse why you need that cool new gadget for several hundred bucks? Try that by telling her/him the technical specifications. Won't work very well, right? What we take for granted in such situations (i.e., adapting the communication to your communication partner), represents also a conditio-sine-qua-non in software engineering. So, better mind the stakeholder-specific communication in the future :-)

Real Software Engineers

2010-01-09T12:04:00.001+01:00

Real software engineers don’t need safety nets. Instead, they are ready to face all dangers and risks without any fear. For this reason, they dislike (unit) testing, because (unit) testing is for pussies (i.e., those who do not trust their own code.) Since real software engineers never make mistakes, testing would be a mere waste of time.
Real software engineers won’t document their design, because these designs contain an incredible amount of design pearls other mortals would not be able to comprehend, anyway.
Nor do real software engineers read other design documents nor manuals, either because their fellow engineers did not provide documentation or because they do not trust these documents.
Instead, their main guidelines are:

All you need is code
Bubbles don’t crash

Real software engineers do not check or prioritize requirements. Instead, they got intuition for customers’ needs which drives all their activities.
Real Software Engineers do not communicate (much) with other stakeholders, because other stakeholders are just obstacles on the way to project completion. That’s also the reason why real software engineers hate meetings.
Real software engineers do rarely speak English, German, …. They rather express themselves using C#, Java, C++, …
Real Software Engineers feel more comfortable in the solution domain. Their goal is to provide solutions instead of getting lost in problem domains. This is why they prefer playing around with solution technologies.
Real software engineers can turn any problem into a nail for which they can use their favourite hammer.
Real software engineers do not need software architecture because a predefined architecture would just limit their creativity. A good architecture is always created ad-hoc for solving real problems in a pragmatic way.
Real software engineers do not foster re-use. First of all, re-use has never worked in practice. And secondly, re-using design or code typically requires more resources than reinventing the wheel. This is often due to the fact that re-usable artifacte were created by other real software engineers who failed to document their work.
Real software engineers won’t read this blog because in the time needed for reading, they could add another feature to their software.

Architecture Documentation Revisited

2010-01-07T19:09:00.001+01:00

Did you ever read a novel which later has been used as foundation for a movie? I bet, you were rather disappointed how the movie differed from your reading experience and thus from your expectations.

Did you ever read an architecture document – yes, there is vague evidence and hope that something such as architecture documents exists – and then compared the documented design with the implemented system?

In software architecture design, documents are not just by-products created for write-only brain dumps. Instead, they are supposed to be read by someone – a fact that might be surprising for some engineers.

Architecture documents serve as the basic means for communicating architectural decisions, i.e. they explain the “what” and the “why” – aka design rationale. Target audience of architecture documents are foremost all stakeholders, not just the authors themselves.

In addition, those documents should be up-to-date. Did you ever struggled through 100s of pages of a design document, and right after you finished, someone told you, the document was outdated?

Does this mean, we should adapt the documentation after each change? No, it only implies, architecture documents should be updated regularly. Consider versioning architecture documents in this context!

An architecture specification should also be subject for testing. Don’t take testing too literally! What I suggest is that documents are being reviewed for their usability/readability, internal consistency and completeness, as well as their consistency with the actual implementation. Needless to mention, that the reviewers shouldn’t be the authors but a selection of stakeholders.

Always use the right style of documentation depending on purpose and target audience. A user document that describes a system should contain tutorials as well as a reference manual. If no one understands it, the best design ever won’t help you much.

My strategy for documenting is to walk to the other side of the fence thinking about how a specific stakeholder would like to get the material presented. I am not writing the document for myself but for others, anyway. And I always take care of updating the documentation whenever the system has been updated to a new (minor or major) version. Readers should not be punished by reading my document but consider this as an entertaining and profitable activity. Reading design documents is fun or at least should be! Who claims, IT documents must always be written in a technical and boring style?

Remember: the most important capability of a software architect are communication skills. Documenting is one particular way of communicating.

A picture can say more than thousand words - a word can say more that thousand pictures

2009-12-09T20:26:00.001+01:00

One of the key problems in architecture documents is the idea illustrations are sufficient for explaining design.

Let me give you an example:

Eo---oP

What could the diagram illustrate?

Could this describe a hydrogen atom consisting of a proton and an electron?

Does it represent the association between an entity bean and its persistence store?

Or is it something completely different?

Diagrams are models. They can only express a subset of reality. If you do not know the concrete context, you’ll have no way to figure out what the diagram actually means. But if you are familiar with s special context, you might be inclined to map a diagram to your own context which, however, might be painfully wrong.

Thus, an architect should always explain the context of a diagram as well as the constituents and their meaning textually in the document.

What about the other way round?

Again, an example:

Component A is connected with B, C, D. D is connected with E and C. F is connected with D and A … a lot of more information …

In the example above, plain text is not really helpful. Here seems to be a good place to add a diagram that graphically illustrates the components and their connections. Text alone is not capable of describing complex structures which often appear in architecture design.

Hence, we recognize that diagrams need explanations to understand their semantics. Otherwise they just represent syntactic glibberish. In addition, UML, SysML and other graphical notations might lead to ambiguous illustrations which are open to interpretation. Thus, tell the reader what exactly the model behind the diagram is supposed to be.

As a consequence, all design documents that are automatically generated have almost no value. For example, design rationale (such as requirements and their association with architectural decisions) is better described using textual language instead of diagrams.

Thus, use diagrams carefully in your design documents. But also be aware of text that does only describe what is depicted in diagrams, but does not introduce information such as pre- and postconditions, invariants, contextual information, design rationale, …

Guess, why there are UML use case diagrams that illustrate a set of use cases, but no graphical means for describing the use cases themself?

Use text and diagrams wisely! Don’t consider design documents as write-only-entities. You will see what I mean whenever you have to understand documents of your fellow engineers.

Maturity matters

2009-11-29T21:33:00.001+01:00

This time a rather small posting only.

I am wondering what the top research fields in software architecture currently are? Of course, fields such as MDSDL/DSLs come immediately to my mind. Cool stuff! Investigate these topics and fame is guaranteed!

Others are already sufficiently mature. I am thinking of mapping requirements to architecture decisions. Ok, maybe there are still some deficiencies in this area. Hmmh, there might be even significant gaps, especially when dealing with operational and developmental qualities. Must reevaluate my opinion.

What about documenting software architecture? There are dozens of templates available, even books. A lot of agreement here, but what about modeling? Is UML the “ultimate ratio”? How can we model, what should we model? How can we cover design rationale? How deep should we go? Did I ever read a really good architecture documentation?

I got it. Patterns are super mature. I have been author of pattern books myself. These books are now available since 14 years. Some architects even know the names of all patterns. But do most architects know patterns others than GoF? And even if they do, do they apply them?

Isn’t refactoring a good candidate or reverse engineering? Architecture refactoring isn’t established in most development organizations. If I say architecture refactoring, I really mean architecture refactoring, not code refactoring.

What about software architecture design? We have been designing software systems for decades and they work! No, most of them suck! Unfortunately, design by accident is the norm, not the exception.

… thinking, diving deeper, diving broader, getting frustrated …

Why is my impression that most areas in software architecture are still rather immature?

Maybe, the answer is that software architecture itself is a big collection of research topics.

Or maybe, I am blinded by the light.

Architect – what architect?

2009-11-10T18:22:00.001+01:00

In many projects and organizations I am constantly meeting software architects. Given the quantity I’d only expect only a fraction of software development projects to encounter any issues in software architecture and design. As we all certainly know, quite the opposite is true.

Ok, nice and true words, but what exactly is the reason for this observation?

Of course, I absolutely have no clue :-; So let me just make some assumptions.

Could it be that the term “software architect” on a business card is subject to a variety of different interpretations?

In one company, someone may be promoted to software architect, just because she/he is the only one who owns a pattern book. [Beware of humour!]

Consulting companies may entitle some of their engineers as software architects, because they can charge more for architects than for developers.

A handful of companies may offer a challenging education program to become a software architect.

With other words, it is not clearly defined what a software architect is or should be. Nothing prevents you from calling yourself a software architect. You may remember that I am proposing a rather tough profile for software architects. Well, I know too many projects that failed miserably because of lack of competencies, experience or skills by the responsible software architects.

Note: not only the skills count! As software engineering is heavily related to communication and learning from failure, becoming a software architect requires starting as a developer, taking over more and more responsibilities over the years, before finally turning into a butterfly.

Another issue I often encounter is the missing separation between software and system architects. Today, many system architects are also considered as software architects. This might work very well if the system architect is an experienced software architect. Likewise, the aforementioned setting is adequate for software-intensive developments. But here again, the system architect needs also to be an experienced software architect. If the latter doesn’t hold, the software relevance is obviously underestimated and will inevitably lead to trouble.

Hold on, there is even another possibility. Even if you can justify that you are a software architect now, this doesn’t mean you’ll be a software architect forever. One of the most fundamental challenges is to know the state of the art of relevant software technologies and methods as well as appreciating and practicing coding and testing skills. Guess, you have been appointed as a software architect ten years ago, but in the meantime you were constrained to using UML tools, Powerpoint and other Office applications. With other words: would you trust a surgeon to operate on your heart who lacks ten years of practice?

I bet, you made some similar experiences in your environment. If that is the case, a comment would be highly appreciated. If not, even more!

Sick Architecture

2009-11-06T16:59:00.001+01:00

You may wonder why my publication rate in this blog has decreased dramatically. I am staying at home due to an infection for three weeks now which wasn’t that helpful for my blog.

To keep things running I have decided to create a new posting. Well, the title is intentionally related to my current health situation.

To be honest, I think that software architecture design reveals some interesting commonalities with sickness:

Both often require intense treatment by experts
Different experts offer different opinions and treatment plans
Both are typically open-ended to some extent
Both are not subject to planning in most situations
Both mostly feel very miserable
To reduce the risk of getting infected some kind of prevention strategy is always a good advise
It is better to cure the cause not the symptoms. Thus, be aware of all interdependencies and follow a holistic approach
Viruses are a problem in both worlds
Healthcare never ends – it is important in the whole lifecyle
Learning from failure is recommended to avoid similar problems in the future
Prevention is always better (i.e, cheaper) than healing

Most software architectures I know are not in a good and healthy shape - at least in some intermediate phases. Why? Because they were not devleoped with the aforementioned bullet list in mind. Thus, health monitoring (i.e, reviews) and treatments (i.e., refactoring) should be an essential tool in the architect’s framework.

Does this sound reasonable?

Data centric Architecture

2009-10-15T09:14:00.002+02:00

In most projects there is a big emphasize on functionality. Use case modelling defines responsibilities of the system as well as how it interacts with actors. Logical modelling then maps the use cases to related subsystems/components which, of course, is functionality based. At the end we'll get functional interfaces between subsystems or between components. Do you recognize that methods such as UML mostly focus on methods?

However, in many cases the data exchanged between architectural entities is rather complex. For example, in healthcare scenarios complex structures with patient information or charging information are the rule. Not dealing with data as first class citizens might lead to various issues such as versioning problems, or inadequate partitioning or aggregation of data. Thus, architects should deal with data modelling very soon if they are facing complex structures in their development projects.

Data modelling starts with the domain model. What are the main entities in your problem domain and which kind of data do they need to persist, share, or transfer?

Then in the next "layer" when we are introducing the subsystems we'll have to think about data structures in subsystems and data structures exchanged between subsystems. For example, what exactly are the data structures required in the functional interfaces. Obviously, these data structures can be partially derived from the domain model with further data structures required by infrastructural entities or non-functional qualities.

In the component layer we are starting to deal with finer grained data types.

Last but not least we'll have to define how data is mapped to implementation artifacts.

Thus, we should also follow a top-down approach for data modelling. And likewise we need to consider bottom-up constraints such as special data requirements of infrastructures or platforms. With other words, data modelling must go hand in hand with all the other architectural modelling activities:

Important questions to ask when modelling data:

Who is the originator of data?

Who is the consumer of data?

How does data move from the originator to the consumers?

Do we need to collect/aggregate data from different sources (transfer objects)?

What about quality constraints such as performance (e.g., caching data), security (e.g., encrypting data), modifiability (e.g., versioning data), concurrency (e.g., locking data), availability and reliability (e.g., replicating data, data persistence)?

Where and how is data stored?

How do different data structures depend/relate?

I think, data modelling is an underestimated and undervalued activity in architecture design. What about your experiences and opinions?

Ineffective Communication

2009-09-18T16:36:00.001+02:00

It might not be that obvious why time management is of high importance especially for software architects. But of course, being a software architect in a mission critical project requires your full attention. However, in a couple of projects I met software architects who were simply overwhelmed by communication requests and finally got stuck in a kind of waste time singularity. Ineffective communication can significantly reduce your productivity.

Unfortunately, there are many opportunities where your focus on the real important things is easily lost.

My favourite are unnecessary meetings (including telephone conferences). As software architects are caught in the gravitational center of the project they receive a lot of meeting requests, by managers, developers, other engineers. If you are a very communicative and friendly person – and I bet you are – you might be inclined to say “yes, we can” to all those nice invitations. It is often overseen that meetings also require preprocessing and postprocessing activities. Hence, don’t just count the time you’re spending in the meeting room! Consider carefully what type of meeting you are asked to attend. Is it an essential meeting where your participation is absolutely required (class A meeting)? Or more one of those nice-to-have-meetings? Only accept class A meeting invitations! Even for class A meetings ask yourself whether this meeting has the right setting in terms of people invited, time frame, goals, agenda. With other words: is the benefit you obtain larger than the time spent? Sometimes, a small face-to-face meeting offers more value than a large official meeting. And of course, act thoughtfully if you invite to meetings yourself! As an Outlook junkie do not only enter meetings in your calendar but also time slots for your actual work. Mind the fact that other people tend to fill the gaps in your calendar assuming those gaps are indications of idle time. Personally, I also enter slots for my spare time, just in case. Needless to say that you should also consider travel time in this context. Care about meeting netiquette: there is no place for cell phones or notebooks in a meeting – a few exceptions granted.
Another critical challenge is e-mail. Communication by e-mail can be very ineffective (but also the most effective means in other situations). Try to reduce corporate spam which consists of all those mails which steal your time for no good reason. Organize your inbox in folders and prioritize mails as soon as it arrives. Process the mail depending on its priority. Read mail only at specific points of time and tell your communication peers about this fact. And do not send unnecessary mail yourself! Before writing an e-mail, always hold on and consider whether you would also have sent an letter containing the same information in the stone age of communication.

At the end, it is the project success that counts. Good results require effective architects with as much productive working time as possible. Productive hours depend on effective communication. It is the quality not the quantity!

Vacation - the end is near

2009-09-05T23:50:00.001+02:00

The last weeks I have relaxed a lot: visiting Dresden, running, swimming and biking, as well as practicing with my DSLR. In addition, I read two books on Scala, updated an article on D (the language), installed several apps on my IPhone (yag – yet another gadget), prepared slides for ADC 2009 and OOP 2010, participated in a new episode for our SoftwareArchitecTOUR podcast, … and spent a little time on further expanding my work on architecture refactoring.

However, I intentionally did not work on software architecture or on this blog. This will change beginning with upcoming monday. Now, that I got some new and fresh ideas, I can continue my journey on software architecture issues.

By the way: I did not mention it so far but maybe you like to follow me on twitter (username: stal_m).

And now for something completely different

2009-08-21T00:41:00.001+02:00

This is my first week of a 3 weeks vacation. It is really hot in Munich why I am currently enjoying swimming, biking and visiting beer gardens more than software engineering.

Nonetheless, I am currently educating myself in a new programming language. Scala by Martin Odersky combines a pure object-oriented langauge with functional features. And it runs on the JVM. An earlier version of Scala does also support the .NET CLR.

What I like most is the proper integration of functional features. Closures, for comprehensions, nested functions, pattern matching are notable examples. While Scala is statically typed (it is not a dynamic language), it offers type inference for variables, arguments and results.

In addition, it provides some cool extensions to Java: all types are objects, no static members, support of covariance and contravariance, to name a few. Traits are available instead of interfaces – they can behave like interfaces but also allow implementations of methods. With traits the problem of multiple inheritance (mind the diamond configuration) can be avoided. This is possible by introducing inheritance linearization and a kind of virtual super.

With all of these features Scala code can be much more compact than Java - and it interoperates well with Java components.

Well, I am really kinda enthusiastic. If you ask me, Scala could be a strong competitor for Java and C#. But of course, there is not a big company backing Scala such as Sun or Microsoft. Thus, the community will decide.

Loving the JVM but preferring other languages than Java? No more excuses, Dave!

Neverending Story

2009-08-01T11:39:00.001+02:00

One of the prominent questions I often hear in software development projects is when to start with architecture design. Sometimes, architects hesitate to start until almost all requirements are available in high quality. This strategy is highly recommendable when you like to postpone architecture design forever. There is not a single project with medium to high complexity I remember where we knew all forces in full depth and breadth at project start.

First of all, it heavily depends on the software development process. In a Waterfall model without iterations you are stuck anyway. Thus, I generally recommend an iterative-incremental approach. In software developments with all their volatility only piecemeal growth can succeed.

It is important to have a sufficient quality and quantity of requirements available, for example, the most important 30%-50% of use cases and non functional qualities with unique priorities. This way, you can come up with a conceptual architecture which you can then use as a base for iterative-incremental refeinement.

After the baseline is available prioritize all user requirements (use cases), map them to end-to-end scenarios and integrate them into the architecture, keeping the non-functional qualities in mind. To deal with non functional qualities let key developers, architects or even teams work on principles and guidelines how to address these qualities. Do this after the baseline architecture design but before starting the end-to-end iterations.

As soon as new requirements appear or have been concretized assign them priorities and put them into the “backlog”.

To introduce a kind of safety net for embracing change, introduce test-driven design methods, flash architecture reviews after each iteration, and (architecture and code) refactoring activities after each flash review.

Keep in mind that sometimes you need courage to decide even when not all details are available. It is typical for all creative and volative acts that not all details are known in advance, nor the ways how to cope with them.

As a consequence, it is much more productive and effective to follow wrong paths sometimes instead of following no path at all.

To keep it simple: Prefer acting over reacting

May the force be with you

2009-07-23T07:25:00.003+02:00

Sometimes it is getting confusing. There are so many influential factors driving software architecture. Here is my small domain language

Forces is the set of all influential factors an architect is facing: requirements, risks, constraints, business needs.
Requirements are the (documented) set of all expectations for a software system. According to Wikipedia a requirement represents a necessary attribute, capability, characteristic, or quality of a system in order for it to have value and utility to a user.
System requirements address the system as a whole. From system requirements we need to derive architecturally relevant requirements that drive architecture and design. That is the architect's job.
A Functional requirement defines functionality a system is supposed to provide, sometimes also known as capability. Use cases are functional requirements. They define what the user expects from the system on a black-box level. User stories according to Wikipedia denote software system requirements formulated as one or two sentences in the everyday or business language of the user.
A Non-functional requirement (also known as quality requirement, quality attribute or "ility") is constraining a solution. There are two subcategories here: operational requirements as the name suggests constrain the runtime behavior of the system (e.g., security, performance) while developmental requirements constrain the design from a developmental perspective (e.g., maintainability, modifiability). As operational requirements often have a systemic approach to a system they are subject to architecture design and therefore are also called strategic requirements. Tactical requirements such as modifiability mostly have only local scope.
Constraints - what a surprise - are constraining the solution. A constraint might be technical (we need to use Windows 7 as our OS), organizational (subsystem A should be developed at location A), standard & law (we must abilde to FDA rules), business-related (target costs should not exceed xxx$).

It is the job of a software architect to base all decisions on the forces of the project. Still she/he has sufficient freedom to make architecture and design a complex task. Architecture is about communication and courage to decide.

No future for architects?

2009-07-20T09:24:00.001+02:00

I am a big fan of Star Trek, especially of The Original Series and The Next Generation. From a technology viewpoint, it is amazing how all of these technologies such as warp drives or beaming have influenced science. Did you know that the inventor of the mobile phone had also Star Trek in mind?

However, there is one sad aspect. What about Software Engineering? Whenever software plays a role in any of the episodes, you’ll recognize Spock or Data just coding. Do they ever design? There is also no software engineering team in the Enterprise although software should be an important asset in the system architecture. I am missing a chief engineer for software engineering aspects!

Does the computer program itself? Will UML disappear in the future? Or is it just software engineering being too boring for SciFi fans? Interestingly, there are also seem to be no restrooms in the spaceships. This is what they have in common with software engineers :-)

Thus, we could ask how software engineering especially the discipline of software architects will evolve in the future? Will we still use a successor of UML in hundred years? How could the role of software architects evolve?

The best approach to predict the future is identifying space of improvement. It is very likely that such areas will be addressed. How should the ideal process of software design look like? What is the software architect supposed to know and to do?

In most SciFi movies automatization/Model-Driven Software Development and AI are often the answer. Architects express their intent, smart systems then try clarify open issues, and eventually the result is generated.

Another approach could be an organization like the Borgs who strongly follow an agile approach with code ownership, collective programming and test-first.

Any opinions?

Keep in mind: Resistance is futile!

RRC Card

2009-07-19T10:23:00.001+02:00

Most of you already know the CRC (depicting Class, Responsibilities, Collaborators) which is a low tech technique for discussing software artifacts in brainstorming sessions for software design.

I propose another kind of card for depicting the roles in a particular project: the RRC Card: Role Responsibilities Collaborators.

The idea is simple:

The first R stands for role by which I mean a concrete person and her/his role
The second R stands for Responsibilities: what responsibilities should this role have to meet its expectations
The C stands for Collaborators: from which other roles does this role depend, e.g., with which other roles must it interact

If you start with a project, you could ask the parties involved to jointly fill out these RRC cards, thus making clear how responsibilities are partitioned and how roles are supposed to interact.

Example:

---------------------------------------------------

R: Michael / Architecture Consultant

---------------------------------------------------

Responsibilities Collaborators

Mentor and coach Lead Architect: Paul

Help design Architect: Tom

Implement

----------------------------------------------------

If you already in a project in deep trouble you could ask persons to fill out how they live their own role and likewise fill out RRC cards for other persons, thus identifying conflicts between expectation and reality.

Document these RRC cards in a document (e.g., paper-based or Wiki-based)

Risk-driven

2009-07-18T12:14:00.001+02:00

In addition to requirements and business expectations, risks represent an important driver of architecture design activities. Not addressing risks appropriately is a major reason for project failure. But what exactly is a risk and how can I identify potential risks as an architect.

On a very coarse grained level, risk might be considered as not knowing the outcome of an event or activity. It is the threat or propability that “an action or event will adversely or beneficially affect an organisation's ability to achieve its objectives” [Wikipedia].

There are as many different types of risks in a software development project as there are influential factors.

For example:

Technical risk: Is EJB the right technology for achieving 100000 transactions per hour in our Web store?
Organizational risk: Is the organization appropriate for this kind of development project?
Political risk: Does this product manager follow her/his hidden agenda? Is this supplier really trying to support us or is he in fact competing with us?
Design Risk: Is this design decision appropriate for dealing with the requirements?
Requirements risk: Do we really understand the requirements of our organization or the customers?
Process risk: Is this the right development process?
…

For dealing with risks, there are lot of different ways. Technical risks and architecture risks can be reduced by a requirements-driven, test-driven approach with explicit assessment activities and refactoring activities. Political risks require the architect to communicate and lead efficiently and effectively.

An important point is this context: to be able to deal with risks, we must make them explicit. Implicit risks are often overseen, forgotten or ignored. You should really know your “enemy”.

Making risks explicit means to document AND communicate them.

I recommend to introduce a template for risk assessment comprising:

Name of the risk
Priority/Relevance of the risk
Type of risk: organizational, technical, ….
Rationale: explain why this is a risk
Context: what else should we know when dealing with this risk
Strategies: possible ways to deal with the risk
Decision: the recommended strategy to deal with the risk for the concrete context, and the rationale for this decision
See also: other risks this risk implies or similar risks that could be treated with the same strategy

Not all of these fields need to be available in the beginning. You could start with name, type, context, thus collecting the risks first. Then, you could prioritize the risks, and eventually extend the templates for this risks identified as high risks with missing information.

Let me give you an example:

Name: Uncertainty Using EJB
Priority: high (******)
Type: technical
Rationale: capability to meet the performance requirements of 100000 transactions/hour. We are not sure whether EJB will be appropriate
Context: Our developers are not experienced with EJB
Strategies:1) Build performance prototype first. 2) Hire external expert. 3) Use Enterprise OSGi instead. 4) Outsourcing.
Decision: Hire external consultants. Let them build a performance prototype and then help implementing the system when EJB proves to be a reasonable choice.
See also: we are not sure whether MySQL can cope with the throughput requirements.

It is worth mentioning that risk assessment is not exclusively provided by software architects. In fact, all roles in the project must help and closely cooperate for risk assessment. This way, everyone involved is aware of potential risks. And of course, some risks such as those implied by organization, test quality, or quality of requirements must be identified and clarified by other roles such as product and project management, requirements engineering or test and quality management.

Architecture and design decisions should then take risks into account.

Mind the risks: The highest risk is not systematically dealing with risks.

Who is responsible

2009-07-17T08:13:00.002+02:00

Suppose, you are working in a software development project as a software architect. You are still waiting for the requirements to be available in sufficient quality and quantity and priorization. Even after months almost nothing happens. Sure, the problem is caused by requirements engineering. But what is your responsibility as an architect? Your responsibility is to communicate and escalate which requires courage. For example, you could tell the stakeholders what implications the current situation might have - for example, you cannot start with architecture design which means development cannot start with implementation etc. You should never fall into the "this is not my job" trap. After all, software architects need to drive the project. Sometimes, it might be even possible to take responsibility for requirements as an architect but this should be the exception not the rule.
Similar issue for test & quality: if test coverage is only limited, bugs are seldomly found during testing, or tests themselves are faulty, then you could finger point to testers and developers. However, it is your job as an architect to deal with quality. Don't try to escape your responsibility in this area. Test First Design makes it very clear that testing requires the different roles to cooperate.
What about business aspects? I know, you got this product manager who is in charge of all business aspects. But software architects are responsible in helping develop the technology roadmap, identifying patents, estimating the business implication of architecture decisions. Even more, architects must base every architectural decision on requirements and business needs.
Hmmh, what about project management? Of course, an architect should not be forced to also act as a project manager. These two rules are very difficult to live at the same time in the same project. But what if project management needs to estimate costs and resources? Here the software architect needs to provide support. What skills do developers need? How many developers and how much time are necessary to develop subsystem A.
I don't claim that architects shuld be jack of all trades but masters of none. Their main focus should be on software architecture. However, they cannot ignore what is happening on the boundaries between software architecture and the rest of the project. They should feel responsible for project success.

Domain Models (continued)

2009-05-21T18:52:00.001+02:00

Throughout my life as a software engineer I have participated in many different projects in various domains. Needless to say, I encountered some deja vu experiences in those projects, also known as war stories. One remarkable example always has been the problem domain itself. Even in projects with lots of experienced participants the problem domain had been often root of misunderstandings and misconceptions. In large projects with outsourcing partners, external partners and other suppliers common knowledge of the problem domain is essential. A project with lack of problem domain knowledge among the stakeholders is doomed to fail. Remember the tower of babel where lack of communication and insufficient common understanding lead to lethal catastrophe.

As I am mostly acting as software architecture mentor and coach or technology expert I am often master of the solution domain, but not of the problem domain. For me it is inevitable to obtain a detailed knowledge of the problem domain. This is even more important when developing product lines or platforms, because in that case lack of common understanding influences more than one product or solution.

How could we address that problem? My recommendation is to introduce a domain model. Of course, I am referring to the problem domain in this context. A domain model is not just a glossary of terms – a common misconception I am often facing.

A domain model consists of all core concepts (actors, subsystems, components, services, …) relevant within that domain as well as the relationships and interactions between these core concepts.

A large domain typically consists of subdomains. For example, a telecommunications domain for Unified Communication might introduce subdomains such as basic communication or presence. As a consequence, the domain should be partitioned hierarchically into subdomains.

There are different ways to descible a domain model, from informal approaches to strictly formal definitions such as a domain-specific language. In all cases, stakeholders might select textual or graphical representations, whatever is more appropriate.

Note that I have not assigned the responsibility of introducing a domain model to software engineers. Defining a common domain model should rather be a joint endeavor of all stakeholders, maybe driven by requirements engineers and architects.

If you have ever read definitions of software architecture like the one by ANSI/ISO, you might recognize that architecture definitions match closely with my definition of domain models. First of all, those architecture definitions are insufficient (due to their genericity). Secondly, a domain model is a good starting point for creating the architecture. Take the use cases (black box view) and specify their dynamics using the domain model entities (white box views). Supplement this architecture core with design tactics for non-functional requirements. This will lead to the architecture baseline. Using the domain model also helps you staying focused on the problem domain instead of diving to the solution concepts too early.

Eventually, a domain model defined, understood and agreed by the various stakeholders such as requirements engineers, developers, testers, architects, product managers, customers serves as a fundamental mean for architecture design and communication.

Of course, the solution domain itself can also be expressed by a domain model. This is why the main task of an architect consists of mapping a problem domain to a solution domain.

Formalizing both to a sufficient extent helps automating this process using model-based software development approaches. Such automation, however, might be too expensive for one-off applications or smaller product lines.

Whenever you are participating in a development process in the presence or future: do not forget to specify the domain model if not already available.

Podcast on Software Architecture

2009-05-17T19:29:00.001+02:00

For those of you capable of understanding German, I’d like to point you to our new Podcast on Software Architecture called Software ArchitekTOUR. You may also search in the ITunes Store for the podcast and find it – for example, search for ARCHITEKTOUR in the podcast category.

The Podcasters are

Markus Völter (freelancer, associated with ITEMIS)
Stevan Tilkov (innoQ)
Christian Weyer (thinktecture)
Michael Stal (Siemens)

To stay in touch with reality we have split up the podcast in three subcategories:

General Architecture Topics (sponsored by IBM)
Java and Software Architecture (sponsored by innQ, itemis, thinktecture and Siemens)
Microsoft Technologies and Architecture (sponsored by Microsoft)

There are already 4 episodes available. Episode 0 is more intended to introduce ourselves. Episodes 1-3 cover patterns in general, as well as patterns in Java respectively .NET.

Listen, enjoy, and give us feedback

Simplicity rules

2009-05-12T09:00:00.001+02:00

Many architectures are overwhelming. Not that they shine due to their excellent quality. Rather they appear as a big ball of mud. I already mentioned what I am calling design erosion in the context of architecture refactoring.

To keep an architecture small, expressive and simple we can do a couple of things as software architects.

For example:

Each and every design decision must be based on an architecturally-relevant requirement. There is no decision without documented reason. Of course, this requires architects to feel responsible for the quality of these architecture requirements, but not for requirements engineering, of course! It is recommended practice to keep track on where in the architecture which requirements are “located”. Requirements traceability is an essential value, especially when assessing, refactoring or evolving a system. I am assuming here that architects already checked for the feasibility and essence of the requirements from a business perspective. Availability of 99.99999% is not reasonable in a project with a budget of 100k bucks.
Introduce symmetry and conceptual integrity. This includes the prescription of specific patterns, the introduction of design and coding guidelines, as well as the introduction of guidelines for (at least the most important) non-functional qualities. This prevents that the same problem is solved in various ways within the same solution space and it makes sure, we are applying best practices from experts instead of reinventing the wheel. War story: A system where designers have introduced dozens of ways for fault handling will be inexpressive and complex.
Don’t consider technologies as toys. Many projects play with technologies. They base their decisions on a technology-first approach. This is human and we as software engineers like to experiment with the new hip operating system or SOA middleware. But in practice, we should first create an appropriate architecture meeting the requirements (see first bullet point) before we should start introducing new technologies. The business and the requirements drive the architecture, not technology.
Test technology. The previous point does not mean we should not care about technology – only that we first address the requirements. Every technology must be checked for its appropriateness. If your task is to build a high performant messaging backbone, then you better check whether EJB is really the proper platform to use. Don’t trust any vendor’s promises here! They do not know your problem context and system environment. Building throw-away prototypes is the best way to check technologies.
Apply a test-driven development approach: this is to introduce a kind of safety net. Emphasize testability in your architecture design, e.g., by proper modularization, interfaces, and service contracts. Test-driven in this sense also includes regular design and architecture assessment where you try to identify architecture smells.
Regular architecture refactoring helps address architecture issues early. When a dependency cycle has just been introduced unintentionally, it is not a big issue to detect this kind of design erosion early with architecture reviews or architecture analysis tools. You are then able to get rid of the problem early by architecture refactoring. The more you wait, the more architecture entities will make it difficult to address the problem without heavy impact. Thus, do regular architecture reviews and refactorings.
Strategic before tactical design. Don’t try to introduce over-generic solutions with myriads of configuration options. First address all strategic requirements (functional requirements and operational requirements) before dealing with tactical issues such as modifiability. Use the Open/Close principle to open your architecture for variation.
Introduce priorities for all requirements/features. The priorities help decide which requirement to address first in architecture design. If security is more important than performance, you’ll end up in a completely different solution architecture, then wheny performance has the higher priority.
Follow an approach of piecemeal growth. In all but the most trivial applications a big design upfront approach will be doomed to fail. Thus, assign requirements depending on their priorities to iterations and build your system incrementally.

These are only a handful of recommendations. Certainly, there are even more. Any tipps from your side?

Get off of my Cloud

2009-03-28T14:00:00.001+01:00

It enters the stage as one of the great buzzwords of the last months and if we had not to deal with an economic crisis these days it would gain much more attention. Cloud Computing and its relatives such as SaaS are dominating the IT world. Ever heard about Windows Azure, Google Web Apps, Amazon E2C and S3? The idea behind this paradigm is rather old: why not provide resources transparently as services in a hosting infrastructure? This would let IT departments and CIOs refrain from being infrastructure providers. Instead they could focus on their actual job: providing IT functionality to drive their company's business.

By services I don't just mean applications, but all kinds of services. An application such as a CRM app could be a service. Likewise, services could comprise a piece of middleware such as a persistence solution or a workflow management system. And in the most extreme case, even the infrastructure itself, for example, operating systems, storage or network solutions might be deployed as services. Services where ever you look.

The idea behind cloud computing is a very promising one. Just take trusted infrastructure providers, let them host the applications (services) we need within an Internet-based cloud, and access these services from an inexpensive device. No need to buy or maintain your own data center solutions anymore. A data center on everyone's desktop if you like. Whenever you need more resources, just take them from the cloud and only pay for what you use. "Elastic computing" made real. Sounds like a good idea in an economic crisis, doesn't it? The strength and weakness of Cloud computing today is its dependence on external providers. Would you run your mission-critical applications with sensitive data on a 3rd party infrastructure that is not under your control and which supports multi-tenancy? What about services running in a remote country that you do not trust? How can a provider guarantee quality-of-service in an Internet-based infrastructure? Remember all the recent downtimes of Google Mail. What if all your company mail is running on such a network?

These problems imply a strategy where mission critical applications only run in a private cloud, while only a fraction of less critical applications may leverage public clouds. As soon as the Cloud Computing technology evolves and matures, more and more services might be moved to public clouds. Then, private clouds could also be extended using public clouds. By the way, this extension approach is only feasible if standards foster interoperability of cloud solutions.

From 10000 feet Cloud Computing denotes sort of a business solution that introduces pay-per-use models for all types of services. Regarding technology this solution is based upon a bunch of existing paradigms such as virtualization, SOA, Multicore CPUs, NAS, or Web 2.0.

What all those people enthusiastic about Cloud computing tend to forget, is that there is no free lunch. It, indeed, represents a promising technology with a high coolness factor. However, from a software architecture perspective, it is rarely sufficient to just deploy application silos in virtual machines that are hosted somewhere else in the network. To really leverage clouds your applications need to be aware of the cloud infrastructure. You might split islands of functionality into interconnected services, and integrate application services with other platform or infrastructure services. What about issues such as collocation of services and data? In addition, engineers need to create desktop-like GUIs that communicate with backend services as promoted by Web 2.0. Last but not least, quality of service such as availability or responsiveness does not only depend on infrastructure and platform but is heavily influenced by the solution architectures. The best infrastructure or platform does not help if the software architecture is a bunch of crap.

Cloud Computing offers fascinating new possibilities. But we shouldn't suffer from the hammer-and-nail syndrome. When applied inefficiently Cloud computing will inevitably lead to problems and disappointments. If your management or CIO department is all of a sudden promoting Cloud computing, this might be a good thing, but it also requires you as a software engineer to deal with expectation management. And finally, mind the architecture!

My Domain is my Castle

2009-03-27T19:30:00.001+01:00

One of the most important (first) steps in architecture modeling encompasses the description of the domain model. This model introduces all entities relevant within the current domain as well as their relationships and interactions. It represents the main language all stakeholders should understand. All further activities in architecture modeling basically take the domain model and enrich it with additional infrastructure entities.

Sounds very abstract, right? Let me give you a concrete example. Suppose, we are going to develop a Web store. What are the typical objects that appear in the problem space and are well known to all stakeholders?

For example, I'd expect entities such as:

web store user: someone accessing the web store
customer: someone interested to buy items
shopping cart: used to add, remove, pay goods
product catalog: presents all available products classified by categories and allows to search for these products using different types of query
customer database: place where all customer information is stored
order processing: responsible to process orders

If you think about this example further, you'll recognize there are some typical relationships between domain objects. For example: a web store user could be a customer or an administrator. A customer typically owns at most one shopping cart at the same time.

You also can easily defer some interactions between the entities. It is obvious there must be a relationship between the shopping cart and the product catalog, because the information about purchased items is read and updated from the product catalog. Of course, the shopping cart needs to interact with the order processing system after the customer has pressed the order button.

With other words: when thinking about how use cases would be mapped to sequence or interaction diagrams, the knowledge about the domain model is essential. It is the step taking you from a black box perspective to a gray box perspective.

There are different ways to express such a domain model. You could invent a graphical representation or use a textual language instead. If a domain object model gets formalized, we call this a DSL (Domain-Specific Language). In this case, engineers could even provide generators that map from the problem domain to the solution domain, i.e., that map problem domain objects and relations to solution domain objects and relations.

Using domain models offers huge advantages:

it introduces a common vocabulary among all stakeholders which supports effective discussions and often prevents a steep learning curve for domain dummies
it reduces the chance of missing to address important parts of a domain
it eases architecture modeling significantly
it helps focusing on the problem domain instead of always diving into the solution domain
when enriched with domain patterns (analysis), it boosts productivity

It is neither necessary nor useful trying to come up with a complete domain model in the first place. In most cases, there will be an initial, maybe incomplete, domain model which engineers together with the other roles will evolve over time.

There are several examples, where the domain model is already available. Think about GUIs, compilers, and some kinds of healthcare domains. In these domains, it is useless and a complete waste of time to come up with your own domain model.

So, whenever you are being involved in a new software architecture, always mind the domain object model! It will be your best friend.

Software and System Architecture

2009-03-21T12:24:00.001+01:00

Currently, I am involved in a large-scale project where Siemens Healthcare is building a couple of very innovative and exciting particle therapy systems. These systems are rather challenging as they consist of complex software, hardware - for example, a circular particle accelerator - and all the other constituents such as the building, the CT devices. It is a typical project where software while being a major element is only contributing a small part of the system. However, within our company which yields revenues of over 70 billion euros a year, software engineering has become a major business factor. According to an estimation 60-70% of these revenues already depend on software. Now imagine, 10% or even more of our projects would fail - what a nightmare! There are two implications: first of all, lead software architects must be well educated in software architecture and technology and show the right leadership skills. Secondly, system and software architecture must be treated as two sides of the same coin. So, what are the responsibilities of software architects and system architects and how should they cooperate? I tried to find an appropriate definition in Wikipedia and failed. Especially, system architecture seems to be not well understood or ambiguous.

From my viewpoint an effective partitioning in software and system architecture depends on the characteristics of the project itself. If software constitutes the main part of the product or solution, then the lead software architect could and should also be system architect. Obviously, an additional IT architect makes a lot of sense if the underlying infrastructure is sufficiently complex, but that's a different story.

In projects where hardware and software are of the same relevance or where hardware dominates, for example in most embedded systems, a system architect should be in the lead and co-operate with a hardware and a software architect. The system architect then should care a lot about testing. I have been involved in projects where I was in charge of the software system but got only rarely access to the target system. Thus, we had to use the development system where everything worked stunningly fine, but got deadlocks when finally moving to the target system - the OS of the target system had a faulty implementation of the Sockets libraries. This implies, that the system architect should allow software and hardware development to happen almost independently, but introduce a lot of sync points where software/hardware integration is checked thoroughly. By the way, a similar problem occurs when partitioning a software system into modules which are then designed and implemented by different teams - think of outsourcing in the extreme case. If integration and system testing is neglected, you will encounter a lot of bad experiences. It is much more complex and challenging to check hardware/software integration than the integration of software modules - at least in theory :-; One of the bad experiences you might need to cope with is that hardware requirements are often considered almost fixed, while software requirements are considered "soft". Would you ever change the main specifications of an airplane shortly before it is delivered? Definitely not! But this does not hold for software systems. One of the consequences of this fact might be that the hardware team has already finished their part, being reluctant to change anything. As the software requirements have changed in the project, software development has fallen behind schedule. In these situations, software architects sometimes become the scapegoats of system development. A good system architect should be capable of handling these situations effectively and efficiently by better synchronizing the different teams.

A system architect should also make sure that the overall system requirements are mapped consistently and completely to software architecture and hardware related requirements. He or she should care about requirements traceability, especially for those requirements that have to be implemented in software and hardware as well. If hardware comprised the core part of the system, software architects should have a sound understanding of the overall system, not just of the "soft" parts.

Basically, you could consider the partitioning of software and system architecture responsibilities in almost the same way like splitting responsibilities between software architects and the lead architect. The system architect is in charge of the whole system, while the software architect is in charge of a part of the system.

There is a lot more to say about this topic. But I am curious what you think and I am wondering about your experiences with software and software architecture as well as with the respective roles. Any comments highly appreciated.

News from the Engineering Frontier

2009-03-18T20:33:00.006+01:00

I have been to the Canary Islands for a month. Since then I have been totally busy in two projects. One of them is dealing with medical particle therapy solutions, while the other one consists of setting up a certification program for software architects within Siemens. I am in charge of this program. Last week I was giving a seminar on software architecture with 12 guys from Siemens. I had a really great time and hope the same holds for the participants.

It is time to post more on this blog. Yes, I feel guilty for being so lazy. But you know, I am in love with Work-Life-Balance. To be honest, I needed a timeout from all software engineering business for a while, instead focusing on sports activities and digital photography. Now, I am really motivated to enter the software architecture bandwagon again. Last monday, we (= Markus Voelter,Stefan Tilkov, Christian Weyer, and myself) started with a new podcast series on software architecture. This will be in german with some company sponsors. I will keep you informed about the details when things mature. I am wondering whether a software architecture podcast in english would make sense.

In the medical particle therapy project we have just defined a possible partitioning between system architecture and software architecture. Seems to be a major source of misunderstandings in many projects. Same for differentiating problem and solution domain. Engineers seem to be addicted to the solution domain drug. Instead of thinking hard about the problem domain we always try entering safe ground, in particular by addressing solutions very early. The problem is: how can we define a solution when we don't understand the problem? On the other side, bubbles don't crash, as we also know. We could endlessly define UML diagrams without ever dealing with implementation. As Bertrand once said: all you need is code. Thus, we need to ensure the feasability of our solution very early. How can we satisfy both goals? Needless to say, agile development is the right answer.

Of course, there are a lot of additional issues when addressing the boundary between system and software architecture. This will be subject to upcoming posts, All feedback and comments as always highly appreciated

Views on a Cat

2009-01-20T00:43:00.001+01:00

Suppose, you're going to design a new software system for a specific domain. One of the first steps is to introduce a domain object model which comprises the core entities in the domain as well as their relationships. That's easy, you might say. If it is that simple, why do so many projects fail in establishing the right domain object model? And, of course, this challenge inevitably appears in all kinds of modeling activities.

But now for something completely different - as Monty Python would say. I got two nice cats at home. In order to model a cat, there are different possibilities:

(ATOM) We could consider a cat as a mere collection of molecules.
(GENE) Obviously, a cat can be uniquely identified using its DNA.
(BODY) Another approach is modeling a cat as an aggregation of subsystems such as legs, joints, muscles, intestines.

Does it make sense to apply the first model (ATOM) in order to understand how a cat moves? No, because the detail level is overwhelming.

Is it useful to use the third model for understanding all chemical activities within a cat? Definitely not!

With other words, we can model the same physical concepts in different ways, each of these views strongly depending on the purpose of the model. Don't let yourself be confused. It is not as simple as having only one model for one purpose. For example, to understand the physical appearance of a cat, you could apply model ATOM as well as model BODY.

What are the implications of this observation?

First of all, you might require different models (views) in order to define and understand a software system.
Secondly, each model should be motivated by a concrete and useful purpose.
Thirdly, all participants should agree on syntax and semantics of the model.
And last but not least, all models should be documented.

Mind the word "useful" in the second bullet point. As organization drives architecture, the organization often directly maps to entities in the model. If your organization is badly structured, your architecture will reveal the same problem. Think about this in the context of your own organization.

As another consequence I'd like to emphasize that it is not sufficient just inventing a couple of sophisticated models. The set of models should also be simple, complete and consistent. All of the models should complement each other in a meaningful and appropriate way. That's the reason why we got view sets such as the UML 4+1 view.

Unfortunately, I often sit in project meetings where people strongly believe they are all sharing the same view of their domain, while in fact they are not. Endless discussions might be an indicator for this problem. Thus, it is essential to explicitly come up with a common domain model that is agreed among all participants. Basically, the design of such a model should be one of the early steps in a development project (directly after scoping but before architecture design). Needless to say that software engineers are surrounded by models: they have specific views of the problem domain and the solution domain, use tools such as compilers or database systems that themselves are built on top of models.

If a software developer's life is about models and model transformations (e.g., the mapping from the problem domain to the solution domain) we should consider models as first class citizens and invest sufficient time for choosing the right viewpoints and establishing the right models.

Interestingly, even our view of the physical world such as the way we understand a cat is determined by models. But this is a different story.

Archidictatorship versus Develepocracy

2009-01-09T16:55:00.001+01:00

In the software development projects where I am involved as an architect I can typically observe different types of software architects. Let me illustrate two extremes:

Some architects try hard not to decide anything or at least not too early. These architects typically integrate all other project participants in the decision process. On one hand, this can be a very successful approach. If everyone agrees to important decisions, everyone remains motivated. On the other hand, the same habitude leads to the risk of decisions being postponed endlessly. Eventually, architects constantly drawing new UML diagrams or deep diving into technical details are a problem not a solution. In most cases, wrong decisions are better than no decisions, because wrong decisions can be detected very early and all problems resolved. Think of refactoring as a tool!
Other architects prefer a kind of tyranny. They got a lot of self esteem, believe to know everything, adore the Borgs for their "resistance is futile" strategy. This style of leadership can come very handy in critical situations where project success depends on strong leadership, for example when you require fast decisions. The downside of this dictatorship is that architects behaving this way may not get buy-in from other participants, thus leading to a high level of demotivation. And obviously, too much power in the hands of the wrong persons can lead to projects being doomed to fail miserably. A negative implication might be that those dictators want to get involved in every decision, even in unimportant ones. As no one is able to understand all details in the case the project is large enough, you'll inevitably will experience lots of wrong decisions.

You will argue that these leadership styles are not specific to software architects, and, of course, you are absolutely right!

What I'd like to emphasize in this posting is the importance of leadership skills for software architects. Of course, there are situations where an architect should be dictator, and other situations where democracy might be the more appropriate choice.

Unfortunately, or should I say fortunately, all of us got different personalities. Your leadership style should fit with your personality, because otherwise everyone will recognize the discrepancy.

I won't cover leadership styles in too much detail because there are persons much more knowledgeable than I in this context. Just read this one or that one. A Web search will even yield more material.

What I consider really helpful is a kind of self analysis. For instance:

Personality: analyze your own personality, try to find your strengths and weaknesses. Ask other people how they assess you as an architect. How do you evaluate yourself acting as an architect? A very helpful tool might be to analyze previous projects: where did you succeed and why and where did you fail.
Leadership Styles: read about all possible leadership styles. Nothing more to add here.
Context: Think in your project context what leadership style should be applied in which situation. Of course, you can't act as a dictator in front of your head of R&D. Likewise, always striving for full agreement by every developer can result in endless recursion.

For a software architect, the time spent for communication might be as large as 50%. Communication means interacting with other people, informing them, guiding and mentoring them, or getting their buy-in. Without the right social skills and leadership style, a software architect is doomed to fail.

Panic mode

2008-12-03T12:24:00.003+01:00

If you carefully observe the current economic crisis, you'll recognize that some market participants are acting in panic mode. This comes to no surprise if you ever have read books like those by Jose Ortega y Gasset. Unfortunately, panic mode is never helpful because it leads to spontaneous activities that are not motivated by thorough and systematic planning. Instead, people tend to think they have to react quickly and often draw the wrong conclusions which worsens the situation instead of improving it. If you look at some current stock values such as Intel or Siemens the situation is somehow out of control. With other words: it could be a could idea to buy stocks now. Maybe, Douglas Adams was wright when he once mentioned that nothing can travel faster than light with one exception: bad news!

If you look at software development projects, you'll often find the same issues. As soon as deadlines are getting delayed or dropped, participants are increasingly starting to get nervous. After a while the whole organization is typically entering panic mode. There is a possible alternative flow which I call Lemmings mode. This is when people refuse to recognize they are facing some severe problems and continue the project without any countermeasures which denotes a typical human behavior.
As an alien you'll recognize panic mode in an existing project by encountering different indicators. For example, if meeting frequency is high, and architects and developers spend more time for meetings than for productive work, this might either be a clue for developer's hell or for a project in panic mode. The same holds for situations where the organization is constantly generating new or enforcing uncontrolled and unplanned activities (actionism) in a higher rate than they can be processed. If you interview people in such a project context, you'll receive inconsistent and contradicting statements, because there is no synchronization or coordination in place.

Although it is obvious that panic mode worsens the situation and even leads to conflicts among the persons involved, projects fall into this trap pretty often. I believe, this is caused by the psychological fact that all participants in such a project are caught in the trap. In these situations, I generally recommend to involve an external person who is not part of the problem (aka as project) and acts more as an external observer and mediator. In addition, this mediator needs a peer in the organization's senior management able to enforce all recommendations (if accepted), because otherwise all suggestions would be ignored. To be honest, architecture reviews or other forms of assessments work exactly that way.

A way to address problems very early in projects is to use an agile approach, especially to elaborate what risks exist and how to address them systematically. In contrast to common believe such risks are non constrained to architecture and technology but also extend to politics, partner and vendor relationships, organization, quality, development processes. Risks analysis is not just about recognizing the risks but also about strategies and tactics for addressing them.

By the way, this is exactly what currently happens in the economic crisis. Since market participants don't seem to be able to address the challenges themselves, politicians and central banks are forced to bring countermeasures in place. All the risks in the market have been subject to constant ignorance.

The later risks are detected, the more likely the project will enter panic mode. This is why I always recommend regular flash reviews of architecture, code, test strategy, development process (for example, at the end of an iteration). They represent means for detecting problems very early and for getting rid of them (for example, by refactoring).

To cite Douglas Adams: DON'T PANIC!

Why the waterfall model does not work

2008-11-07T17:57:00.001+01:00

Again and again I have met managers and software engineers who tried to convince me that agile processes while being appropriate for somebody else's project were simply not applicable to their problem contexts for several reasons. They came up with rather lame reasons like being used to waterfall models or the claim that agile processes do only work for small teams. Especially I am enjoying people's reaction when I am talking about really huge projects I had been involved and which followed the Scrum method.

From my viewpoint the question shouldn't be whether to use agile development processes instead of a waterfall approach. I would rather suggest to let all those waterfall addicted prove why they consider waterfall approaches more suitable in terms of a concrete project.

It is true that an agile method is not superior in planning projects. Even in an agile context you will need to estimate resources, budgets, or time-frames in advance. But in contrast to a waterfall approach you get an early feedback if your initial planing proves to be plain wrong: If after the third iteration you got an substantial delay you are able to react with appropriate countermeasures. In a waterfall model the old saying is that while 80% of the project have been completed, you still need to address the remaining 80%. You may be doomed to fail at the beginning of the project but a waterfall model lets you live with this illusion till the sad end. An agile method on the other hand just provides a whole range of safety nets. You may fall but contact with ground usually is much smoother.

Let me take it the other way round. What are the preconditions to make a waterfall model succeed?

All requirements are available at project start and will remain unchanged.
All of the requirements are consistent and complete.
No errors will be introduced in any phase of software development.
Likewise, all tools, legacy components, infrastructure parts fully adhere to their specifications and, of course, are consistent and complete themselves.
All stakeholders start with a full understanding of the problem domain. Developers, testers, architects also have a full understanding of the solution domain.
There will be no misunderstandings in any communication between stakeholders.

Obviously, most of these preconditions never hold in any real-life project. We have to cope with error and failure. In agile approaches, the principle of piecemeal growth helps to master complexity. Safety nets such as test-driven development or reviews enable quick detection of potential problems. Means like refactoring support us in addressing the problems we detect and to embrace change. Agile communication and customer participation help in keeping all stakeholders synchronized.

Two of the main principles of agile engineering might be called "embrace change" and "learning from failure".

And if we look at nature, evolution basically also represents an agile approach. For all creationists: evolution does not necessarily rule out that there could be a master plan behind everything.

What's your take on the agile versus waterfall dispute?

Integration

2008-10-24T21:05:00.001+02:00

Integration represents one of the difficult issues when creating a software architecture. You need to integrate your software in an existing environment and you typically also integrate external components into your software system. Last but not least you often even integrate different home-built parts to form a consistent whole. Thus, integration basically means to plug two or more pieces together where the pieces were built independently and the activity of plugging requires some previous efforts to make the pieces fit together. The provided interface of component A must conform to the required interface of component B. If these components run remotely, interoperability and integration are close neighbors.

For complex or large systems, especially those distributed across different network nodes, integration becomes one of the core issues.

Integration necessities of external components or environments should be treated as high-priority requirements, while internal integration often is caused by the top-down architecture creation process. Examples for external component integration comprise using a specific UI control, running on a specific operating system, requiring a concrete database, or prescribing the usage of a specific SOA service. Internal integration comes into play when an architect partitions the software systems into different subsystems that eventually need to be integrated with each other.

Unfortunately, integration is a multi-dimensional problem. It is necessary to integrate services, but also to integrate all the heterogeneous document and data formats. It is required to integrate vertically such as connecting the application layer with the enterprise layer and the enterprise layer with the EIS layer. Likewise, we need to integrate horizontally such as connecting different application silos using middleware stacks such as SOAP. In addition, shallow integration is not always the best solution. What is shallow integration in this context? Suppose, you have developed a graph structure such as a directory tree. Should you rather create one fat integration interface with complex navigation functionality or better provide many simple integration interfaces for all nodes in the tree?

What about UI integration where you need to embed a control into a layout?

What about process integration where you integrate different workflows such as a clinical workflow that combines HIS functionality (Hospital Information System) with RIS functionality (Radiology Information System)? Or workflows that combine machines with humans?

What if you have built a management infrastructure for a large scale system and now try to integrate an addition component that wasn't built with this management infrastructure in mind?

Think about data integration where multiple components or applications need to access the same data such as accessing a common RDBMS. How should you structure the data to meet their requirements? And how should you combine data from different sources to glue data pieces to a complete transfer object?

Semantic integration is another dimension where semantic information is used to drive the integration such as automatically generating adapter interfaces for matching the service consumer with the service provider.

And finally, what about all these NFRs (non-functional requirements)? Suppose, you have built a totally secure system and now need to connect with an external component?

It is needless to say that integration is not as simple as it often appears in the beginning. Even worse, integration is often not addressed with the necessary intensity early in the project. Late integration might require refactoring activities and sometimes even complete reengineering if there are no built-in means for supporting "deferred" integration - think of plug-in-architectures as a good example for this.

Architects should explicitly address integration issues from day 1. As already mentioned, I recommend to place integration issues as high-priority requirements into the backlog. Use cases and sequence diagrams help determining functional integration (the interfaces) as well as document/data transformation necessities (the parameters within these interfaces). During the later cycles in architecture development, operational issues can also be targeted with respect to the integrated parts. Last but not least, architects and developers need to decide which integration technologies (SOA, EAI, middleware, DBMS, ...) are appropriate for their problem context. Integration is an aspect crosscutting through all your system!

Agile processes consider integration as a first-class citizen. In a paradigm that promotes piecemeal growth, continuous integration becomes the natural metaphor. From my perspective, all software developments with high integration efforts must inevitably fail when not leveraging an agile approach.

To integrate is agile!

Trends in Software Engineering

2008-10-03T22:29:00.001+02:00

It has been a pretty long time since my last posting. Unfortunately, I was very busy this summer due to some projects I am involved in. In addition, I enjoyed some of my hobbies such as biking, running and digital photography. For my work-life balance it is essential to have some periods in the year where I am totally disconnected from IT stuff. But now it is time for some new adventures in software architecture.

Next week I am invited to give a talk on new trends in software engineering. While a few people think, there is not more to discover and invent in software engineering, the truth is that we are still representatives of a somehow immature discipline. Why else are so many software projects causing trouble? I won't compare our discipline with other ones such as building construction because I don't like comparing apples with oranges. For example, it is very unlikely that a few days before your new house is built, you're asking the engineer to move a room from the ground floor to the first floor. Unfortunately, those things actually happen in software engineering on a regular basis.

A few years ago I wrote an article for a wide spread german IT magazine on exactly the same issue. I built my thoughts around the story of a future IT expert traveling back with a time machine to our century. When he materializes, a large stack of magazines falls on his head - unintentionally - and he cannot remember anything but his IT knowledge. How would such a person evaluate the current state of affairs in software engineering? Do you think, Mr. Spock from spaceship Enterprise will use pair programming or AOP? Wouldn't look very cool in a SciFi series, would it?

To understand what software engineering is heading for in the future, it is helpful to understand that all revolutions always are rooted in continuous evolution. At one point in time, existing technology cannot scale anymore with some new requirements which forces researchers to come up with new ideas. Mind also the competition challenge. The first one to address new requirements may be the market leader. Learning from failure is an important aspect in this context. So, what is currently missing or failing in software engineering? Where do we need some productivity boosts? An example for such new requirements might be new kinds of hardware or software. Take Multi Core CPUs as an example. How can we efficiently leverage these CPUs with new paradigms for parallel programming.

As another example I consider the creation of software architecture. In contrast to many assumptions, most projects develop their software architecture still in an ad-hoc manner. We need well educated and experienced software architects as well as systematic approaches to address this problem of ad-hoc development. Especially, the agility issue is challenging. Software engineers need embracing change in order to survive. Architecture refactoring is a promising technology in this area, among many others, of course.

Another way of boosting productivity is to prevent unnecessary work. Here, product line engineering and model-based software development are safe bets. If the complexity of problem domains grow we cannot handle the increased complexity with our old tool set, We need better abstractions such as DSLs. And I simply cannot believe for the same reason that in a world with an increasing amount of problem domains, general purpose languages are the right solution. Wouldn't this be another instance of the hammer-nail syndrome? Believe me, dynamic and functional languages will have increasing impact.

In a world where connectivity is the rule, integration problems are inevitable. In this context, SOA comes immediately to my mind. With SOA I refer to an architecture concept, not to today's technology which I still consider immature. New, better SOA concepts are likely to appear in the future. SOA will evolve such as CORBA and all other predecessors did. Forget all the current ESB crap which already promises to cure all world problems.

There is a lot more which will influence emerging software engineering concepts. Think of grids, clouds, virtualization, decentralized computing! Look at CMU SEI's document on Ultra-Large Scale Systems Linda Northrop was in charge of. You'll find there sufficient research topics we have to solve before ULS will be available.

For us software engineers the good news is that we are living in an exciting (although challenging) period of time.

The Silver Bullet

2008-08-28T10:39:00.004+02:00

As I just read in a comment about silver bullets, I couldn't resist to address this topic :-)
Frederick F. Brooks brought up the concept of silver bullets. Of course, Frederick was not refering to some weird persons who are turning into computer geeks during full moon and can only be stopped using these nice little silver bullets. Instead, he claimed that there never ever will be any software technology that can boost software development productivity by a factor of 10 or more. So far, he has been right. Object-Orientation? Really productivity-boosting, but not to an order of magnitude. Models? Nice try, but someone has to build the model and the generator.
But won't there be any technology in the future that can refute this thesis? Suppose, an alien lifeform with incredible technology is visiting planet earth. How would they design software systems? Do we really believe, they would leave their giant space ship with a notebook, start modeling their systems with a kind of UML tool, and program in Java?
Thus, could the Silver Bullet be a universal law, or just a kind of constraint only humans are facing due to their limited capabilities?
Frederick F. Brooks also brought up the concept of accidental versus inherent complexity. Basically, each problem has an inherent complexity. You can't specify a a linear algorithm for the general sorting problem. Forget it, there is really no way! But we ourselves are causing a lot of problems by accidental complexity. Using an array for storing a sparsely populated 1000x1000 matrix isn't a very smart approach, right?
No technology can get rid of inherent complexity, but it could hide this kind of complexity from the user. Hiding complexity is one way to increase productivity - just make it a SEP (Somebody Else's Problem). Helping to address accidental complexity is another. Consequently, the best productivity boost results from combining both.
According to philosopher Thomas Kuhn, evolution of technologies is more a linear process. A technology can only scale upon specific limits. If it reaches the limits, workarounds will help postponing the necessity for a new approach. But at some point in time, a completely new and revolutionary approach will appear (such as switching from structured programming to OOP) and the same kind of technology cycle will restart. It is a kind of spiral model if you will.
The only point where we could observe productivity boosts are the technology revolution events. However, new technology revolutions are mostly rooted in existing technologies. Thus, an immediate productivity gain is simply not possible.
But as a thought experiment: if you just compared the way of approaching software development challenges now and 40 years ago, you could recognize a tremendous increase of effectiveness. There is a long way from software engineering in the sixties to that of today. No person could have skipped all the intermediate results and challenges.
Thus, if an alien race would land on earth, it appears very likely they could introduce us to new ways of building software systems that offers magnitude of productivity increase.
Conclusion: Silver bullets are theoretical beasts only existing in the Star Trek Universe. As pragmatic architects we must leverage the tools and technologies existing today. And we shouldn't believe all those technology panaceas vendors and market analysts are constantly promoting such as SOA, SaaS, you name it.

Now it is proven - NESSI exists!

2008-08-12T21:52:00.001+02:00

ok, but what is it?

According to its web site, "NESSI is the European Technology Platform dedicated to Software and Services. Its name stands for the Networked European Software and Services Initiative."

Yes, it is not very clear from this definition. So, let me try my own explanation.

Suppose, you'd like to leverage SOA to connect heterogeneous islands. Maybe, you are going to build a healthcare system that connects different international regions, allowing physicians to access your healthcare data if required and permitted. An underlying SOA infrastructure should provide:

SOA based middleware and process modeling
Enterprise Application Integration facilities
a kind of internet bus that supports various protocol families
registries and repositories
horizontal services such as security

Sure, you can obtain such ingredients, if you accept vendor lock--in. I don't list all the big SOA proponents here as they are quite obvious. The truth is that is almost impossible to cope with such heterogeneity. If you try, you will be soon overwhelmed by sheer complexity.

Unfortunately, there is another dimension to this problem:

Even if you have all these facilities at hand, you need to standardize the vertical domains. What if the whole SOI (Service Oriented Infrastructure) is standardized but two banks don't agree in what a customer or bank account is?

So far, solutions have either attempted to address the technology issue or to address the domain standardization.

What NESSI tries is to address those challenges.

It is required to create a kind of standardized SOA operating system and to define domain-specific standards for how to create applications on top of this platform. From my personal viewpoint the combination of Open Source Software with standards would be a perfect fit. It is necessary to create both, standardized APIs and a reference implementation able to handle mission critical applications.

Is a platform sufficient? Absolutely not, because we also need to cope with service engineering and governance issues and non functional requirements and tools. Consequently, NESSI is also trying to address these points.

All of these aforementioned problems are hard to solve but on the other hand we have to solve them anyway, because they root in inherent complexity. If we need to solve them to finally live the dream of "the network is the computer", then NESSI represents an excellent opportunity.

But that's my personal viewpoint. What is your's?

Michael's Event Calendar

2008-08-12T17:22:00.001+02:00

In September I will visit my colleagues in Bangalore and give a seminar on Distributed System Architecture.
On 14th October I will give a keynote on COMPARCH 2008 in Karlsruhe. I was invited by Ralf Reussner and Clemens Szyperski, two well-known gurus on architecture and components. Guess, what topic I will cover? Yes, architecture refactoring, my preferred topic these days.
Originally, I planned to attend this year's OOPSLA conference in Nashville. I have two accepted tutorials, one on high quality software architecture, the other one on Software Architecture Refactoring. Unfortunately, I am so busy within Siemens that I won't make it to the OOPSLA. However, an excellent member of my staff will give the two tutorials instead. Marwan is not an emergency solution but an excellent architect himself, formerly working for Lufthansa Systems. You will have a lot of fun attending our two tutorials. Mentally, I will be there, of course :-)
In November I will most likely be invited to the Microsoft TechED in Barcelona giving two talks.
In December I will be available on a panel at the Service Wave Conference of the European NESSI initiative.
In January 2009 there is my local event, the OOP 2009 in Munich. I don't yet know what topics I will address but be sure I will be around :-)
I am planing to submit for next year's ACCU in Oxford (April).

Mojave - Episode II

2008-08-12T17:03:00.001+02:00

Did you hear about Microsoft's Mojave Project where they told a bunch of selected people - most of them Vista critics - of a cool new operating system called Mojave? Supposedly, most of them were kind of enthusiastic about Mojave. To their big surprise, Mojave turned out to be Vista - the devil in disguise if you will in remembrance of an old Elvis hit.

What can architects learn from this?

First of all, we can hide all the mess of our systems behind a nice look and feel. The best software system will not be accepted if its usability sucks, and vice-versa.
Secondly, if we got an excellent system that people for some strange reasons don't appreciate, we could try to put a new API or Look & Feel on top. Maybe, the interface is the only thing that sucks in our system.
Last but not least, we could also learn that nice architecture and design diagrams might impress others so much that they consider the system as a neat piece of design while in fact the implementation is totally different from the design.

Regarding the original Mojave experiment I really doubt that the reviewers had the write level of expertise and mind set. Don't judge a software system by its cover! Review and evaluate the design and the implementation thoroughly.

Architects shouldn't just believe but rather evaluate and enforce.

But of course, in some stakeholder presentations such as presenting the system to your top management you can add a marketing layer while remaining politically correct and honest.

Art, Craft or Science?

2008-08-11T23:11:00.001+02:00

Is software design an art, a craft or a science? This question has been the starting point for many entertaining discussions. And many different opinions and answers exist.

So what is my take on this?

To build a software system requires engineers to come up with a mapping that takes the problem and maps it to a concrete solution. Requirements describe the problem and the desired properties of the solution. Unfortunately, the mapping typically is not unique in that there might be many solutions solving the same problem. With other words, architects and developers face a lot of freedom. The bad thing about freedom is that it always forces architects to take decisions. In other words, to architect is to decide.

Of course, we appreciate if we can immediately find an appropriate solution for all problems, just following a divide-and-conquer strategy of software development. But this never happens in practice.

Two problems come to my mind in this context:

Sometimes we encounter problems for which we neither know a solution nor can find a solution elsewhere (finding a solution means either knowing a direct solution or being able to map the actual problem to another problem for which you already know a solution - for example, if we are able to map our algorithmic problem to a sorting problem.)
In contrast to mathematics, the problem itself is a moving target in software engineering. Pantha rhei.

The first issue implies that you may not find a pattern or some other kind of best practice for your concrete problem context. Instead, you need some intuition. Hopefully, you recognize that the problem at least resembles another kind of problem for which you already have a vague solution idea so that this can serve as a good starting point. By the way, this is the reason why it turns out to be so hard to build a general problem solver for software engineering. Freedom is very hard to deal with by a general problem solver.

The second issue, coping with change, requires some sort of agility. In contrast to logic, our axioms keep changing while we are applying the calculus. The only way to deal with change is to embrace change everywhere, in the process, in the tool chain, in designing, in testing, in refactoring, in programming, in communicating.

What makes software design so hard sometimes is when both problems appear in parallel.

When people ask whether software engineering is a craft, a science, or an art, the answer should be obvious now. It is a science as we try hard to follow formal approaches where feasible. It is a craft because solving problems often requires experience and best practices. And it is an art as software engineering leaves enough freedom for intuitive and even artistic solutions.

I guess this is what makes our job so interesting.

Architect always implements

2008-08-08T23:20:00.001+02:00

When I once invited Bertrand Meyer to our location in Munich, he gave an excellent talk on Software Engineering. His most memorable messages were "Bubbles don't crash" and "All you need is code".

With the first message "Bubbles don't crash" Bertrand referred to all those engineers who have turned into Powerpoint or UML freaks. A Powerpoint slide or UML diagram will never crash (if you don't use it as a model for code generation, the MDSD guys might argue). It is surprisingly easy to impress people with simple boxes and lines (sometimes dubbed software architecture). Especially it is rather simple to impress top management which is why the wrong consultants can potentially cause so much damage. Some might respond that high abstraction levels are the only way to understand and build complex systems. That's certainly true. On the other hand, you would not be able to build a ship if you never had any idea about water. The same holds for software systems. It is simply not possible to develop a software system without any idea on top of which runtime environment it is supposed to execute.

Even if you had in-depth knowledge of your software system's runtime environment and built an excellent software architecture, it is NOT a no-brainer to transform a software architecture to a working implementation. That's why Bertrand introduced the "All you need is code" attitude. What helps you the best theory, if you can't turn it into practice? As an architect you need to have implementation skills, either for implementing parts of the system yourself or for assessing the results of other team members. How could you ever build a good bicycle without biking yourself?

The problem arises how to ensure that we as software architects can meet these requirements. "Architect always implements" means that you should also implement in a project, but definitely not on a critical path. By implementing you will earn the respect of developers, while keeping in touch with the implementation at the same time. And you will have to eat your own dog food. Whenever something is not as expected you can detect the problem and help solving it. However, in some projects architects are busy due to communication, design, and organization efforts so that spending time for implementation will be almost impossible. Never say impossible in this context, do not even think it! Even in such scenarios I recommend spending some of your time for code reviews and some of your spare time to get familiar with software technologies being used in your project context. Otherwise you will create kinds of Software Frankensteins: creatures you are not proud of and who tend to live forever unless not eliminated in an early stage.

Architects are not just masters of Microsoft Office and of a bunch of drawing tools they leverage to create design pearls, throw them over the fence to the developers who ought to build a perfect implementation, while the architects enjoy their holidays on a nice Carribean island.

Be pragmatic! I guess, the Pragmatic Programmers book series was created for exactly the same reason; to make us aware of the relevance of practical experience.

Never ever forget: ARCHITECTS ALWAYS IMPLEMENT. There is no exception to this rule.

Pseudo Effectiveness

2008-07-26T17:40:00.001+02:00

Do you know these co-workers who are always answering e-mail in the middle of the night, who are in their offices until midnight, who have so many contacts to customers, who have an infinite list of activities, who have their mobile phone directly integrated into their body, and who always tell you about the big work overload they have? All those unknown heroes who always sacrifice their spare time on the altar of the god of infinite work?

Of course, many of us in the IT world are more busy than it proves to be healthy or effective. But occasionally people only claim to be active and involved, while in fact they are not. Unfortunately, it is not easy to differentiate between actual effectiveness and that kind of pseudo effectiveness. Sometimes, the one who does the job gets not promoted while the one presenting beautiful slides about the work results of others is. Of course, you may also refer to the Peter principle in this context :-)

Why do I discuss pseudo effectiveness in this architecture blog? As an architect, you might experience high impact if there are people with high pseudo effectiveness involved in your projects. Think about outsourcing sites where you have the feeling that productivity is very low. Think about developers or architects who have the same skill level, but whose work results are completely different in terms of quantity and quality. Also think about job promotions where you got the gut feeling of being ignored while in fact you did an excellent job in contrast to this other guy.

But be aware of the fact that all of us are somehow guilty of pseudo effectiveness. A little chat with co-workers in the cafeteria, some daydreaming in front of the workstation, some Internet surfing. All this may count up to a significant amount of time during the day. But as you might remember from my previous posts breaks are important. I claim that productivity significantly drops when ignoring such kind of work-life balance.

How can you detect real pseudo effectiveness? The only way is by analyzing actual productivity. For which tasks has the person been responsible for and what are the actual outcomes? In this context, the goals should be defined in such a way so that there is no space for pseudo effectiveness. Someone who needs a week to write a simple letter may seem very active, while in fact being very ineffective and inefficient.

Design Pearls

2008-07-25T12:30:00.002+02:00

If you ever have been to Florence, you surely have admired all those incredible ancient buildings fully covered with ornaments. In the past, constructing beautiful buildings has been en vogue. Today, we simply could not afford the costs required for such endeavors.

What holds true for building construction, is also applicable to software development. But software engineers don't consider themselves just as craftsmen, but also as artists. We strive for architectural beauty and aestethic design, not being satisfied with functional aspects.

While architecture beauty is an important achievement, improving qualities such as expressiveness, simplicity or symmetry, design pearls define the other extreme.
By design pearls I am referring to masterpieces of software design which don't stop just meeting the requirements, but also add unnecessary burden (or ornaments if you will) to the system.

While design pearls show the mastership of one single individual they are counterproductive for team development as they decrease simplicity (there are more artifacts than necessary for the given task) and symmetry (they don't follow the KiSS principle but try to constantly reinvent the wheel).

We can avoid design pearls by strictly following a requirements-driven approach. All architecture concepts in a system must be rooted in a concrete requirement. In addition, we should use patterns where useful. A pattern is a "complete disregard of originality". It just helps reusing common wisdom, instead of coming up with new solutions. And we should strive for architecture quality indicators such as simplicity, expressiveness, orthogonality and symmetry. One way to aim for symmetry is to introduce design guidelines from day 1. Instead of letting developers figure out how to cope with cross cutting concerns such as exception handling, it is better to give them a framework of rules and guidelines.

In summary: design pearls are really nice to look at but ugly to change and extend.

An Anecdote on Architecture Training

2008-07-20T21:49:00.001+02:00

Recently, I gave an architecture training within Siemens. There was another Siemens seminar in the same hotel dealing with project teams. In one of the breaks, a guy asked me whether I knew more details about this architecture seminar. He could not believe, that someone within Siemens was teaching on building architecture. I explained to him that the seminar was actually covering software architecture, not building construction. He asked me whether I could recommend the seminar. "Yes, absolutely", I said to him, "to be honest, I am the speaker". What can we learn from this: be cautious if you tell someone you are an architect.

Stop those Control Freaks

2008-07-16T23:29:00.001+02:00

Suppose, you are in a meeting with other people. You need more information on the Hubble constant and ask the audience for details. Someone (maybe a friendly physics professor) comes to the white board and explains the theory.

Hmmh, how does that relate to software architecture? If you think about the situation, you'll recognize that the solution is based on a decentralized approach.

Normally, we are more used to centralization. Maybe, we are even addicted to all those centralized concepts. There is a central directory server, a central hub in the EAI solution, a central database, ... But as the example illustrates, sometimes decentralization is the better choice.

In the real life example we have used a P2P concept by leveraging broadband communication - ok, you just spoke very loud so that everyone paid attention to your question. Is this the only way to implement decentralization? No, there are several other alternatives.

For example, Swarm Intelligence: Suppose, I can't remember where I have parked my car. To increase search speed, all family members will visit different areas to locate the car. All searchers carry mobile phones to communicate with each other. After my wife found the car, she tells everyone where it is.

Another example could be Ad Hoc Networking: In the nearby forest, I find an interesting bird. I tell other people approaching my location about the bird who themselves tell other people who ... After a short period of time the scene is crowded with interested people.

Decentralized thinking means getting rid of central nodes or central controls. This might increase fault tolerance, but decrease efficiency. Often centralized concepts are mixed with decentralized ones. For example, there might be registries where all possible peers in a P2P network are known.

From an architecture viewpoint, several patterns support decentralization. For instance, the Lookup/Locator pattern, Blackboard architecture pattern, or the Leader/Followers pattern.

One of the qualities we can get from a decentralized system is emergence. The system as a whole reveals smart behavior that is more than just the combination of all nodes. Think of swarm intelligence like in an ant population.

From my viewpoint, decentralization is still neglected in software engineering, especially in software architecture. And as the examples shows, real life offers an abundance of decentralized concepts.

Architecture Design - How deep should it go?

2008-07-14T19:11:00.001+02:00

In my last posting I introduced the Onion model for architecture design. Often it is very difficult to determine when architecture design stops and fine-grained design begins. Thus, I am going to illustrate some of my thoughts.

Before you start the most important precondition for architecture design is the availability of

the business goal of the software architecture
a subset of all requirements (15-30%), but make sure the most important ones are already available
a coarse-grained test strategy.

There are different recommendations for architecture design. First of all, I consider it essential to differentiate between strategic design and tactical design:

Strategic design is about all those requirements that have a systemic impact on the whole software architecture. Basically, these are the inner layers of the onion. All architecture decisions that do not only reveal a local impact on a component together contribute to architecture design.
Tactical design is simply about the rest: all decisions that show only local affects. Interestingly, variability issues often denote tactical design issues, even in product lines. Things like exchangeability of an algorithm only imply, we need to open our strategic architecture to support variation of the particular family of algorithms (a.k.a. strategies).

This definition comes close to what Grady Booch once meant by "software architecture is about the important things".

Another consideration consists of reducing the number of abstractions. This is what I call the Pyramid model:

the top of the pyramid is the system itself,
the middle layer represents all subsystems,
the bottom layer comprises all components the subsystems consist of

For architecture design we should only follow the pyramid model. All abstractions underneath the bottom layer of the pyramid are part of fine-grained design. Note, that it might be subject to your own definition how the system, subsystems, or components of the pyramid model map to the concrete abstractions in your domain. This might look different between an embedded system and an Enterprise SOA system where subsystems might become services.

All architecture decisions should be completely rooted in requirements and risks. All extra add-ons increase complexity and thus decrease simplicity and expressiveness.

Don't be caught by the trap that architecture is only about problem domain and not about solution domain although you always should strive for keeping your architecture as independent from specific technical issues as possible (always try to defer all technology decisions to a later phase). "Hard" requirements such as "cycle time must not exceed 1 msec" or "use of SQL Server 2003 is mandatory" will be treated as first class requirements. And those requirements may also have systemic impact, thus contributing to strategic design.

In the documentation of your architecture follow a newsletter style. This implies, each artifact (system, subsystem, component) should be explained using a newspaper-like article. Architecture design is just that and no more.

In addition to architecture design you should:

specify design and architecture guidelines for all crosscutting concerns. This is best done before architecture design for all important systemic requirements. For other concerns you might create guidelines in parallel to architecture design.
create technology prototypes for all important risks. For example, if you need to guarantee 2000 transactions per second, make sure your middleware of choice really supports this requirement.
refine the test strategy you should have established after requirements engineering.

These rules of thumb are intentionally somewhat vague. Again, I'd like to refer to Grady Booch in this context: "there is no cookbook for software architecture".

Fortunately, there are some good practices to share.

The Onion Model

2008-07-03T13:33:00.003+02:00

Whenever people ask me how to systematically create an architecture, I present them my Onion Model.

If you like to sow an onion, you need some preparation. It is important to take care of the right environment and also to think about the requirements an onion needs to grow and what risks you might encounter. From a business perspective, you should also consider what type of onion you like to harvest, but that's a different story.

The inner core of the onion will be the domain model including all core components, their relationships and interactions.

For the next layers you will need a prioritized list of strategic requirements. These requirements influence systemic architecture decisions. For example, reliability, availability, or security. All of them represent operational requirements that result in specific infrastructures. If performance is the most important operational requirement, you will build a performance layer around the domain logic core. With other words, the domain logic will be embedded into the performance infrastructure. Then you turn to the second most important operational requirement. If security is the number two requirement, your existing onion that consist of the functional core embedded to an performance infrastructure will get wrapped by a security layer. This continues until your architecture is stable.

At the end you need to address tactical issues such as flexibility or maintainability. These will become the outer layers of the onion. For some of them, you will need open up your strategic architecture, for instance, by providing hooks.

Eventually, you'll get an "onion" which was designed with a functional core in mind. All important requirements had an higher impact on the onion than the less important ones which was ensured by priorization of onion layers.

Focus matters

2008-06-23T07:34:00.003+02:00

A lot of software systems do not meet some of their requirements and business objectives. Other systems have turned to design pearls only a few architects can understand. In either case you are in trouble. If a system does not meet its requirements or business goals customers will be dissatisfied. In addition, the system cannot be evolved to future versions before the problems are addressed, often causing large and expensive refactoring activities. If a system is a design pearl, it contains a lot of unnecessary architecture artifacts, making the system less expressible and simple. After short or medium time, these systems tend to become big balls of mud.
To prevent such problems there are different means. One of the most important one being requirements traceability. Each architecture change must address a requirement and/or a business goal. You should also clearly trace back the architecture decision to the requirement or business goal in your documentation. Architecture review methods such as ATAM use this information to figure out tradeoff points, sensitivity points, risks and chances of your architecture.
In the optimal case there will be patterns available to support your architecture design. For orthogonality you should apply the same patterns for the same problem contexts. Likewise, relevant cross-cutting concerns such as error management should be explicitly addressed by architects and enforced through guidelines.
Note, however, that architects may come up with the best architecture or the most consistent and complete guidelines, but will fail if the implemented architecture differs from the envisioned architecture. Thus, architects should walk around and communicate to enforce the architecture vision.
Just staying in office, throwing the architecture over the fence, and hoping everything will be in place after a while, is the best receipt for project failure.
As change is the rule and not the exception, architects must always ensure that any change in the set of business goals or requirements will lead to an appropriate architecture refactoring. With other words: if the focus may change and you need to stay in focus, then you'll have to change if the focus actually changes.

There is no business like office business

2008-05-30T23:22:00.001+02:00

In the last weeks I enjoyed small talk with several software engineers from various companies. Topics included technology and architecture, but also office work, management, colleagues and career paths. And several times I heard from companies where a couple of characteristic metrics for measuring engineer productivity exist. Let me exaggerate a little bit:

One of the wide spread metrics is office availability. I am speaking about engineers sitting in their offices or cubicles from early in the morning until late in the evening. This is something I fully understand for projects being in critical phases or delayed. Of course, I also did it and will do it in the future. But I also heard about engineers staying in their offices very late to send an implicit message to their management. Something, like "Look, I am working so hard for the company. If you don't promote me, you should suffer from remorse." However, the truth according to several studies is that you cannot be productive for 8 hours or more per day, at least not over a longer time span. Even worse, if you stay in office, your bug rate will increase significantly and the results will be not better than staying in office for less time. I know the stories about employees in Silicon Valley or Redmond sleeping near to their office PC. There is a reason why eXtreme Programming does recommend reducing work time to under 40 hours per week. And all of you believing in 16 hour work days, no matter wether you are good (engineer) or evil (manager): productivity in such overload situations is not possible. You either have many simple tasks or spend significant time twiddling your thumbs. Today, I read an article that an increasing number of IT workers suffers from the burn-out syndrome or other diseases which are caused by work overload.
Another wide spread metrics is internal visibility. Do you remember Bertrand Meyer's "bubbles don't crash"? Management tends to rather promote engineers who spend significant time presenting and chatting to management. If you don't provide a steady stream of good news (even if you invented this stream of good news) to the right persons, they won't even notice your existence in the worst case. Often, not the real results count but management visibility which is measured by .. guess what .. Powerpoint presentations, Word documents and Excel sheets. Don't forget that e-mails and meetings represent additional, important means in this context. Considered from this perspective, virtual reality has already conquered the world.

There are a lot of lessons we can learn from these statements.

First of all, Work/Life balance may be a buzzword, but from a useful one if you ask me. For example, if I recognize that my productivity is dropping to almost zero, I am leaving office, go for running or biking, and then continue with my work, often until late in the evening. Such extended breaks provide me with a significant productivity boost. It might appear unbelievable, but I am definitely more productive that way than if I had spent the breakout time for office work.
If I need to concentrate I will also work at home from time to time rather than spending the time in the office. With my phone constantly ringing, colleagues dropping by, and other continuous interrupts, offices don't really provide an adequate environment for creativity and brain workers. Ok, I am missing my colleagues and the social interactions after a short time. Thus, home office is the exception. Fortunately, there are so many opportunities (phone conferences, meetings) to interact :-)
Effectiveness and efficiency are/should be the only things that count. It does not matter when or where you created your results. However, you need to make sure, managers are aware of your work and what you achieve. Marketing your own person is an important issue in a universe where managers can't afford tracking all work results from all engineers. If you coach some colleagues as a mentor and if you promote their visibility and career path, inform your managers. Improving skills of co-workers and other more indirect results offer high business value to a company.

Interestingly, all these points are also valid for architects. However, home days are often a bad idea in those hot project phases when creating or enforcing software architecture. For an architect, communication represents the most important instrument. Communication works best in Face-to-Face meetings. But, nonetheless, also in these situations you need to plan your recreational breaks to enforce Work/Life Balance.

Developer Usability

2008-05-12T18:44:00.001+02:00

In the last months I've seen a lot of frameworks, platforms and libraries which revealed a substantial lack of usability. Developers reported to me that using those APIs was more a nightmare than a pleasure. Some of them just stopped using these artifacts, others built their own workarounds on top. When asking the API-developers for their comments, it turned out that most of those APIs have been built without any consideration of usability. In addition, adequate documentation was missing very often, which of course is also related to usability. Interestingly, I often hear the same complaints from end users. The UIs of most applications or devices are a pain. For instance, I am using a SAP-based system for ordering work items. What I need to fill in in all those fields is by no means self-explanatory. Help is missing or inappropriate. Error messages don't give any guidance what could be wrong. Suppose, shops like Amazon would come up with such UIs. How successful, do you think, would they be? Whenever we are building such systems we need two things:

a clear understanding of the domain
a tight interaction with users to ensure usability

For example, if we read the book by Krzysztov Kwalina and Brad Adams on how to design frameworks, you will find the recommendation on scenario-based design. Find out how end users could best use the system you are going to build in terms of user scenarios. Implement the API to support the most important and common scenarios. After the first prototype/iteration of the system is available, ask the target audience to use it and give feedback what they like or don't like. Then improve it!

In contrast to common belief, architecture is not only about functional requirements and operational qualities. It is also about usability. A system that meets all requirements but is not usable, is of no value for its users. As an example: I owned Rio and Creative MP3 players which provided an incredible amount of features, but lacked usability. After a while I switched to Apple IPods which have less features and are significantly more expensive but offer a great user experience.

At the end, you should strive for making the end user's life a pleasure. In this context, just remember that you yourself are using many more systems than you are developing. Something similar is written in many public restrooms within Germany: Leave the room in such a way you'd like to discover it.

On vacation

2008-03-14T13:14:00.003+01:00

If you already wondered, I am currently on vacation travelling abroads. That is the reason why I have only seldomly access to the Web. Now, I am sitting in front of an Internet PC in my hotel's lobby. I will be back on 28th March. Then, I will provide you with some new stuff. So far, I wish you some nice easter holidays.

DSLs revisited

2008-02-25T09:13:00.006+01:00

DSLs are everywhere. At least, we can read and hear a lot of good things about Domain Specific Languages. Unfortunately, many experts I meet are not always that sure what a DSL really is.

As a lazy person, I will not bore you with my own definition but just cite what Markus Voelter once said. In a former posting Markus defined a DSL as follows:

A DSL is a concise, precise and processable description of a viewpoint, concern or aspect of a system, given in a notation that suits the people who specify that particular viewpoint, concern or aspect.

Given that definition, UML is a vehicle to define DSLs. Take use case diagrams as an example. However, UML might better suit software engineers.

An ADL (Architecture Description Language) like Wright is a formal language to describe architectural designs.

BPEL or ARIS are DSLs to describe business processes. While BPEL is much more system centric, ARIS can also be understood by business analysts.

Thought questions:

Is a programming language such as Java also a DSL?

And what about natural language?

DSLs are useful to describe a viewpoint, concern or aspect of a system. But, what means do we have to introduce our own DSLs?

Meta data annotations (Java) or attributes (.NET) can serve as the core elements of a DSL. Take the different attributes WCF (Windows Communication Foundation) introduces to help describe contracts.

UML provides a meta model for introducing DSLs as well as specific DSLs (the different diagrams to specify viewpoints).

XML can serve as a foundation for introducing DSLs. Example: BPEL, WS-*, XAML.

Programming languages like Ruby or Smalltalk let you intercept the processing of program statements, thus enabling DSLs. Example: Ruby on Rails.

Integrated DSLs like LINQ are embedded into their surrounding language.

APIs define a kind of language.

The hard way might be designing your own language using plain-vanilla tools such as ANTLR. Note: It is not only the compiler or interpreter but also the run-time system we have to cope with.

According to the Pragmatic Programmers designing a DSL can be very effective.

It may help you visualize and communication the viewpoint, aspect, concern the DSL addresses.

It may enable you providing a generator that generates significant parts instead of forcing you to hand-craft. This is not restricted to code but you could also generate tests or documents.

Stefan Tilkov provided an excellent example. Suppose, you are developing a Java application. Writing the unit tests in JUnit would be the normal way of operation. But, maybe JRuby would be a much better way to write unit tests. As both run on top of the JVM, this can be achieved easily. In this context, we use JRuby+xUnit APIs as our DSL.

I believe, DSLs help climb over the one-size-fits-all barrier traditional programming languages introduce. Creating a DSL does not need to be as complex as defining a programming language. Using XML Schema or Meta data annotations or UML Profiles are nice examples.

Generative versus Generic Programming

2008-02-23T19:57:00.001+01:00

Andrey Nechypurenko, a smart colleague of mine, recently brought up an excellent thought. If we already know a domain very well, why should we bother building model-driven generators? Couldn't we just provide an application framework instead? Indeed, each framework introduces a kind of language. It implements commonalities but also hooks to take care of variability. So, when should we use model-driven techniques and when should we better rely on frameworks?

My first assumptions were:

A DSL is much closer to the problem domain which helps different stakeholders to participate establishing the concrete model. If stakeholder participation is essential, than a DSL might definitely be the better way.
DSLs are (often) more productive. It is much more verbose to program against a framework than designing using a DSL. Needless to say, that we need to take the efforts for providing the generator into account.

At least in my opinion the answer could also be to use both, a framework that captures the domain's core abstractions and a generator that generates applications on top of this framework.

Any thoughts about these issues?

Relativity

2008-02-18T23:55:00.001+01:00

Today, I had an inspiration. My spouse told me it will take 5 minutes to get ready for leaving home, but actually it took her over 20 minutes. As those "events" constantly happen, my assumption is that women are moving with much faster speed than men. This way, 5 minutes on their spaceship equals 20 minutes of our time.

I also remember a professor at my university asking a student about the "evaluator problem". The student did not really understand the question. Do you? Actually, the right answer turned out to be that he was using C style malloc for allocating storage in his compiler's attribute evaluator.

What can we learn from this? Humans might have a completely different perspective on reality depending on her/his background and context. Whenever we are dealing with architecture design, we'll have to communicate with all those stakeholders. In order to prevent misunderstandings like the ones illustrated above, we need to introduce a kind of formal language and a common scale - with other words: a common understanding.

One of the most critical points in design is the consideration and elicitation of requirements. This also holds for requirement priorities. For example, a project manager might be more interested in partitioning the architecture so that the different teams can develop in the most effective way, while an architect will follow a completely different rationale for modularisation.

In many projects I was also surprised by the fact how important a common project glossary is. Otherwise, people will interpret the same terms differently which is not a big boost for effective communication. For example, in one SOA-based project senior managers considered process monitoring to reveal the business perspective (Business Process Monitoring). They were interested to observe the number of successful business transactions or cash flow. What they actually got from development was system monitoring functionality such as showing load, bandwidth or faults. Guess, how long it took development to correct this?

It is very likely, that you also experienced the architecture/implementation gap in your career. This inevitably occurs whenever architects throw their design specification over the fence and then leave the development team alone. Did you ever read a specification and understood it without any problems? Me neither! Some parts might be missing, other ones might not be implementable, and others might be contradicting. Even formal standards such as CORBA, SOAP, HTTP that were developed by expert groups within long time periods, reveal those problems. How can you then expect your own specification to be perfect? What is not perfect, will then be corrected or interpreted in unanticipated ways.

Thus, it is essential that such clarifications happen as soon as possible in a project. It is your responsibility as an architect to tackle this problem. Remember how often I already mentioned that effective communication is the most important skill - one of the few moments where I do not abide to the DRY principle. There are lot of methods and tool that explicitly address this issue. Think of agile methods, domain driven design, use case analysis, to name just a few.

Reality is relative. Thus introduce a common frame of reference for the stakeholders. This holds for software engineering, but also for real life :-)

Workflows

2008-02-16T11:59:00.001+01:00

Workflow engines such as Microsoft's WF (Workflow Foundation) are only rarely used these days. Most developers totally underestimate their power. However, in an integrated world where processes respectively workflows increasingly need to connect disparate heterogeneous applications, just hard coding these workflows in traditional languages such as Java does not make any sense. First of all, coding in Java or C# requires programming skills, while often business analysts are the ones in charge of business processes. And secondly, hard coding in Java, C#, C++ implies that all changes must happen in source code with the code base representing a beast containing both business logic and workflow related content.

Before covering the meat of this posting, I need to define what I understand by a workflow. By the way, I will use the terms workflow and business process interchangeably as I don't consider the differences stated by other authors very convincing.

If we just refer to Wikipedia we will find the following definition:

A workflow is a reliably repeatable pattern of activity enabled by a systematic organization of resources, defined roles and mass, energy and information flows, into a work process that can be documented and learned. Workflows are always designed to achieve processing intents of some sort, such as physical transformation, service provision, or information processing.

Sounds rather confusing, right? Let me try my own definition.

A workflow defines a set of correlated activities each of which representing a (composite) component that takes input, transforms it, and sends the result via its output ports to subsequent activities. Transitions to an activity may be triggered implicitly by (human or machine initiated events) or by explicit transfer of control
Workflows can be considered as a Pipes & Filters architecture with the filters being the activities and the pipes being the communication channels between them
All activities within a workflow share the same context such as common resources
Workflows are (composite) activities themselves
A workflow language is a DSL that allows to declaratively express workflows. Examples include BPEL, ARIS, or XAML as used in WF
A workflow runtime manages the (local) execution and lifetime of workflow instances

In fact, a workflow closely resembles traditional program code with the statements being the activities.

A typical real-life example is visiting a restaurant.

The workflow starts when you enter the restaurant and arrive at the "wait to be seated" sign.
The waitress will now ask you whether you have a reservation.
If yes, she will look up the reservation and escort you to your table
If no, she will try to determine whether a table is available
If yes, she will escort you to the table
If no, she may ask you to come back later => stop
The waitress will provide you with a menu
At the same time, another waitress might offer you water
...

You will recognize easily that UML activity diagrams could be used to design workflows. Activities are either triggered by events or by active commands (e.g., you ask the waitress to bring you another beer which is like an event or she could ask you if you need anything else which is more like an "invocation"). Thus, you could either use such a diagram to express a workflow or maybe a more domain-specific kind of language/model such as BPEL. This language could be graphical, textual, audio-based, and so forth.

The advantage of a (domain-specific) language is obvious. It is much easier for domain experts to express the workflows themselves. Moreover, the workflow descriptions could be either used to generate the actual code or to execute the workflow in a runtime engine.

Sometimes workflow transitions depend on conditions such as "if customer is a special guest, give him a window table." These conditions however might also be subject to change. Coding such conditions in activities thus reveals the same problems as coding workflows in C# or Java. As a consequence, we should also specify such rules in a declarative fashion. This is where Business Rules Engines enter the stage. Business rules might be explicitly expressible in a workflow language or just be hidden in special activity types.

How can we retrieve the workflows in the act of architecture creation. User stories, Use cases and scenarios, help determing the workflow in a system. Which of these should be hard coded and which ones should be better implemented using a Workflow language, depends on requirements, frequency of change, required variability, and other factors, questions a software architect needs to address.

Removing unnecessary abstractions

2008-02-15T14:30:00.001+01:00

A good architecture design should always only contain a minimal set of abstractions and dependencies. According to Blaise Pascal a system is simple, when it is not possible to remove something without violating the requirements. Again, we can consider this from an architecture entropy viewpoint. Minimalism and simplicity imply the lowest possible entropy.

Let me give you an example how a design should not look like. Suppose, you are developing a graphical editor. In the first design sketch you introduce an abstraction called Shape with functionality such as Draw(). From this base type you are then inheriting subtypes such as PolygonicShape and RoundedShape. Finally, the abstractions Rectangle and Square and Triangle will become direct descendants of PolygonicShape, and so forth.

After a while you might recognize that it does not make any sense to introduce the intermediary layer consisting of PolygonicShape and RoundedShape. You find out that in your context all concrete classes such as Ellipse, Rectangle, Triangle, ... should be directly derived from Shape, because you don't differentiate between types of shapes in your software.

By removing the intermediary layer you have eliminated unnecessary abstractions and dependencies. Obviously, this also adheres to the KiSS Principle. But why, you may ask, are we not removing Shape too? The answer is very simple: Shape introduces common behavior required to handle all concrete shape subtypes uniformly in the editor. Otherwise, we would need to introduce large switch statements, type codes or reflection mechanisms in several places of our application. Thus, removing Shape would impose a much higher entropy.

As you can surely see, determining the applicability of this architecture refactoring is not trivial, because you need to consider different layers and levels of abstractions in your design. For example, in another context it might be important to introduce intermediary abstractions such as PolygonicShape because you need to treat polygons different from rounded shapes. In this sense truth is always relative.

Work Life Balance - Revisited

2008-02-14T22:22:00.001+01:00

Right after each of my postings related to Work Life Balance, my inbox grew due to people asking me about how I personally keep the balance. That's the reason why I want to spend some sentences on this personal issue although it is not related to Software Architecture (al least at the first sight).

I still consider myself a workaholic on rails. Only a few years ago I used to work the whole week without too many exceptions. After some health problems, I remembered the time when I had been totally enthusiastic about sports. And then I started to go biking and running regularly. That's where the rails come in. Ok, I did too much running and got problems with my knee which is the reason why I am more biased to biking these days. Typically, I will run or bike almost every day from spring to autumn. For example, I enjoy to bike to work (10 km). On my way back to home I will often use a detour leading through a large forest and along a bike lane near the Isar river here in Munich (30-50 km). Sometimes, I leave office late afternoon and continue my work at home. If possible, I am also running which is an incredibly excellent way to relax, especially in winter when biking is not that cool (or should I better say when it is too cool). After 10-12 km which may take more than 60 minutes depending on my training intensity all problems are gone and my mind is open for new ideas.

My other hobbies include composing music, reading books, listening to podcasts, audiobooks or music on my IPod, among many other interests. Yes, I definitely got more interests than time.

My experience has taught me that sports (no matter what) is the most important way to keep your balance. So is meeting people who have no interest in any IT topics. Same for all hobbies that help forgetting about any work related challenges (formerly known as problems). I have never been as creative as when keeping my work/life in balance. The more time I spend for sports and other interests, the more productive I will be. This is kind of surprising as I always believed in the opposite statement (the more time spent, the more productivity).

Just relax!

All about Cycles

2008-02-14T18:11:00.001+01:00

Here comes a real world example. In a management application the distributed agents in charge of managed objects (such as routers, load balancers, switches, firewalls) are accessed by a centralized monitoring application. For this purpose, the sensors offer functionality such as setter/getter functions. The monitoring subsystem inherently depends on the agents it monitors. After a while, the developers of the agent subsystem think about returning time stamps to the monitoring subsystem whenever it requests some information. But where could they obtain these central time stamps? The team decides to add time stamp functionality to the central monitoring subsystem. Unfortunately, the agents now depend on the monitoring subsystem. Developers have established a dependency cycle.

"So, who cares?", you might say. The problem with such circular references is that they add accidental complexity. Mind the implications such as:

If there is a cycle within your architecture involving two or more subsystems, you won't be able to test one of the subsystems in isolation without needing to test the other ones.
Re-using one of the subsystems in the cycle implies you also need to re-use the other subsystems in the cycle.

It is pretty obvious that dependency cycles are a bad thing. They represent an architectural smell. But how can we cure the problem? Can you hear the horns playing a fanfare? The answer is: by applying an architectural refactoring!

So, how could we get rid of a dependency cycle?

Option 1: by redirecting the dependency to somewhere else. In the example above, we could introduce a time subsystem the agents use to obtain time stamps instead of adding this functionality to the monitoring subsystem. This way, we broke the cycle.
Option 2: by reversing one of the dependencies. For example if we exchange a publisher/subscriber pull model (event observer itself asking for events) with a push model (event provider always notifies event consumer). Another example is introducing interfaces: instead of letting a component depend on another component we could introduce interfaces. This way, a component always uses interfaces and never component implementations directly.
Option 3: by removing the dependency completely if it is not necessary. For example consider Master/Detail relationships in relational databases. Only one dependency direction makes sense here. The other one is obsolete.

Dependency Injection as provided by all those IoC Containers is very helpful to avoid dependency cycles. Even if you got no dependency cycles in your system, you should always avoid unnecessary dependencies - because they inevitably increase entropy and thus the chance of circular references.

But how can we detect such dependency cycles within our system? This is where CQM (Code Quality Management) tools enter the stage. Of course, these tools are not restricted to problems with dependencies. They are also helpful with a lot of other architecture quality related measurements.

Can we always avoid cycles? No, because sometimes they are inherent in the domain. For example, in Microsoft COM due to the usage of reference counting cycles were required in graph structures consisting of COM objects. However, one could certainly ask if the design of COM was appropriate in this respect. But that's a completely different issue.

Thus, the general guideline should always be: avoid unnecessary (accidental) dependencies, break dependency cycles!

Emergence

2008-02-12T17:23:00.001+01:00

Suppose, you are going to develop a traffic control system for a rather large city. The goal is to optimize traffic throughput which could be measured by average miles of each cyclist and motorist. In a first approach, you may suggest to use a purely centralized control subsystem.

There are various problems with this task:

In a rather large traffic infrastructure, traffic can never be controlled from a centralized component. First of all, this component would represent a single point of failure. Secondly, it would not scale or perform at all due to the sheer amount of traffic signals involved.
The requirement of traffic optimization reveals a significant problem. Assume, that there is a main road through the city. The best way to achieve the goal could be to just set all traffic lights on the main road to green. Participants driving on side streets would then need to wait forever. That's what is called starvation in multithreading.

How could we solve those problems?

Instead of using a centralized approach, we could rely on a completely decentralized approach. For example, each street crossing could permanently measure the traffic and adapt its traffic lights automatically to the current situation. If all street crossing act independently, local tactics might dominate global strategy. Obviously, this is not what we want. Strategy should always dominate tactics. Hence, let local decisions of street crossings depend on "neighboring" street crossings. If appropriate, we could also mix centralized control with decentralized autonomous adaptations. That basically means, that we control the overall strategy, but allow the system to automatically and locally adapt its tactics. Is it always useful to apply decentralized approaches? No. Here is a real life experiment: Take a number of people in a row each of them stretching one of their fingers. Put a large stick on their fingers. And then tell the group to systematically and gently put down the stick to the ground. They won't succeed until finally one person will become the leader and coordinate the whole action.
We also learn from the example that we need really to address requirements with much more care, especially in systems with complex interactions. In the example above the target function should not only consider average flow but also take possible starvation into account. By the way, the traffic example could also be replaced by a Web shop and Web traffic. In this example, starvation would imply that some shoppers would have a fantastic experience while others could not receive any Web page in an appropriate response time. Guess, how often the latter ones would re-visit the shop. Wrong understanding of requirements can seriously endanger your whole business goals.

The question is how to design such decentralized systems. There are lots of options:

Infrastructure: P2P networks are one way to organize resources so that they can be located and used in a decentralized way. Cloud Computing formerly known as Grid Computing is another cool example.
Algorithms: Genetic algorithms help systems to automatically adapt themselves to their context.
Cooperation models: Swarm Intelligence such as ants also illustrates how emergent behavior can substitute centralized control.
Architecture: Patterns such as Leaders/Followers or Blackboard help introduce decentralized approaches.
Technologies: Rules engines and declarative languages such as Prolog are useful for providing decentralized computing.

The problem with decentralized versus centralized systems is often when to use what. And in addition, to decide how much decentralized the system should be. This is not always obvious. Let me give you some examples instead:

In a sensor network you might place many small robots to a specific area in order to monitor environmental data. These robots would be completely autonomous. Even if some of them fail, the overall system goal can be achieved.
In a Web Crawler several threads could search for URLs in parallel, but they would need to synchronize themselves (e.g., using a tuple space) in order to prevent endless loops. This resembles ant populations.
In a car or plane you need central control, but some parts can also operate and adapt independently such as Airbags or ESP. This is more a hybrid approach with the global strategy implemented by a centralized approach.
In embedded (real-time) controllers (such as your mobile phone) everything typically is centralized because behavior must be statically predicted and configured.

The more co-operation your components require and/or the more determinism you need, the less a decentralized approach seems appropriate, and vice versa.

The best way to design such decentralized systems is to introduce the concept of emergent behavior. For example, define a set of simple local rules to which the system parts must abide. And then let the parts take control. By the way, this is exactly how the Internet works. But in the Internet, you will also find a hybrid approach combining centralized backbones with decentralized nodes.

Nervous Breakdown by Communication Overkill

2008-02-09T22:04:00.001+01:00

Whenever I meet IT experts in airline lounges or conference rooms they are often very busy in synchronizing their memory with the continuous information flow from their company, customers, and other sources. Typically, they are armed with a quite impressive weaponry that includes blackberries, (>= 1) mobile phones, notebooks, and PDAs. Asking them about the effectiveness and efficiency of their work, they mostly complain about the impossibility to perform real work in their office in addition to the burden of all these travels and meetings. If we did a cost/benefit analysis relating the quantity of information exchange to work effectiveness, we would certainly be (negatively) surprised.

Let me just provide a theoretical example.

In average, meetings will require 2 hours per day. Given the fact, how ineffective meetings are planned and performed, I assume that at least 2 thirds of the time spent is a complete waste.
If you receive 100 mails per day, then throwing away 50 of them, might only take 5 minutes. However, reading 30 mails will require additional 30 minutes. And answering the remaining 20 will require 60 minutes. Thus, you've spent at least 1 1/2 hours for handling email.
Travel might also require 1 hour in average per day.
From time to time, you will have some personal face to face communication with colleagues, some of them just flocking to your office.
Telephone calls will require at least another hour per day.

Each media break (such as the arrival of e-mail, SMS, phone calls, meetings, etc) also interrupts your work and causes continuous context switches. As you know, context switches are ineffective time durations, not only in a CPU but also in real life.

If you summarize all of this, it is not very difficult to recognize the problem.

How does this relate to software architects? As you definitely know, a software architect is in charge of designing a software architecture which requires a lot of communication activities. The more ineffective communication proves to be, the less time can be spent for the actual job.

We are all part of a vicious cycle that involves ineffective communication, unnecessary breaks and context switches. The only way to solve the problem is not to stop communicating but to spend your time much more effectively.

Personally, I consider the following means very helpful:

If you need to work on a document or review, use your home office, given that you can work there without any major interrupts.
When you feel tired, exhausted or relax, go to the gym, or do whatever you feel appropriate to improve your mood. It is simple math. If you need to spend n hours for your work, it is not necessary to spend these hours in a row. For your company, mixing work time and recreation time is much more productive and less error-prone, especially when your job requires creativity.
Time box all meetings. I found out that m one-hour meetings are much more effective than one m-hour meeting. Needless to mention, meetings should be effective themselves (agenda known in advance, specific and clear objectives of the meeting available, have a moderator, and so forth).
Prioritize mails and phone calls. Tell people and train them, that you are not able to answer all mails or phone calls all the time. Consider voice-mail and mailboxes as friends. They allow you answering whenever you are prepared, not when the callers are. Switch off mobile phones, blackberries when you are in meetings, need to relax. Not to communicate sometimes is not a luxury but a necessity. By the way, it is not rude to tell colleagues that you are not ready to talk to them but that you will come back tothem later, if you are currently busy.
In your spare-time or holidays don't accept any business-related communication except for emergencies.
Travel is important as face-to-face meetings are the most effective form of communication, but don't underestimate the impact of traveling on your effectiveness. Thus, only travel when really necessary.
Plan your time thoroughly (time management). And also plan sufficient spare time for recreation to keep your work/life balance and may be even your relationships healthy.
Diversify your work day. Three meetings or telephone conferences in a row are boring, exhausting and ineffective. The same holds for sitting in front of your PC or notebook (or TV) for several hours.

There is the common misconception that knowledge workers such as software architects should always be available for communication and work 12 hours a day without any break. Eventually, the job of architects is to build high quality software architectures, and to do this successfully they need motivation, fun, and focus. In the end, multitasking such as communication in one thread while designing in the other is not feasible for humans. Sufficient recreation time (for example due to avoidance of communication overkill) is an essential means to do our jobs successfully.

Sometimes relaxing is more productive than working.

Architecture Governance

2008-02-09T18:27:00.001+01:00

I consider cross-cutting concerns such as error handling as one of the major challenges in any software development project. These cross-cutting issues should be subject to what I call architecture governance.

We need to establish rules how to uniformly handle these issues throughout the project. Otherwise, every developer or architect will provide her/his own solution which significantly reduces symmetry and orthogonality. I've seen many projects where error reporting was provided by some subsystems using return values while others used exceptions instead. Even in projects where exception handling had been common practice, developers introduced their own exception types for similar exceptions. Guess, how well such an architecture is balanced.
We need to define roles and persons in charge of supervising software development and enforcing the rules. In one project architects had created documents for all of these concerns. Unfortunately, developers didn't care about the documentation. Policy enforcement tools such as FxCop for .NET can provide help. Trust is good but control is better.

The best way to guarantee proper handling of these concerns is to provide guidelines from day 0. Architects are in charge to establish a strategic architecture document as well as the design guidelines and programming conventions in the beginning, not as an afterthought. All places where those rules are then violated within the project are subject to refactoring.

What are typical cross-cutting concerns in this context? Examples include:

Error Management
Security Checks (such as most cross-cutting non-functional qualities)
Naming Conventions
Mandatory Patterns

Why is that important? Because allowing everyone doing whatever she/he considers best, will lead to less expressiveness and readability, high accidental complexity, less developer usability, missing orthogonality and symmetry. With other words, it will have an negative impact on all architecture qualities.

This is why Architecture Governance is essential!

Learning from failure: Bangalore and Software Architecture

2008-01-31T19:06:00.000+01:00

I am currently enjoying my time here in Bangalore (India) where I am giving some talks and trainings on Software Architecture. Especially, the hospitality here is marvellous. Bangalore is a very fast growing city with very nice people and a hot spot due to its IT companies and IT experts. On one hand, all the high tech industry is quite impressing. On the other hand, it is astonishing to see hightech centers in a lowtech infrastructure. Interesting, you might think, but how is this related to software architecture? When speaking to colleages here and to my driver (you cannot and should not manage to drive yourself as someone more used to the US or German car driving patterns :-), they told me a lot of interesting things related to Bangalore's infrastructure which I consider likewise applicable to software architecture. Don't be embarassed that I explicitly use Bangalore. It is a pars-pro-toto. Any other big metropoles would do also, but now that I am here ....

Bangalore's population significantly increased over the last years. Unfortunately, the infrastructures could not be increased accordingly. What formerly has been garden city, a small green place, has now become a big metropole. A good example is the traffic infrastructure. It has turned to a significant bottleneck (also known as nightmare) so that it might take you an hour even for small distances. What can we learn as software architects: We must plan for non-functional qualities such as scalability and prepare the system to increase when the load increases. Doing this as an afterthought can prove to be almost impossible. Of course, this is hard for a city.
As an implication of the insufficient traffic infrastructure, drivers of cars, motorbikes, bicycles, busses, trucks, rickshaws will just use any available gap of a traffic lane. Thus 3 lane streets will become 6 lane streets. There is almost no traffic control except for a few police men (and I am not sure how much control thy really have). The most important tool is the horn to tell other persons that you are here. What can we learn as software architects: governance is an important issue. Make sure that everyone abides to the rules. Overload situations in a system also make it nondeterministic as you can never anticipate how much time particular activities will require.
If you drive around in Bangalore, you will find no direction signs at all. Want to go somewhere? Better know the route or follow a trial-and-error approach until the end. What we can learn as software architects: usability and habitability should be considered important. Otherwise, users may have a very unpleasent time with your product.
The current airport is a former military airport and thus can provide only limited and sometimes slow service. Now, the local government is opening a new international airport to solve the problem. Unfortunately, there will be no fast train or highway connections from the airport to the city center. People from Electronics City (the Silicon Valley of India) suppose, it will take them 3 hours to reach the airport. What can we learn as software architects: If a service is very performant or offers otherwise high quality of service, this won't help you much once it takes a long time to send your request to the
service or to receive a response. You always need to look on the system as a whole. And you always should remember the weakest link in the chain defines what quality you will obtain.
To improve the infrastructure several construction works are on the way. For example, there will be a new multi lane highway. However, this highway will end in a two lane highway. Guess what what happens on the junction of these highways? What
can we learn as software architects: Also mind the connections. Local optimizations might help locally but can not solve problems that should be addressed globally. First think strategically and then tactically.
In the previous days there were some news about people who suffered severe illnesses from drinking the local drinking water. It turned out that some people didn't follow the regulations and just installed sewage pipelines near water pipelines.
Imagine what will happen if erosion leads to lecks where both systems join? What can we learn as software architects? First of all, it is not sufficient to provide guidelines and throw them over the fence. An architect must also control that developers abide to the rules. Second, design erosion can lead to unwanted side effects. Instead of waiting for problems to occur, it is better to apply refactorings or even a complete reengineering. The latter is almost impossible for a real city infrastructure. Third, you need to keep in mind how infrastructures interact with each other. Fourth, you need to plan upfront for modifiability and later evolution to prevent such workarounds.

Take this example and think about ULS (Ultra Large-Scale Systems). The ULS proponents compare ULS with building cities instead of building houses where you also will also have to deal with human interaction and creating/maintaining/evolving large infrastructures. As soon as we are starting to deal with such ULS (systems of systems) we need to make sure such design erosion will not happen. There is a lot to learn from real life for cyber techies.

My Trip to India

2008-01-25T19:33:00.001+01:00

Tomorrow I will travel to Bangalore, India, for one week. There I will give some talks on Software Architecture in the context of a software architecture education program for our colleagues. I am really looking forward to this trip.

expect further postings after my return

(Re-)Use a Component

2008-01-18T23:17:00.001+01:00

It sounds so incredibly simple. Suppose, you are currently developing a car navigation system. During lunch break you meet some friends who are working in another part of your company which has already developed a component for calculating routes. "That's cool", you think. "Why not just take the same component which will significantly reduce costs". Now, you start dreaming about extra holidays in Honolulu paid by your boss for saving so much money. You can't stop immediately walking to Wally who is one of your cynical colleagues in order to check the value of your recent inspiration.

"Hmmh", he murmurs, "sounds like a really great idea". But then he continues, "... but there may be some - well - small problems on your way to paradise". He starts his endless list of potential hurdles:

The existing routing component which I will coin R has been developed for Windows Mobile, but your application needs to support Linux and Android as well. You'll require some wrapper facades to make R portable. This could take a significant amount of time and developers.
Also the interfaces of R are not exactly in the shape you expect. Your developers may integrate some of them using plain-vanilla wrappers while other interfaces will require significant refactorings. For example. incompatible data types, different protocols and some internal dependencies make your life much harder. And mind the multiple components dilemma! Components may influence each other. I remember commercial components that caused serious problems whenever we used them in combination. And, of course, R also expects some interfaces from its environment which are currently not available in your application.
Another issue is the dependency problem. Component R requires different other libraries and components. This means, developers must either also migrate these other dependent software artifacts or try to transform the component in such a way that it can leverage similar artifacts of the target environment.
In the application for which the partner development team has created R were some special requirements related to fault management, availability, security and reliability which are a little bit different in your application. Either you need to adapt your application to R (very unrealistic) or R must be adapted to your application.
As R is part of a multiple products some variability has been identified and appropriate variability mechanisms established within R. Unfortunately, these mechanisms do not match your variability requirements very well.
Another problem you may encounter is the 99% problem. R might provide almost all functionality you require. Only 1% is missing, but adding this 1% will require a significant amount of time and resources.
Tests for R have been developed for the exact context in which R currently resides. Thus, you'll need to establish modified test plans and a test strategy for quality assurance.
Unfortunately, you also have to consider an additional dimension. R currently is owned by the partner development which also is responsible for its evolution. With other words, bugs, technical innovations or requirement changes will cause the component to be constantly modified in the future. As R is owned by someone else, you cannot expect that the owning department will be able to consider your requirements in the first place.

Ok, sounds like bad news. And as you know, bad news are the only thing traveling faster than light. It is not a good idea to rip a component from its context and then hope it will run unchanged in another context. Instead, a lot of efforts are necessary to transform a component from context A to context B. And the problem will get even worse when the same component is supposed to execute in many contexts.

You learned your first lesson: ad-hoc re-use causes more harm than benefits. So let us take another direction. Why not take the component R and make it a re-usable component in your organization serving more than one application? With other words, let us take a systematic approach.

A systematic approach requires a completely different setting.

All shared components are developed for the application in different products from the beginning. You may not always start from scratch, but use existing components as a base.
In your organization you need to establish an independent team that owns the component (or various components) and acts as a supplier to all product groups. It is also the component group's responsibility to take care of quality assurance. Of course, it is necessary to establish some governance measures as well.
All product groups and the component group need to define a common development process which integrates component development & evolution with product development & evolution. Component release planning must follow release planning of products. However, different versions or branches of the same component may co-exist.
It is also essential that component development would suffer from a whole spectrum of diverse contexts in which components reside. Thus, products should abide to some constraints and share practices and guidelines. One example to make this point more concrete: If product P expects errors to be reported by exceptions, while product Q is using error values instead, how should a component be able to provide all of these diverse policies. This simply isn't feasible. In this case, the company policy would force the component and all products to use exception handling. Another example is documentation and design. Anyway, the organization should introduce (and enforce!) some common guidelines for product and component development.
For the case of bug fixes or other relevant changes to components a feedback loop between product teams and development teams needs to be established.
It must also be defined which requirements the component group must address. This includes functional and non-functional qualities. Especially, the variability requirements need thorough consideration. A component should not be a jack of all trades (trying to meet all product requirements) but also can't be a master of none (just providing functionality without considering product requirements and usability issues).
Such a systematic approach does not only require changes of the development organization but of the organization as a whole. For instance, product managers must now deal with different groups. Test managers must adapt their test strategies.

Obviously, such a systematic approach for re-use requires a cost benefit analysis. It is rather unlikely that an organization would establish such an approach for only two products that share the same component.

All your beautiful dreams have vanished. What looked so easy and promising, turned out to be rather complex and expensive. There is no free lunch.

But wait a second: what if your products often had a very similar scope containing almost the same functionality? What if were are in the business of building lots and lots of car navigation systems? What if these products could use all these powerful components for routing, MMI, GPS, speech recognition, ...? What if you had a framework combining these components to a powerful and adaptable whole? All of a sudden, the sun is rising again. In this case, a systematic re-use might be expensive to establish but once appropriate processes and organization structures are in place, it will be very effective. We have entered the universe of product line engineering.

Even then, you have to consider all the dangers lurking in derelict parts of the galaxy. Mind the dark energy that tears everything apart and mind the dark matter that leads to system collapse. Be aware of black hole components in your architecture! But that's a different story.

OOP Conference 2008

2008-01-12T17:54:00.001+01:00

Only as a side remark. I will participate in this year's OOP conference giving a talk and a tutorial. Note: the presentation language will be german.

In the talk I will address Software Architecture Refactoring, currently my favorite topic. As far as I have heard over 170 people already have already registered for the talk. I am quire impressed how many people are interested in the topic. On the other side, architecture refactoring is something new (which really strike me when I came up with this field because I assumed that almost every architect should have knowledge in this field)
In the full day tutorial (on friday, the 25th January) I will introduce SOA with .NET. Despite of the title, the tutorial could also be interesting for people not interested in .NET, as I also will include many parts that address SOA in a more general way. But, of course, technologies such as WCF, WF and BizTalk will get sufficient coverage.

If you are planning to attend any of these events, don't hesitate to approach me.

Architectural Entropy

2008-01-12T17:42:00.001+01:00

Several years ago I introduced the term entropy in the context of software architecture. Entropy defines the number of concepts appearing in a software architecture. If your design consists of two simple components associated with one unidirectional relationship, this design probably has a low entropy. I only say "probably" because it depends on whether the internals of the component are complex or not. If, however, you increase the number of components or relations or their internal complexity, entropy inevitably also increases. High entropy often is an indicator for high complexity. High complexity always means high entropy. But high entropy does not enforce high complexity. A system that consists of almost infinite simple components that are not related with each other implies high entropy but not high complexity. You should also consider the other direction. A system that consists of only one single god component may be very complex. Hence, entropy can non only be measured by addressing one view or abstraction level (e.g., the component layer) but must consider all views and abstraction layers.

Obviously, incremental architecture design must lead to higher entropy, because we constantly add things. One of the fundamental issues in achieving high architecture quality is minimizing entropy. Activities such as software architecture refactoring are doing exactly this. Quality factors such as loose coupling, symmetry, conceptual integrity, orthogonality strive for reducing entropy or at least keeping it small.

The art of software architecture is controlling entropy.

Context and Boundaries

2008-01-03T10:00:00.000+01:00

One of the most important issues when starting a new project is the differentiation between those parts that are in the software system and those which are external. We need to specify clear boundaries between the system going to be built and its environment. This is an issue developers often neglect. One of the obvious implications of not considering boundaries are misunderstandings during the design and architecture modelling phase. Are these boundaries simply a consequence of the underlying domain? Not at all! If we are going to create a Web Shop, the order processing subsystem is definitely an important concept of the domain. However, we could either introduce our plain-vanilla, homegrown order processing subsystem or just integrate an existing system such as SAP R/3 to provide the same service. In other words, first we need to consider which services are required, then we need to decide whether we will build these services as part of our software system or just (re-)use other external services for the same purpose.
How to proceed: I am a big fan of component-based design. I just consider components to be independent artifacts (i.e., modular units of testing, deployment, implementation, modularization,...) providing services to their environment (services may be functionality or events or configuration interfaces) and consuming services from their environment (such as functionality or events). Note: components can aggregate other components. I will design all subsystems including the whole system itself using this component concept. Hence, I need to define which interfaces my system will provide to its environment and which it will need to obtain. For example, one typical required interface of a Web Shop is access to a credit card company to verify customer credibility.
When designing such context diagrams the obvious choice is to leverage UML component diagrams.

Requirements Part II - The Art of Multi-Facet Design

2007-12-19T16:27:00.000+01:00

I'd like to continue my previous posting with an architectural design consideration. Basically, an architect maps the concrete problem (which is an entity of the problem domain) to a concrete solution (which is an entity of the solution domain).
Requirements help to decide which paths to take at specific points. This implies that requirements constrain the solution space. If multiple requirements are "applicable" at a specific point, priorities define the precedence which is why we must assign unique priorities to them. Not very complex so far, right?

But how do we really implement a solution architecture?
The first thing we obtain is a functional object model consisting of all the entities in the domain as well as their relationships. For instance, if we are going to build a Web shop, this model would probably consist of concepts such as customer, shopping cart, order system, product catalog, payment system. It is highly recommendable to define a domain model (domain-driven design) first to achieve a sound understanding and a common vocabulary among all stakeholders. Use cases are the means to drive this activity. They help designing the external boundaries of the system to be built but also the internal components required to implement the use cases. Obviously, functional requirements also have priorities which is why the use case priorities should drive the process.

Then we need to take into account infrastructural issues such as distribution, security, ... According to their priorities, we start with the most important infrastructure which needs to integrate into and with the functional model. For example, if we need to add distribution-specific infrastructure to our Web shop, we would introduce a broker-based architecture with customer browsers being the clients and the web shop being the server. Now let us assume, security is the next important requirements. We are now going to integrate an appropriate securiy infrastructure into our result we achieved after embedding the functional entities into the distributed infrastructure. We will add security components such as firewalls, identity management functionality, ... We continue with this approach until we have considered all operational requirements. What we did is extending the functional model (problem domain driven) with non-functional infrastructure parts (solution domain driven).

In the third step we add the infrastructure required by developmental qualities such as configurability, extensibility, and the like. This means that all those strategies, interceptors, configuration interfaces will be added at the end.

What we reached is an onion model with the functional architecture being the core part, the layers derived from the operational/infrastructural requirements being the middle part, and the developmental infrastructures being the outer part.
If you think about it, this is exactly what a building architect or other engineering disciplines do. Our resulting software architecture represents an integration of different functional and non-functional perspectives which is why I call it Multi-Facet Design.

But what about Software Product Line Engineering, In SPLE we will have to introduce a Commality/Variability analysis in order to determine all commanlities and variants. Variants will be considered as variations of a commonality and hence can be defined in terms of a commonality. Sounds abstract, but let me introduce an example. If we take our Web shop, we could define that we need to support different payment options (credit card, bank transfer, Paypal) depending on the customer. The variability is now defined as possible payment options. But the commonality in this example is the payment system itself. As a consequence, we need to consider the payment component in the core design, but specify the exact variations in the developmental design. For example, we could introduce the strategy pattern for the payment system to allow different options.

What about embedded systems?
All stringent constraints emerging from embedded systems such as scarce memory resources or CPU limitations or limited battery life will be handled as high priority requirements. One of the consequences is that static configurations are always preferred over dynamic configurations in such systems to allow determinism and control of QoS properties. Needless to say that realtime capabilities place additional burden to the engineer.

Requirements

2007-12-17T22:48:00.001+01:00

Recently, I have been involved in consulting some friends about a software development project. At some point in time we started intensively discussing requirements. Soon I recognized there were some funny misunderstandings related to what requirements are. Sentences such as "Oh, all these use cases are our requirements" emerged from the void of the universe and gave me some hints there could be a problem. When I asked them for their use-cases, the only thing they could show me was a UML use case diagram. Not really what you expect as an architect. So, my first activity was explaining the theory of requirements, at least my small, probably incomplete perspective on the world of requirements.

Functional requirements define the what of the system in terms of the problem space. Thus, they should be expressed in the language of the target domain. All functional requirements span a full range of possible solutions.

Non-functional requirements come in two flavors, operational requirements and developmental requirements. They impose constraints on the full range of solutions and thus represent beasts of the solution domain.

Operational requirements are directly related to the operation of the software system. They define qualities such as security, performance, fault-tolerance and can be measured or tested.

Developmental requirements are directly related to the architecture of the software system. They define qualities such as maintainability, usability, or flexibility. While some metrics exist for developmental requirements, they are normally not measurable.

It is interesting that most projects don't fail because of functional requirements but because of failing to meet the non-functional ones. Thus, it is often recommendable to assign sub teams or at least experts to the most important non-functional qualities in the project. They should then prepare checklists, strategies and tactics for ensuring the quality they are responsible for.

I often use the invasiveness as an additional property. A concrete instantiation of a requirement is invasive if it has an impact on the functional subsystems or relationships. Cross-cutting concerns such as security are mostly invasive.

To express functional requirements use-cases represent an excellent means. Note: Use cases are textual descriptions with sections such as actors, goals, exceptions, preconditions. A use case diagram is merely a graphical summary of use cases. For all other, i.e. non-functional requirements I recommend using utility trees such as introduced by Bass, Clements, Kazman. For example, a requirement such as flexibility could imply a lot of different things, maybe changeability, extensibility, or removability.

In the activity of architecture design, all decisions are based on requirements. Unfortunately, requirements can contradict each other. For instance, mind the availability/consistency paradox or flexibility versus performance. This means that all requirements must get a unique priority to let architects decide which path to choose at all decision points. NEVER accept requirements to have the same priority. If they have, assign priorities yourself. This also means: all architectural decisions must be solely based on requirements. Don't create design pearls, create the simplest possible architectures that work (i.e., that meet their requirements)!

Features are also requirements. So are commonalities and variants. However, they are at a very high level of abstraction and need to be mapped to finer levels first.

Requirements are the most important entities in a software development project. They help asking the right questions and making the right decisions. Requirements are a tool to constrain the problem and solution domain in such a way that an implementation can be developed effectively and efficiently. If you don't value requirements, you are doomed to fail, because you don't know what you are expected to build. Requirements are your friends, treat them as such!

About Learning & Documenting

2007-12-12T23:20:00.001+01:00

Currently, I am pretty busy. That's the reason why I couldn't add more postings for a while. This posting will also be a short one.

It might seem somehow unrelated to architecture, but nonetheless I find it valuable. I am currently very busy preparing some presentations as well as an article on Software Architecture Refactoring for the german OBJEKTspektrum magazine.

I made some observations about my work style in the last months that I'd like to share with you.

For learning new IT-related things I am often documenting what I learned using Powerpoint presentations. Then, I will constantly rearrange the slides in such a way that they reflect how I consider the material should be organized to help educating others in a better, more effective way. Powerpoint also serves as a tool for organizing my thoughts and works better for me than mind mapping tools for this purpose.

When I am writing an article, I often prepare the material as a presentation and then use the presentation as a story line for the article. This helped me writing scientific papers as well as other professional articles. The presentation contains much less content by nature but forces me to design the rough storyline, while I am sometimes lost in details when I start writing articles.

I often also use pattern templates to organize my efforts when I need to solve architectural problems. Pattern templates are very powerful, because they require me thinking in terms such as context, problem, forces, solution, consequences and so forth.

In summary, I guess such tools really help me being better organized. Any similar experiences by someone else?

I am really wondering what tools and methods others recommend.

SOA Pitfalls - Integrating your Legacy Apps

2007-11-17T20:16:00.001+01:00

The Past: I can still remember the time when television was mostly black and white and color TVs were not widely available. One company constantly kept claiming in newspaper advertisements they had developed a color filter you just needed to place on your B+W TV set to transform it to a full Color TV set. What an easy solution! And the product did a perfect job given you enjoy random and constantly changing colors such as magenta faces or green skies. Sounds funny, but, of course, no one of us techno geeks would ever be so naive, right?

40 years later - The Present: Very often I am now involved in SOA projects. And in many cases, even developers believe it is sufficient to simply wrap existing application APIs using WS-* based interfaces to open and integrate all applications into SOA wonderland. However, you might ask, if that is so simple, why do so many SOA integration endeavors fail?

The Future: SOA integration is not as simple as providing plain adapters. If business processes are the main reason for SOA-enabling an enterprise, it is much more effective to identify an appropriate partitioning of functionality into separate services that are then subject to orchestration. As a first step you need to obtain a consistent and prioritized list of business requirements from which you then define your enterprise architecture in a top-down manner. This might typically begin with simple company internal workflows and end in complex B2B scenarios. You'll need to refactor or even reengineer existing applications if securing investments is important in your company. This implies opening applications by providing service interfaces (in a bottom-up approach). Of course, qualities such as operational and developmental ones introduce additional levels of complexity. For example, opening up an existing intranet application for a B2B partnership scenario requires a whole lot of security considerations.

In summary, the adapter pattern might be simple but its concrete application can incur a whole lot of complexity. Building SOA applications naively will save a lot of time and resources in the short term, but imply a large amount of additional costs in the mid- and long term.

What is Software Architecture

2007-11-16T10:24:00.000+01:00

Like Andrew Tanenbaum, I could argue that there are so many definitions for software architecture that it is difficult to choose from them. You may also define it as follows: Software architecture is taken into consideration only when it hurts.
One possible definition for software architecture could be as follows:Software architecture denotes the set of artifacts and practices required to build a software system so that it achieves its implicit and explicit qualities. It includes the systematic partitioning of the software system into appropriate subsystems and relationships as well as the guiding principles used for that purpose. The partitioning of responsibilities and interactions need to be modelled using an interrelated set of viewpoints to address the functional and non-functional qualities.
Note that I am using "systematic" as an important attribute here as I don't consider ad-hoc systems (for example just hacking a Java program) as software architecture. From this personal definition follows that software architecture is both an entity and an act.
I am wondering what your favourite of software architecture is. Maybe one of the dozens available at the CMU SEI webpages?

Beware of DSLs

2007-11-11T13:05:00.001+01:00

At ooPSLA 2007 in Montreal there had been a very entertaining (and educating) panel on object-oriented programming languages and Simula 67 as their common ancestor. And the panelists were pretty excellent: Anders Hejlsberg (C#), James Gosling (Java), Guy Steele (Lisp/Scheme/Fortress), Bertrand Meyer (Eiffel), Ole Lerman Madsen (Beta). Now you might ask how this is related to software architecture.

First of all, programming languages have some influence on the way we think about architecture. Don't believe those experts that want to make you believe architecture design is completely unrelated to paradigms and languages. For example, one of the goals of Simula 67 was to provide a means for modeling systems which often got lost in state-of-the-art languages.

And secondly, we are currently facing a lot of discussions about DSLs (Domain specific languages). The panelists expressed their concern that now people are starting to develop DSLs who have no experience in language design. It is not trivial to design a language that is complete and consistent as well as usable. Believe me, I worked on such topics during my time at university.

The conclusions the panelists drew was that they prefer to add more modeling capabilities to programming languages over DSLs. Ruby is one of the examples in this area, but ,of course, it is only the beginning. Upcoming languages such as Scala, offer a lot of cool features for this purpose.

I discussed that issue with Markus (Völter), one of the gurus in Model-Driven Software Development and he shared this conclusion.

My conclusion and observation: Bad DSL design can cause more harm than value. Only language experts should become DSL designers. Designing DSLs on top of programming languages might be an appropriate approach. I already discussed Integrated DSLs in a previous posting.

Project Diary

2007-11-11T12:43:00.001+01:00

Did you ever experience those never ending and and continuous discussions about project topics and decisions which you thought had been already addressed?

Did you ever read an architecture document feeling a little bit confused or lost, because you couldn't remember the rationale of all those decisions?

I am sure, you know what I mean. The problem with software development projects is that there are basically two sources of information besides personal communication:

Meeting Minutes
Architecture Documents

Unfortunately, most architecture documents only describe the what, but not the how or why. Meeting Minutes strive for brevity and often don't include all those discussions. In addition, spontaneous meetings of project (sub) teams are often not documented at all.

Your gut feeling may tell you that something is missing here.

What I propose for development projects is an additional document, which I call "Project Diary". This document does not need to be very formal. It should not describe what is already available in other documents (meeting minutes, architecture documents) but instead refer to these sources. And, of course, architecture documents and meeting minutes, should also refer to the project diary. Its sole purpose is to complement these aforementioned documents by adding information such as alternatives discussed for solving a particular problem and the reason why a specific decision was preferred.

The organization of such project diaries may be by date or by topic or by both.

I don't recommend any specific template to use. My experience with all kind of project documents tells me that it is more important to come up with a uniform, complete and consistent style than to strictly follow a specific template. Just take your style of choice.

Sometimes, project members don't like the additional efforts of a project diary, especially in very small teams. In these cases I often have written a project diary without telling anyone. As soon as critical situations or fruitless discussions appeared, I would then just read from my project diary. This way, I could convince many people that a project diary offers more value than costs.

Corny Joke - somehow adapted

2007-11-07T23:35:00.001+01:00

After their death three IT persons arrived in hell. Among them a senior manager, a consultant and a software architect. One of the devils was in charge of taking care of these unfortunates. However, hell population has the same kind of feelings towards IT experts like the rest of mankind. Thus, the devil offered a deal to the newcomers. "There is a chimpanzee around this corner. Each of you you will need to make the chimpanzee first laugh, then cry, and finally make him return back to his cage. If you succeed, we'll send you back to earth." First the senior manager approached the chimpanzee. No matter what he said or did, the monkey showed absolutely no reaction. Then the consultant tried his luck. After an hour he also gave up. Finally, it was the turn of the software architect. After a few seconds the chimpanzee started screaming with laughter. After some more seconds he was moved to tears. And as soon as the architect had spoken some additional words, the monkey started panicking, returned immediately to his cage, locked the door and threw away the key. "Ok" the devil said, "I will keep my word, but could you, please, tell me what exactly you said to the chimpanzee?" "Of course!", the architect responded, "First, I told him what job I have which made him laugh. Then I told him what income I get which made him cry. Finally, I told him that we are still searching for new architects!"

OOPLSA 2007 (Additional Info)

2007-10-28T11:11:00.001+01:00

All keynotes are now available as Podcasts. Interested? Click here!