Hitchhiker's Guide to Software Architecture and Everything Else - by Michael Stal

Thursday, January 31, 2008

Learning from failure: Bangalore and Software Architecture

I am currently enjoying my time here in Bangalore (India) where I am giving some talks and trainings on Software Architecture. Especially, the hospitality here is marvellous. Bangalore is a very fast growing city with very nice people and a hot spot due to its IT companies and IT experts. On one hand, all the high tech industry is quite impressing. On the other hand, it is astonishing to see hightech centers in a lowtech infrastructure. Interesting, you might think, but how is this related to software architecture? When speaking to colleages here and to my driver (you cannot and should not manage to drive yourself as someone more used to the US or German car driving patterns :-), they told me a lot of interesting things related to Bangalore's infrastructure which I consider likewise applicable to software architecture. Don't be embarassed that I explicitly use Bangalore. It is a pars-pro-toto. Any other big metropoles would do also, but now that I am here ....

  • Bangalore's population significantly increased over the last years. Unfortunately, the infrastructures could not be increased accordingly. What formerly has been garden city, a small green place, has now become a big metropole. A good example is the traffic infrastructure. It has turned to a significant bottleneck (also known as nightmare) so that it might take you an hour even for small distances. What can we learn as software architects: We must plan for non-functional qualities such as scalability and prepare the system to increase when the load increases. Doing this as an afterthought can prove to be almost impossible. Of course, this is hard for a city.
  • As an implication of the insufficient traffic infrastructure, drivers of cars, motorbikes, bicycles, busses, trucks, rickshaws will just use any available gap of a traffic lane. Thus 3 lane streets will become 6 lane streets. There is almost no traffic control except for a few police men (and I am not sure how much control thy really have). The most important tool is the horn to tell other persons that you are here. What can we learn as software architects: governance is an important issue. Make sure that everyone abides to the rules. Overload situations in a system also make it nondeterministic as you can never anticipate how much time particular activities will require.
  • If you drive around in Bangalore, you will find no direction signs at all. Want to go somewhere? Better know the route or follow a trial-and-error approach until the end. What we can learn as software architects: usability and habitability should be considered important. Otherwise, users may have a very unpleasent time with your product.
  • The current airport is a former military airport and thus can provide only limited and sometimes slow service. Now, the local government is opening a new international airport to solve the problem. Unfortunately, there will be no fast train or highway connections from the airport to the city center. People from Electronics City (the Silicon Valley of India) suppose, it will take them 3 hours to reach the airport. What can we learn as software architects: If a service is very performant or offers otherwise high quality of service, this won't help you much once it takes a long time to send your request to the
    service or to receive a response. You always need to look on the system as a whole. And you always should remember the weakest link in the chain defines what quality you will obtain.
  • To improve the infrastructure several construction works are on the way. For example, there will be a new multi lane highway. However, this highway will end in a two lane highway. Guess what what happens on the junction of these highways? What
    can we learn as software architects: Also mind the connections. Local optimizations might help locally but can not solve problems that should be addressed globally. First think strategically and then tactically.
  • In the previous days there were some news about people who suffered severe illnesses from drinking the local drinking water. It turned out that some people didn't follow the regulations and just installed sewage pipelines near water pipelines.
    Imagine what will happen if erosion leads to lecks where both systems join? What can we learn as software architects? First of all, it is not sufficient to provide guidelines and throw them over the fence. An architect must also control that developers abide to the rules. Second, design erosion can lead to unwanted side effects. Instead of waiting for problems to occur, it is better to apply refactorings or even a complete reengineering. The latter is almost impossible for a real city infrastructure. Third, you need to keep in mind how infrastructures interact with each other. Fourth, you need to plan upfront for modifiability and later evolution to prevent such workarounds.

Take this example and think about ULS (Ultra Large-Scale Systems). The ULS proponents compare ULS with building cities instead of building houses where you also will also have to deal with human interaction and creating/maintaining/evolving large infrastructures. As soon as we are starting to deal with such ULS (systems of systems) we need to make sure such design erosion will not happen. There is a lot to learn from real life for cyber techies.

Friday, January 25, 2008

My Trip to India

Tomorrow I will travel to Bangalore, India, for one week. There I will give some talks on Software Architecture in the context of a software architecture education program for our colleagues. I am really looking forward to this trip.

expect further postings after my return

Friday, January 18, 2008

(Re-)Use a Component

It sounds so incredibly simple. Suppose, you are currently developing a car navigation system. During lunch break you meet some friends who are working in another part of your company which has already developed a component  for calculating routes. "That's cool", you think. "Why not just take the same component which will significantly reduce costs". Now, you start dreaming about extra holidays in Honolulu paid by your boss for saving so much money. You can't stop immediately walking to Wally who is one of your cynical colleagues in order to check the value of your recent inspiration.

"Hmmh", he murmurs, "sounds like a really great idea". But then he continues, "... but there may be some - well - small problems on your way to paradise". He starts his endless list of potential hurdles:

  • The existing routing component which I will coin R has been developed for Windows Mobile, but your application needs to support Linux and Android as well. You'll require some wrapper facades to make R portable. This could take a significant amount of time and developers.
  • Also the interfaces of R are not exactly in the shape you expect. Your developers may integrate some of them using plain-vanilla wrappers while other interfaces will require significant refactorings. For example. incompatible data types, different protocols and some internal dependencies make your life much harder. And mind the multiple components dilemma! Components may influence each other. I remember commercial components that caused serious problems whenever we used them in combination.  And, of course, R also expects some interfaces from its environment which are currently not available in your application.
  • Another issue is the dependency problem. Component R requires different other libraries and components. This means, developers must either also migrate these other dependent software artifacts or try to transform the component in such a way that it can leverage similar artifacts of the target environment.
  • In the application for which the partner development team has created R  were some special requirements related to fault management, availability, security and reliability which are a little bit different in your application. Either you need to adapt your application to R (very unrealistic) or R must be adapted to your application.
  • As R is part of a multiple products some variability has been identified and appropriate variability mechanisms established within R. Unfortunately, these mechanisms do not match your variability requirements very well.
  • Another problem you may encounter is the 99% problem. R might provide almost all functionality you require. Only 1% is missing, but adding this 1% will require a significant amount of time and resources.
  • Tests for R have been developed for the exact context in which R currently resides. Thus, you'll need to establish modified test plans and a test strategy for quality assurance.
  • Unfortunately, you also have to consider an additional dimension. R currently is owned by the partner development which also is responsible for its evolution. With other words, bugs, technical innovations or requirement changes will cause the component to be constantly modified in the future. As R is owned by someone else, you cannot expect that the owning department will be able to consider your requirements in the first place.

Ok, sounds like bad news. And as you know, bad news are the only thing traveling faster than light. It is not a good idea to rip a component from its context and then hope it will run unchanged in another context. Instead, a lot of efforts are necessary to transform a component from context A to context B. And the problem will get even worse when the same component is supposed to execute in many contexts.

You learned your first lesson: ad-hoc re-use causes more harm than benefits. So let us take another direction. Why not take the component R and make it a re-usable component in your organization serving more than one application? With other words, let us take a systematic approach.

A systematic approach requires a completely different setting.

  • All shared components are developed for the application in different products from the beginning. You may not always start from scratch, but use existing components as a base.
  • In your organization you need to establish an independent team that owns the component (or various components) and acts as a supplier to all product groups. It is also the component group's responsibility to take care of quality assurance. Of course, it is necessary to establish some governance measures as well.
  • All product groups and the component group need to define a common development process which integrates component development & evolution with product development & evolution. Component release planning must follow release planning of products. However, different versions or branches of the same component may co-exist.
  • It is also essential that component development would suffer from a whole spectrum of diverse contexts in which components reside. Thus, products should abide to some constraints and share practices and guidelines. One example to make this point more concrete: If product P expects errors to be reported by exceptions, while product Q is using error values instead, how should a component be able to provide all of these diverse policies. This simply isn't feasible. In this case, the company policy  would force the component and all products to use exception handling.  Another example is documentation and design. Anyway, the organization should introduce (and enforce!) some common guidelines for product and component development.
  • For the case of bug fixes or other relevant changes to components a feedback loop between product teams and development teams needs to be established. 
  • It must also be defined which requirements the component group must address. This includes functional and non-functional qualities. Especially, the variability requirements need thorough consideration. A component should not be a jack of all trades (trying to meet all product requirements) but also can't be a master of none (just providing functionality without considering product requirements and usability issues).
  • Such a systematic approach does not only require changes of the development organization but of the organization as a whole. For instance, product managers must now deal with different groups. Test managers must adapt their test strategies. 

Obviously, such a systematic approach for re-use requires a cost benefit analysis. It is rather unlikely that an organization would establish such an approach for only two products that share the same component.

All your beautiful dreams have vanished. What looked so easy and promising, turned out to be rather complex and expensive. There is no free lunch.

But wait a second: what if your products often had a very similar scope containing almost the same functionality? What if were are in the business of building lots and lots of car navigation systems? What if these products could use all these powerful components for routing, MMI, GPS, speech recognition, ...?  What if you had a framework combining these components to a powerful and adaptable whole? All of a sudden, the sun is rising again. In this case, a systematic re-use might be expensive to establish but once appropriate processes and organization structures are in place, it will be very effective.  We have entered the universe of product line engineering.

Even then, you have to consider all the dangers lurking in derelict parts of the galaxy. Mind the dark energy that tears everything apart and mind the dark matter that leads to system collapse. Be aware of black hole components in your architecture!  But that's a different story.

Saturday, January 12, 2008

OOP Conference 2008

Only as a side remark. I will participate in this year's OOP conference giving a talk and a tutorial. Note: the presentation language will be german.

  • In the talk I will address Software Architecture Refactoring, currently my favorite topic. As far as I have heard over 170 people already have already registered for the talk. I am quire impressed how many people are interested in the topic. On the other side, architecture refactoring is something new (which really strike me when I came up with this field because I assumed that almost every architect should have knowledge in this field)
  • In the full day tutorial (on friday, the 25th January) I will introduce SOA with .NET. Despite of the title, the tutorial could also be interesting for people not interested in .NET, as I also will include many parts that address SOA in a more general way. But, of course, technologies such as WCF, WF and BizTalk will get sufficient coverage.

If you are planning to attend any of these events, don't hesitate to approach me.

Architectural Entropy

Several years ago I introduced the term entropy in the context of software architecture. Entropy defines the number of concepts appearing in a software architecture. If your design consists of two simple components associated with one unidirectional relationship, this design probably has a low entropy. I only say "probably" because it depends on whether the internals of the component are complex or not. If, however, you increase the number of components or relations or their internal complexity, entropy inevitably also increases. High entropy often is an indicator for high complexity. High complexity always means high entropy. But high entropy does not enforce high complexity. A system that consists of almost infinite simple components that are not related with each other implies high entropy but not high complexity. You should also consider the other direction. A system that consists of only one single god component may be very complex. Hence, entropy can non only be measured by addressing one view or abstraction level (e.g., the component layer) but must consider all views and abstraction layers.

Obviously, incremental architecture design must lead to higher entropy, because we constantly add things. One of the fundamental issues in achieving high  architecture quality is minimizing entropy. Activities such as software architecture refactoring are doing exactly this. Quality factors such as loose coupling, symmetry, conceptual integrity, orthogonality strive for reducing entropy or at least keeping it small.

The art of software architecture is controlling entropy.

Thursday, January 03, 2008

Context and Boundaries

One of the most important issues when starting a new project is the differentiation between those parts that are in the software system and those which are external. We need to specify clear boundaries between the system going to be built and its environment. This is an issue developers often neglect. One of the obvious implications of not considering boundaries are misunderstandings during the design and architecture modelling phase. Are these boundaries simply a consequence of the underlying domain? Not at all! If we are going to create a Web Shop, the order processing subsystem is definitely an important concept of the domain. However, we could either introduce our plain-vanilla, homegrown order processing subsystem or just integrate an existing system such as SAP R/3 to provide the same service. In other words, first we need to consider which services are required, then we need to decide whether we will build these services as part of our software system or just (re-)use other external services for the same purpose.
How to proceed: I am a big fan of component-based design. I just consider components to be independent artifacts (i.e., modular units of testing, deployment, implementation, modularization,...) providing services to their environment (services may be functionality or events or configuration interfaces) and consuming services from their environment (such as functionality or events). Note: components can aggregate other components. I will design all subsystems including the whole system itself using this component concept. Hence, I need to define which interfaces my system will provide to its environment and which it will need to obtain. For example, one typical required interface of a Web Shop is access to a credit card company to verify customer credibility.
When designing such context diagrams the obvious choice is to leverage UML component diagrams.