On vacation

If you already wondered, I am currently on vacation travelling abroads. That is the reason why I have only seldomly access to the Web. Now, I am sitting in front of an Internet PC in my hotel's lobby. I will be back on 28th March. Then, I will provide you with some new stuff. So far, I wish you some nice easter holidays.

DSLs revisited

DSLs are everywhere. At least, we can read and hear a lot of good things about Domain Specific Languages. Unfortunately, many experts I meet are not always that sure what a DSL really is.

As a lazy person, I will not bore you with my own definition but just cite what Markus Voelter once said. In a former posting Markus defined a DSL as follows:

A DSL is a concise, precise and processable description of a viewpoint, concern or aspect of a system, given in a notation that suits the people who specify that particular viewpoint, concern or aspect.


Given that definition, UML is a vehicle to define DSLs. Take use case diagrams as an example. However, UML might better suit software engineers.

An ADL (Architecture Description Language) like Wright is a formal language to describe architectural designs.

BPEL or ARIS are DSLs to describe business processes. While BPEL is much more system centric, ARIS can also be understood by business analysts.

Thought questions:

• Is a programming language such as Java also a DSL?


• And what about natural language?

DSLs are useful to describe a viewpoint, concern or aspect of a system. But, what means do we have to introduce our own DSLs?

• Meta data annotations (Java) or attributes (.NET) can serve as the core elements of a DSL. Take the different attributes WCF (Windows Communication Foundation) introduces to help describe contracts.


• UML provides a meta model for introducing DSLs as well as specific DSLs (the different diagrams to specify viewpoints).


• XML can serve as a foundation for introducing DSLs. Example: BPEL, WS-*, XAML.


• Programming languages like Ruby or Smalltalk let you intercept the processing of program statements, thus enabling DSLs. Example: Ruby on Rails.


• Integrated DSLs like LINQ are embedded into their surrounding language.


• APIs define a kind of language.


• The hard way might be designing your own language using plain-vanilla tools such as ANTLR. Note: It is not only the compiler or interpreter but also the run-time system we have to cope with.

According to the Pragmatic Programmers designing a DSL can be very effective.

• It may help you visualize and communication the viewpoint, aspect, concern the DSL addresses.


• It may enable you providing a generator that generates significant parts instead of forcing you to hand-craft. This is not restricted to code but you could also generate tests or documents.

Stefan Tilkov provided an excellent example. Suppose, you are developing a Java application. Writing the unit tests in JUnit would be the normal way of operation. But, maybe JRuby would be a much better way to write unit tests. As both run on top of the JVM, this can be achieved easily. In this context, we use JRuby+xUnit APIs as our DSL.

I believe, DSLs help climb over the one-size-fits-all barrier traditional programming languages introduce. Creating a DSL does not need to be as complex as defining a programming language. Using XML Schema or Meta data annotations or UML Profiles are nice examples.

Generative versus Generic Programming

Andrey Nechypurenko, a smart colleague of mine, recently brought up an excellent thought. If we already know a domain very well, why should we bother building model-driven generators? Couldn't we just provide an application framework instead? Indeed, each framework introduces a kind of language. It implements commonalities but also hooks to take care of variability. So, when should we use model-driven techniques and when should we better rely on frameworks?

My first assumptions were:

• A DSL is much closer to the problem domain which helps different stakeholders to participate establishing the concrete model. If stakeholder participation is essential, than a DSL might definitely be the better way.
• DSLs are (often) more productive. It is much more verbose to program against a framework than designing using a DSL. Needless to say, that we need to take the efforts for providing the generator into account.

At least in my opinion the answer could also be to use both, a framework that captures the domain's core abstractions and a generator that generates applications on top of this framework.

Any thoughts about these issues? 

Relativity

Today, I had an inspiration. My spouse told me it will take 5 minutes to get ready for leaving home, but actually it took her over 20 minutes. As those "events" constantly happen, my assumption is that women are moving with much faster speed than men. This way,  5 minutes on their spaceship equals 20 minutes of our time.

I also remember a professor at my university asking a student about the "evaluator problem".  The student did not really understand the question. Do you? Actually, the right answer turned out to be that he was using C style malloc for allocating storage in his compiler's attribute evaluator.

What can we learn from this? Humans might have a completely different perspective on reality depending on her/his background and context. Whenever we are dealing with architecture design, we'll have to communicate with all those stakeholders. In order to prevent misunderstandings like the ones illustrated above, we need to introduce a kind of formal language and a common scale - with other words: a common understanding.

One of the most critical points in design is the consideration and elicitation of requirements. This also holds for requirement priorities. For example, a project manager might be more interested in partitioning the architecture so that the different teams can develop in the most effective way, while an architect will follow a completely different rationale for modularisation.

In many projects I was also surprised by the fact how important a common project glossary is. Otherwise, people will interpret the same terms differently which is not a big boost for effective communication. For example, in one SOA-based project senior managers considered process monitoring to reveal the business perspective (Business Process Monitoring). They were interested to observe the number of successful business transactions or cash flow. What they actually got from development was system monitoring functionality such as showing load, bandwidth or faults. Guess, how long it took development to correct this?

It is very likely, that you also experienced the architecture/implementation gap in your career. This inevitably occurs whenever architects throw their design specification over the fence and then leave the development team alone.  Did you ever read a specification and understood it without any problems? Me neither! Some parts might be missing, other ones might not be implementable, and others might be contradicting. Even formal standards such as CORBA, SOAP, HTTP that were developed by expert groups within long time periods, reveal those problems. How can you then expect your own specification to be perfect? What is not perfect, will then be corrected or interpreted in unanticipated ways.

Thus, it is essential that such clarifications happen as soon as possible in a project.  It is your responsibility as an architect to tackle this problem. Remember how often I already mentioned that effective communication is the most important skill - one of the few moments where I do not abide to the DRY principle. There are lot of methods and tool that explicitly address this issue. Think of agile methods, domain driven design, use case analysis, to name just a few.

Reality is relative. Thus introduce a common frame of reference for the stakeholders. This holds for software engineering, but also for real life :-)

Workflows

Workflow engines such as Microsoft's WF (Workflow Foundation) are only rarely used these days. Most developers totally underestimate their power. However, in an integrated world where processes respectively workflows increasingly need to connect disparate heterogeneous applications, just hard coding these workflows in traditional languages such as Java does not make any sense. First of all, coding in Java or C# requires programming skills, while often business analysts are the ones in charge of business processes. And secondly, hard coding in Java, C#, C++ implies that all changes must happen in source code with the code base representing a beast containing both business logic and workflow related content.  

Before covering the meat of this posting, I need to define what I understand by a workflow. By the way, I will use the terms workflow and business process interchangeably as I don't consider the differences stated by other authors very convincing.

If we just refer to Wikipedia we will find the following definition:

• A workflow is a reliably repeatable pattern of activity enabled by a systematic organization of resources, defined roles and mass, energy and information flows, into a work process that can be documented and learned. Workflows are always designed to achieve processing intents of some sort, such as physical transformation, service provision, or information processing.

Sounds rather confusing, right? Let me try my own definition.

• A workflow defines a set of correlated activities each of which representing a (composite) component that takes input, transforms it, and sends the result via its output ports to subsequent activities. Transitions to an activity may be triggered implicitly by (human or machine initiated events) or by explicit transfer of control
• Workflows can be considered as a Pipes & Filters architecture with the filters being the activities and the pipes being the communication channels between them
• All activities within a workflow share the same context such as common resources
• Workflows are (composite) activities themselves
• A workflow language is a DSL that allows to declaratively express workflows. Examples include BPEL, ARIS, or XAML as used in WF
• A workflow runtime manages the (local) execution and lifetime of workflow instances

In fact, a workflow closely resembles traditional program code with the statements being the activities. 

A typical real-life example is visiting a restaurant.

1. The workflow starts when you enter the restaurant and arrive at the "wait to be seated" sign.
2. The waitress will now ask you whether you have a reservation.
3. If yes, she will look up the reservation and escort you to your table
4. If no, she will try to determine whether a table is available
5. If yes, she will escort you to the table
6. If no, she may ask you to come back later => stop
7. The waitress will provide you with a menu
8. At the same time, another waitress might offer you water
9. ...

You will recognize easily that UML activity diagrams could be used to design workflows. Activities are either triggered by events or by active commands (e.g., you ask the waitress to bring you another beer which is like an event or she could ask you if you need anything else which is more like an "invocation"). Thus, you could either use such a diagram to express a workflow or maybe a more domain-specific kind of language/model such as BPEL. This language could be graphical, textual, audio-based, and so forth.

The advantage of a (domain-specific) language is obvious. It is much easier for domain experts to express the workflows themselves. Moreover, the workflow descriptions could be either used to generate the actual code or to execute the workflow in a runtime engine.

Sometimes workflow transitions depend on conditions such as "if customer is a special guest, give him a window table." These conditions however might also be subject to change. Coding such conditions in activities thus reveals the same problems as coding workflows in C# or Java. As a consequence, we should also specify such rules in a declarative fashion. This is where Business Rules Engines enter the stage. Business rules might be explicitly expressible in a workflow language or just be hidden in special activity types.

How can we retrieve the workflows in the act of architecture creation. User stories, Use cases and scenarios, help determing the workflow in a system. Which of these should be hard coded and which ones should be better implemented using a Workflow language, depends on requirements, frequency of change, required variability, and other factors, questions a software architect needs to address.

Removing unnecessary abstractions

A good architecture design should always only contain a minimal set of abstractions and dependencies. According to Blaise Pascal a system is simple, when it is not possible to remove something without violating the requirements. Again, we can consider this from an architecture entropy viewpoint. Minimalism and simplicity imply the lowest possible entropy.

Let me give you an example how a design should not look like. Suppose, you are developing a graphical editor. In the first design sketch you introduce an abstraction called Shape with functionality such as Draw(). From this base type you are then inheriting subtypes such as PolygonicShape and RoundedShape. Finally, the abstractions Rectangle and Square and Triangle will become direct descendants of PolygonicShape, and so forth.

After a while you might recognize that it does not make any sense to introduce the intermediary layer consisting of PolygonicShape and RoundedShape. You find out that in your context all concrete classes such as Ellipse, Rectangle, Triangle, ... should be directly derived from Shape, because you don't differentiate between types of shapes in your software.

By removing the intermediary layer you have eliminated unnecessary abstractions and dependencies. Obviously, this also adheres to the KiSS Principle. But why, you may ask, are we not removing Shape too? The answer is very simple: Shape introduces common behavior required to handle all concrete shape subtypes uniformly in the editor. Otherwise, we would need to introduce large switch statements, type codes or reflection mechanisms in several places of our application. Thus, removing Shape would impose a much higher entropy.

As you can surely see, determining the applicability of this architecture refactoring is not trivial, because you need to consider different layers and levels of abstractions in your design. For example, in another context it might be important to introduce intermediary abstractions such as PolygonicShape because you need to treat polygons different from rounded shapes. In this sense truth is always relative.

Work Life Balance - Revisited

Right after each of my postings related to Work Life Balance, my inbox grew due to people asking me about how I personally keep the balance. That's the reason why I want to spend some sentences on this personal issue although it is not related to Software Architecture (al least at the first sight).

I still consider myself a workaholic on rails. Only a few years ago I used to work the whole week without too many exceptions. After some health problems, I remembered the time when I had been totally enthusiastic about sports. And then I started to go biking and running regularly. That's where the rails come in. Ok, I did too much running and got problems with my knee which is the reason why I am more biased to biking these days. Typically, I will run or bike almost every day from spring to autumn. For example, I enjoy to bike to work (10 km). On my way back to home I will often use a detour leading through a large forest and along a bike lane near the Isar river here in Munich (30-50 km).  Sometimes, I leave office late afternoon and continue my work at home. If possible, I am also running which is an incredibly excellent way to relax, especially in winter when biking is not that cool (or should I better  say when it is too cool). After 10-12 km which may take more than 60 minutes depending on my training intensity all problems are gone and my mind is open for new ideas.

My other hobbies include composing music, reading books, listening to podcasts, audiobooks or music on my IPod, among many other interests. Yes, I definitely got more interests than time.

My experience has taught me that sports (no matter what) is the most important way to keep your balance. So is meeting people who have no interest in any IT topics. Same for all hobbies that help forgetting about any work related challenges (formerly known as problems). I have never been as creative as when keeping my work/life in balance. The more time I spend for sports and other interests, the more productive I will be. This is kind of surprising as I always believed in the opposite statement (the more time spent, the more productivity).

Just relax! 

All about Cycles

Here comes a real world example. In a management application the distributed agents in charge of managed objects (such as routers, load balancers, switches, firewalls) are accessed by a centralized monitoring application.  For this purpose, the sensors offer functionality such as setter/getter functions. The monitoring subsystem inherently depends on the agents it monitors. After a while, the developers of the agent subsystem think about returning time stamps to the monitoring subsystem whenever it requests some information. But where could they obtain these central time stamps? The team decides to add time stamp functionality to the central monitoring subsystem. Unfortunately, the agents now depend on the monitoring subsystem. Developers have established a dependency cycle.

"So, who cares?", you might say. The problem with such circular references is that they add accidental complexity. Mind the implications such as:

• If there is a cycle within your architecture involving two or more subsystems, you won't be able to test one of the subsystems in isolation without needing to test the other ones.
• Re-using one of the subsystems in the cycle implies you also need to re-use the other subsystems in the cycle.

It is pretty obvious that dependency cycles are a bad thing. They represent an architectural smell. But how can we cure the problem? Can you hear the horns playing a fanfare? The answer is: by applying an architectural refactoring!

So, how could we get rid of a dependency cycle?

• Option 1: by redirecting the dependency to somewhere else. In the example above, we could introduce a time subsystem the agents use to obtain time stamps instead of adding this functionality to the monitoring subsystem. This way, we broke the cycle.
• Option 2: by reversing one of the dependencies. For example if we exchange a publisher/subscriber pull model (event observer itself asking for events) with a push model (event provider always notifies event consumer). Another example is introducing interfaces: instead of letting a component depend on another component we could introduce interfaces. This way, a component always uses interfaces and never component implementations directly.
• Option 3: by removing the dependency completely if it is not necessary. For example consider Master/Detail relationships in relational databases. Only one dependency direction makes sense here. The other one is obsolete.

Dependency Injection as provided by all those IoC Containers is very helpful to avoid dependency cycles. Even if you got no dependency cycles in your system, you should always avoid unnecessary dependencies - because they inevitably increase entropy and thus the chance of circular references.

But how can we detect such dependency cycles within our system? This is where CQM (Code Quality Management) tools enter the stage. Of course, these tools are not restricted to problems with dependencies. They are also helpful with a lot of other architecture quality related measurements.

Can we always avoid cycles? No, because sometimes they are inherent in the domain. For example, in Microsoft COM due to the usage of reference counting cycles were required in graph structures consisting of COM objects. However, one could certainly ask if the design of COM was appropriate in this respect. But that's a completely different issue.

Thus, the general guideline should always be: avoid unnecessary (accidental) dependencies, break dependency cycles!

Emergence

Suppose, you are going to develop a traffic control system for a rather large city. The goal is to optimize traffic throughput which could be measured by average miles of each cyclist and motorist. In a first approach, you may suggest to use a purely centralized control subsystem.

There are various problems with this task:

• In a rather large traffic infrastructure, traffic can never be controlled from a centralized component. First of all, this component would represent a single point of failure. Secondly, it would not scale or perform at all due to the sheer amount of traffic signals involved.
• The requirement of traffic optimization reveals a significant problem. Assume, that there is a main road through the city. The best way to achieve the goal could be to just set all traffic lights on the main road to green. Participants driving on side streets would then need to wait forever. That's what is called starvation in multithreading.

How could we solve those problems?

• Instead of using a centralized approach, we could rely on a completely decentralized approach. For example, each street crossing could permanently measure the traffic and adapt its traffic lights automatically to the current situation. If all street crossing act independently, local tactics might dominate global strategy. Obviously, this is not what we want. Strategy should always dominate tactics. Hence, let local decisions of street crossings depend on "neighboring" street crossings. If appropriate, we could also mix centralized control with decentralized autonomous adaptations. That basically means, that we control the overall strategy, but allow the system to automatically and locally adapt its tactics. Is it always useful to apply decentralized approaches? No. Here is a real life experiment: Take a number of people in a row each of them stretching one of their fingers. Put a large stick on their fingers. And then tell the group to systematically and gently put down the stick to the ground. They won't succeed until finally one person will become the leader and coordinate the whole action.
• We also learn from the example that we need really to address requirements with much more care, especially in systems with complex interactions. In the example above the target function should not only consider average flow but also take possible starvation into account. By the way, the traffic example could also be replaced by a Web shop and Web traffic.  In this example, starvation would imply that some shoppers would have a fantastic experience while others could not receive any Web page in an appropriate response time. Guess, how often the latter ones would re-visit the shop. Wrong understanding of requirements can seriously endanger your whole business goals. 

The question is how to design such decentralized systems. There are lots of options:

• Infrastructure: P2P networks are one way to organize resources so that they can be located and used in a decentralized way. Cloud Computing formerly known as Grid Computing is another cool example.
• Algorithms: Genetic algorithms help systems to automatically adapt themselves to their context.
• Cooperation models: Swarm Intelligence such as ants also illustrates how emergent behavior can substitute centralized control.
• Architecture: Patterns such as Leaders/Followers or Blackboard help introduce decentralized approaches.
• Technologies: Rules engines and declarative languages such as Prolog are useful for providing decentralized computing.

The problem with decentralized versus centralized systems is often when to use what. And in addition, to decide how much decentralized the system should be. This is not always obvious. Let me give you some examples instead:

• In a sensor network you might place many small robots to a specific area in order to monitor environmental data. These robots would be completely autonomous. Even if some of them fail, the overall system goal can be achieved.
• In a Web Crawler several threads could search for URLs in parallel, but they would need to synchronize themselves (e.g., using a tuple space) in order to prevent endless loops. This resembles ant populations.
• In a car or plane you need central control, but some parts can also operate and adapt  independently such as Airbags or ESP. This is more a hybrid approach with the global strategy implemented by a centralized approach.
• In embedded (real-time) controllers (such as your mobile phone) everything typically is centralized  because behavior must be statically predicted and configured.

The more co-operation your components require and/or the more determinism you need, the less a decentralized approach seems appropriate, and vice versa. 

The best way to design such decentralized systems is to introduce the concept of emergent behavior. For example, define a set of simple local rules to which the system parts must abide. And then let the parts take control.  By the way, this is exactly how the Internet works. But in the Internet, you will also find a hybrid approach combining centralized backbones with decentralized nodes. 

Nervous Breakdown by Communication Overkill

Whenever I meet IT experts in airline lounges or conference rooms they are often very busy in synchronizing their memory with the continuous information flow from their company, customers, and other sources. Typically, they are armed with a quite impressive weaponry that includes blackberries, (>= 1) mobile phones, notebooks, and PDAs. Asking them about the effectiveness and efficiency of their work, they mostly complain about the impossibility to perform real work in their office in addition to the burden of all these travels and meetings. If we did a cost/benefit analysis relating the quantity of information exchange to work effectiveness, we would certainly be (negatively) surprised.

Let me just provide a theoretical example.

• In average, meetings will require 2 hours per day. Given the fact, how ineffective meetings are planned and performed, I assume that at least 2 thirds of the time spent is a complete waste.
• If you receive 100 mails per day, then throwing away 50 of them, might only take 5 minutes. However, reading 30 mails will require additional 30 minutes. And answering the remaining 20 will require 60 minutes. Thus, you've spent at least 1 1/2 hours for handling email.
• Travel might also require 1 hour in average per day.
• From time to time, you will have some personal face to face communication with colleagues, some of them just flocking to your office.
• Telephone calls will require at least another hour per day.

Each media break (such as the arrival of e-mail, SMS, phone calls, meetings, etc) also interrupts your work and causes continuous context switches. As you know, context switches are ineffective time durations, not only in a CPU but also in real life.

If you summarize all of this, it is not very difficult to recognize the problem.

How does this relate to software architects? As you definitely know, a software architect is in charge of designing a software architecture which requires a lot of communication activities. The more ineffective communication proves to be, the less time can be spent for the actual job.

We are all part of a vicious cycle that involves ineffective communication, unnecessary breaks and context switches. The only way to solve the problem is not to stop communicating but to spend your time much more effectively.

Personally, I consider the following means very helpful:

• If you need to work on a document or review, use your home office, given that you can work there without any major interrupts.
• When you feel tired, exhausted or relax, go to the gym, or do whatever you feel appropriate to improve your mood. It is simple math. If you need to spend n hours for your work, it is not necessary to spend these hours in a row. For your company, mixing work time and recreation time is much more productive and less error-prone, especially when your job requires creativity.
• Time box all meetings. I found out that m one-hour meetings are much more effective than one m-hour meeting. Needless to mention, meetings should be effective themselves (agenda known in advance, specific and clear objectives of the meeting available, have a moderator, and so forth).
• Prioritize mails and phone calls. Tell people and train them, that you are not able to answer all mails or phone calls all the time. Consider voice-mail and mailboxes as friends. They allow you answering whenever you are prepared, not when the callers are. Switch off mobile phones, blackberries when you are in meetings, need to relax. Not to communicate sometimes is not a luxury but a necessity. By the way, it is not rude to tell colleagues that you are not ready to talk to them but that you will come back tothem later, if you are currently busy.
• In your spare-time or holidays don't accept any business-related communication except for emergencies.
• Travel is important as face-to-face meetings are the most effective form of communication, but don't underestimate the impact of traveling on your effectiveness. Thus, only travel when really necessary.
• Plan your time thoroughly (time management). And also plan sufficient spare time for recreation to keep your work/life balance and may be even your relationships healthy.
• Diversify your work day. Three meetings or telephone conferences in a row are boring, exhausting and ineffective. The same holds for sitting in front of your PC or notebook (or TV) for several hours.   

There is the common misconception that knowledge workers such as software architects should always be available for communication and work 12 hours a day without any break. Eventually, the job of architects is to build high quality software architectures, and to do this successfully they need motivation, fun, and focus. In the end, multitasking such as communication in one thread while designing in the other is not feasible for humans. Sufficient recreation time (for example due to avoidance of communication overkill) is an essential means to do our jobs successfully.

Sometimes relaxing is more productive than working.

Architecture Governance

I consider cross-cutting concerns such as error handling as one of the major challenges in any software development project. These cross-cutting issues should be subject to what I call architecture governance. 

• We need to establish rules how to uniformly handle these issues throughout the project. Otherwise, every developer or architect will provide her/his own solution which significantly reduces symmetry and orthogonality.  I've seen many projects where error reporting was provided by some subsystems using return values while others used exceptions instead. Even in projects where exception handling had been common practice, developers introduced their own exception types for similar exceptions. Guess, how well such an architecture is balanced.
• We need to define roles and persons in charge of supervising software development and enforcing the rules. In one project architects had created documents for all of these concerns. Unfortunately, developers didn't care about the documentation. Policy enforcement tools such as FxCop for .NET can provide help. Trust is good but control is better.

The best way to guarantee proper handling of these concerns is to provide guidelines from day 0. Architects are in charge to establish a strategic architecture document as well as the design guidelines and programming conventions in the beginning, not as an afterthought. All places where those rules are then violated within the project are subject to refactoring.

What are typical cross-cutting concerns in this context? Examples include:

• Error Management
• Security Checks (such as most cross-cutting non-functional qualities)
• Naming Conventions
• Mandatory Patterns

Why is that important? Because allowing everyone doing whatever she/he considers best, will lead to less expressiveness and readability, high accidental complexity, less developer usability, missing orthogonality and symmetry. With other words, it will have an negative impact on all architecture qualities.   

This is why Architecture Governance is essential!

Learning from failure: Bangalore and Software Architecture

I am currently enjoying my time here in Bangalore (India) where I am giving some talks and trainings on Software Architecture. Especially, the hospitality here is marvellous. Bangalore is a very fast growing city with very nice people and a hot spot due to its IT companies and IT experts. On one hand, all the high tech industry is quite impressing. On the other hand, it is astonishing to see hightech centers in a lowtech infrastructure. Interesting, you might think, but how is this related to software architecture? When speaking to colleages here and to my driver (you cannot and should not manage to drive yourself as someone more used to the US or German car driving patterns :-), they told me a lot of interesting things related to Bangalore's infrastructure which I consider likewise applicable to software architecture. Don't be embarassed that I explicitly use Bangalore. It is a pars-pro-toto. Any other big metropoles would do also, but now that I am here ....

• Bangalore's population significantly increased over the last years. Unfortunately, the infrastructures could not be increased accordingly. What formerly has been garden city, a small green place, has now become a big metropole. A good example is the traffic infrastructure. It has turned to a significant bottleneck (also known as nightmare) so that it might take you an hour even for small distances. What can we learn as software architects: We must plan for non-functional qualities such as scalability and prepare the system to increase when the load increases. Doing this as an afterthought can prove to be almost impossible. Of course, this is hard for a city.
• As an implication of the insufficient traffic infrastructure, drivers of cars, motorbikes, bicycles, busses, trucks, rickshaws will just use any available gap of a traffic lane. Thus 3 lane streets will become 6 lane streets. There is almost no traffic control except for a few police men (and I am not sure how much control thy really have). The most important tool is the horn to tell other persons that you are here. What can we learn as software architects: governance is an important issue. Make sure that everyone abides to the rules. Overload situations in a system also make it nondeterministic as you can never anticipate how much time particular activities will require.
• If you drive around in Bangalore, you will find no direction signs at all. Want to go somewhere? Better know the route or follow a trial-and-error approach until the end. What we can learn as software architects: usability and habitability should be considered important. Otherwise, users may have a very unpleasent time with your product.
• The current airport is a former military airport and thus can provide only limited and sometimes slow service. Now, the local government is opening a new international airport to solve the problem. Unfortunately, there will be no fast train or highway connections from the airport to the city center. People from Electronics City (the Silicon Valley of India) suppose, it will take them 3 hours to reach the airport. What can we learn as software architects: If a service is very performant or offers otherwise high quality of service, this won't help you much once it takes a long time to send your request to the
  service or to receive a response. You always need to look on the system as a whole. And you always should remember the weakest link in the chain defines what quality you will obtain.
• To improve the infrastructure several construction works are on the way. For example, there will be a new multi lane highway. However, this highway will end in a two lane highway. Guess what what happens on the junction of these highways? What
  can we learn as software architects: Also mind the connections. Local optimizations might help locally but can not solve problems that should be addressed globally. First think strategically and then tactically.
• In the previous days there were some news about people who suffered severe illnesses from drinking the local drinking water. It turned out that some people didn't follow the regulations and just installed sewage pipelines near water pipelines.
  Imagine what will happen if erosion leads to lecks where both systems join? What can we learn as software architects? First of all, it is not sufficient to provide guidelines and throw them over the fence. An architect must also control that developers abide to the rules. Second, design erosion can lead to unwanted side effects. Instead of waiting for problems to occur, it is better to apply refactorings or even a complete reengineering. The latter is almost impossible for a real city infrastructure. Third, you need to keep in mind how infrastructures interact with each other. Fourth, you need to plan upfront for modifiability and later evolution to prevent such workarounds.

Take this example and think about ULS (Ultra Large-Scale Systems). The ULS proponents compare ULS with building cities instead of building houses where you also will also have to deal with human interaction and creating/maintaining/evolving large infrastructures. As soon as we are starting to deal with such ULS (systems of systems) we need to make sure such design erosion will not happen. There is a lot to learn from real life for cyber techies.

My Trip to India

Tomorrow I will travel to Bangalore, India, for one week. There I will give some talks on Software Architecture in the context of a software architecture education program for our colleagues. I am really looking forward to this trip.

expect further postings after my return

(Re-)Use a Component

It sounds so incredibly simple. Suppose, you are currently developing a car navigation system. During lunch break you meet some friends who are working in another part of your company which has already developed a component  for calculating routes. "That's cool", you think. "Why not just take the same component which will significantly reduce costs". Now, you start dreaming about extra holidays in Honolulu paid by your boss for saving so much money. You can't stop immediately walking to Wally who is one of your cynical colleagues in order to check the value of your recent inspiration.

"Hmmh", he murmurs, "sounds like a really great idea". But then he continues, "... but there may be some - well - small problems on your way to paradise". He starts his endless list of potential hurdles:

• The existing routing component which I will coin R has been developed for Windows Mobile, but your application needs to support Linux and Android as well. You'll require some wrapper facades to make R portable. This could take a significant amount of time and developers.
• Also the interfaces of R are not exactly in the shape you expect. Your developers may integrate some of them using plain-vanilla wrappers while other interfaces will require significant refactorings. For example. incompatible data types, different protocols and some internal dependencies make your life much harder. And mind the multiple components dilemma! Components may influence each other. I remember commercial components that caused serious problems whenever we used them in combination.  And, of course, R also expects some interfaces from its environment which are currently not available in your application.
• Another issue is the dependency problem. Component R requires different other libraries and components. This means, developers must either also migrate these other dependent software artifacts or try to transform the component in such a way that it can leverage similar artifacts of the target environment.
• In the application for which the partner development team has created R  were some special requirements related to fault management, availability, security and reliability which are a little bit different in your application. Either you need to adapt your application to R (very unrealistic) or R must be adapted to your application.
• As R is part of a multiple products some variability has been identified and appropriate variability mechanisms established within R. Unfortunately, these mechanisms do not match your variability requirements very well.
• Another problem you may encounter is the 99% problem. R might provide almost all functionality you require. Only 1% is missing, but adding this 1% will require a significant amount of time and resources.
• Tests for R have been developed for the exact context in which R currently resides. Thus, you'll need to establish modified test plans and a test strategy for quality assurance.
• Unfortunately, you also have to consider an additional dimension. R currently is owned by the partner development which also is responsible for its evolution. With other words, bugs, technical innovations or requirement changes will cause the component to be constantly modified in the future. As R is owned by someone else, you cannot expect that the owning department will be able to consider your requirements in the first place.

Ok, sounds like bad news. And as you know, bad news are the only thing traveling faster than light. It is not a good idea to rip a component from its context and then hope it will run unchanged in another context. Instead, a lot of efforts are necessary to transform a component from context A to context B. And the problem will get even worse when the same component is supposed to execute in many contexts.

You learned your first lesson: ad-hoc re-use causes more harm than benefits. So let us take another direction. Why not take the component R and make it a re-usable component in your organization serving more than one application? With other words, let us take a systematic approach.

A systematic approach requires a completely different setting.

• All shared components are developed for the application in different products from the beginning. You may not always start from scratch, but use existing components as a base.
• In your organization you need to establish an independent team that owns the component (or various components) and acts as a supplier to all product groups. It is also the component group's responsibility to take care of quality assurance. Of course, it is necessary to establish some governance measures as well.
• All product groups and the component group need to define a common development process which integrates component development & evolution with product development & evolution. Component release planning must follow release planning of products. However, different versions or branches of the same component may co-exist.
• It is also essential that component development would suffer from a whole spectrum of diverse contexts in which components reside. Thus, products should abide to some constraints and share practices and guidelines. One example to make this point more concrete: If product P expects errors to be reported by exceptions, while product Q is using error values instead, how should a component be able to provide all of these diverse policies. This simply isn't feasible. In this case, the company policy  would force the component and all products to use exception handling.  Another example is documentation and design. Anyway, the organization should introduce (and enforce!) some common guidelines for product and component development.
• For the case of bug fixes or other relevant changes to components a feedback loop between product teams and development teams needs to be established. 
• It must also be defined which requirements the component group must address. This includes functional and non-functional qualities. Especially, the variability requirements need thorough consideration. A component should not be a jack of all trades (trying to meet all product requirements) but also can't be a master of none (just providing functionality without considering product requirements and usability issues).
• Such a systematic approach does not only require changes of the development organization but of the organization as a whole. For instance, product managers must now deal with different groups. Test managers must adapt their test strategies. 

Obviously, such a systematic approach for re-use requires a cost benefit analysis. It is rather unlikely that an organization would establish such an approach for only two products that share the same component.

All your beautiful dreams have vanished. What looked so easy and promising, turned out to be rather complex and expensive. There is no free lunch.

But wait a second: what if your products often had a very similar scope containing almost the same functionality? What if were are in the business of building lots and lots of car navigation systems? What if these products could use all these powerful components for routing, MMI, GPS, speech recognition, ...?  What if you had a framework combining these components to a powerful and adaptable whole? All of a sudden, the sun is rising again. In this case, a systematic re-use might be expensive to establish but once appropriate processes and organization structures are in place, it will be very effective.  We have entered the universe of product line engineering.

Even then, you have to consider all the dangers lurking in derelict parts of the galaxy. Mind the dark energy that tears everything apart and mind the dark matter that leads to system collapse. Be aware of black hole components in your architecture!  But that's a different story.

OOP Conference 2008

Only as a side remark. I will participate in this year's OOP conference giving a talk and a tutorial. Note: the presentation language will be german.

• In the talk I will address Software Architecture Refactoring, currently my favorite topic. As far as I have heard over 170 people already have already registered for the talk. I am quire impressed how many people are interested in the topic. On the other side, architecture refactoring is something new (which really strike me when I came up with this field because I assumed that almost every architect should have knowledge in this field)
• In the full day tutorial (on friday, the 25th January) I will introduce SOA with .NET. Despite of the title, the tutorial could also be interesting for people not interested in .NET, as I also will include many parts that address SOA in a more general way. But, of course, technologies such as WCF, WF and BizTalk will get sufficient coverage.

If you are planning to attend any of these events, don't hesitate to approach me.

Architectural Entropy

Several years ago I introduced the term entropy in the context of software architecture. Entropy defines the number of concepts appearing in a software architecture. If your design consists of two simple components associated with one unidirectional relationship, this design probably has a low entropy. I only say "probably" because it depends on whether the internals of the component are complex or not. If, however, you increase the number of components or relations or their internal complexity, entropy inevitably also increases. High entropy often is an indicator for high complexity. High complexity always means high entropy. But high entropy does not enforce high complexity. A system that consists of almost infinite simple components that are not related with each other implies high entropy but not high complexity. You should also consider the other direction. A system that consists of only one single god component may be very complex. Hence, entropy can non only be measured by addressing one view or abstraction level (e.g., the component layer) but must consider all views and abstraction layers.

Obviously, incremental architecture design must lead to higher entropy, because we constantly add things. One of the fundamental issues in achieving high  architecture quality is minimizing entropy. Activities such as software architecture refactoring are doing exactly this. Quality factors such as loose coupling, symmetry, conceptual integrity, orthogonality strive for reducing entropy or at least keeping it small.

The art of software architecture is controlling entropy.

Context and Boundaries

One of the most important issues when starting a new project is the differentiation between those parts that are in the software system and those which are external. We need to specify clear boundaries between the system going to be built and its environment. This is an issue developers often neglect. One of the obvious implications of not considering boundaries are misunderstandings during the design and architecture modelling phase. Are these boundaries simply a consequence of the underlying domain? Not at all! If we are going to create a Web Shop, the order processing subsystem is definitely an important concept of the domain. However, we could either introduce our plain-vanilla, homegrown order processing subsystem or just integrate an existing system such as SAP R/3 to provide the same service. In other words, first we need to consider which services are required, then we need to decide whether we will build these services as part of our software system or just (re-)use other external services for the same purpose.
How to proceed: I am a big fan of component-based design. I just consider components to be independent artifacts (i.e., modular units of testing, deployment, implementation, modularization,...) providing services to their environment (services may be functionality or events or configuration interfaces) and consuming services from their environment (such as functionality or events). Note: components can aggregate other components. I will design all subsystems including the whole system itself using this component concept. Hence, I need to define which interfaces my system will provide to its environment and which it will need to obtain. For example, one typical required interface of a Web Shop is access to a credit card company to verify customer credibility.
When designing such context diagrams the obvious choice is to leverage UML component diagrams.

Requirements Part II - The Art of Multi-Facet Design

I'd like to continue my previous posting with an architectural design consideration. Basically, an architect maps the concrete problem (which is an entity of the problem domain) to a concrete solution (which is an entity of the solution domain).
Requirements help to decide which paths to take at specific points. This implies that requirements constrain the solution space. If multiple requirements are "applicable" at a specific point, priorities define the precedence which is why we must assign unique priorities to them. Not very complex so far, right?

But how do we really implement a solution architecture?
The first thing we obtain is a functional object model consisting of all the entities in the domain as well as their relationships. For instance, if we are going to build a Web shop, this model would probably consist of concepts such as customer, shopping cart, order system, product catalog, payment system. It is highly recommendable to define a domain model (domain-driven design) first to achieve a sound understanding and a common vocabulary among all stakeholders. Use cases are the means to drive this activity. They help designing the external boundaries of the system to be built but also the internal components required to implement the use cases. Obviously, functional requirements also have priorities which is why the use case priorities should drive the process.

Then we need to take into account infrastructural issues such as distribution, security, ... According to their priorities, we start with the most important infrastructure which needs to integrate into and with the functional model. For example, if we need to add distribution-specific infrastructure to our Web shop, we would introduce a broker-based architecture with customer browsers being the clients and the web shop being the server. Now let us assume, security is the next important requirements. We are now going to integrate an appropriate securiy infrastructure into our result we achieved after embedding the functional entities into the distributed infrastructure. We will add security components such as firewalls, identity management functionality, ... We continue with this approach until we have considered all operational requirements. What we did is extending the functional model (problem domain driven) with non-functional infrastructure parts (solution domain driven).

In the third step we add the infrastructure required by developmental qualities such as configurability, extensibility, and the like. This means that all those strategies, interceptors, configuration interfaces will be added at the end.

What we reached is an onion model with the functional architecture being the core part, the layers derived from the operational/infrastructural requirements being the middle part, and the developmental infrastructures being the outer part.
If you think about it, this is exactly what a building architect or other engineering disciplines do. Our resulting software architecture represents an integration of different functional and non-functional perspectives which is why I call it Multi-Facet Design.

But what about Software Product Line Engineering, In SPLE we will have to introduce a Commality/Variability analysis in order to determine all commanlities and variants. Variants will be considered as variations of a commonality and hence can be defined in terms of a commonality. Sounds abstract, but let me introduce an example. If we take our Web shop, we could define that we need to support different payment options (credit card, bank transfer, Paypal) depending on the customer. The variability is now defined as possible payment options. But the commonality in this example is the payment system itself. As a consequence, we need to consider the payment component in the core design, but specify the exact variations in the developmental design. For example, we could introduce the strategy pattern for the payment system to allow different options.

What about embedded systems?
All stringent constraints emerging from embedded systems such as scarce memory resources or CPU limitations or limited battery life will be handled as high priority requirements. One of the consequences is that static configurations are always preferred over dynamic configurations in such systems to allow determinism and control of QoS properties. Needless to say that realtime capabilities place additional burden to the engineer.

Requirements

Recently, I have been involved in consulting some friends about a software development project. At some point in time we started intensively discussing requirements. Soon I recognized there were some funny misunderstandings related to what requirements are. Sentences such as "Oh, all these use cases are our requirements" emerged from the void of the universe and gave me some hints there could be a problem. When I asked them for their use-cases, the only thing they could show me was a UML use case diagram. Not really what you expect as an architect. So, my first activity was explaining the theory of requirements, at least my small, probably incomplete perspective on the world of requirements.

Functional requirements define the what of the system in terms of the problem space. Thus, they should be expressed in the language of the target domain. All functional requirements span a full range of possible solutions.

Non-functional requirements come in two flavors, operational requirements and developmental requirements. They impose constraints on the full range of solutions and thus represent beasts of the solution domain.

Operational requirements are directly related to the operation of the software system. They define qualities such as security, performance, fault-tolerance and can be measured or tested.

Developmental requirements are directly related to the architecture of the software system. They define qualities such as maintainability, usability, or flexibility.  While some metrics exist for developmental requirements, they are normally not measurable.

It is interesting that most projects don't fail because of functional requirements but because of failing to meet the non-functional ones. Thus, it is often recommendable to assign sub teams or at least experts to the most important non-functional qualities in the project. They should then prepare checklists, strategies and tactics for ensuring the quality they are responsible for.

I often use the invasiveness as an additional property. A concrete instantiation of a requirement is invasive if it has an impact on the functional subsystems or relationships. Cross-cutting concerns such as security are mostly invasive.

To express functional requirements use-cases represent an excellent means. Note: Use cases are textual descriptions with sections such as actors, goals, exceptions, preconditions. A use case diagram is merely a graphical summary of use cases. For all other, i.e. non-functional requirements I recommend using utility trees such as introduced by Bass, Clements, Kazman. For example, a requirement such as flexibility could imply a lot of different things, maybe changeability, extensibility, or removability.

In the activity of architecture design, all decisions are based on requirements. Unfortunately, requirements can contradict each other. For instance, mind the availability/consistency paradox or flexibility versus performance. This means that all requirements must get a unique priority to let architects decide which path to choose at all decision points. NEVER accept requirements to have the same priority. If they have, assign priorities yourself. This also means: all architectural decisions must be solely based on requirements. Don't create design pearls, create the simplest possible architectures that work (i.e., that meet their requirements)!

Features are also requirements. So are commonalities and variants. However, they are at a very high level of abstraction and need to be mapped to finer levels first.

Requirements are the most important entities in a software development project. They help asking the right questions and making the right decisions. Requirements are a tool to constrain the problem and solution domain in such a way that an implementation can be developed effectively and efficiently. If you don't value requirements, you are doomed to fail, because you don't know what you are expected to build. Requirements are your friends, treat them as such!

About Learning & Documenting

Currently, I am pretty busy. That's the reason why I couldn't add more postings for a while. This posting will also be a short one.

It might seem somehow unrelated to architecture, but nonetheless I find it valuable. I am currently very busy preparing some presentations as well as an article on Software Architecture Refactoring for the german OBJEKTspektrum magazine.

I made some observations about my work style in the last months that I'd like to share with you.

For learning new IT-related things I am often documenting what I learned using Powerpoint presentations. Then, I will constantly rearrange the slides in such a way that they reflect how I consider the material should be organized to help educating others in a better, more effective way. Powerpoint also serves as a tool for organizing my thoughts and works better for me than mind mapping tools for this purpose.

When I am writing an article, I often prepare the material as a presentation and then use the presentation as a story line for the article. This helped me writing scientific papers as well as other professional articles. The presentation contains much less content by nature but forces me to design the rough storyline, while I am sometimes lost in details when I start writing articles.

I often also use pattern templates to organize my efforts when I need to solve architectural problems. Pattern templates are very powerful, because they require me thinking in terms such as context, problem, forces, solution, consequences and so forth.

In summary, I guess such tools really help me being better organized. Any similar experiences by someone else?

I am really wondering what tools and methods others recommend.