No future for architects?

I am a big fan of Star Trek, especially of The Original Series and The Next Generation. From a technology viewpoint, it is amazing how all of these technologies such as warp drives or beaming have influenced science. Did you know that the inventor of the mobile phone had also Star Trek in mind?

However, there is one sad aspect. What about Software Engineering? Whenever software plays a role in any of the episodes, you’ll recognize Spock or Data just coding. Do they ever design? There is also no software engineering team in the Enterprise although software should be an important asset in the system architecture. I am missing a chief engineer for software engineering aspects!

Does the computer program itself? Will UML disappear in the future? Or is it just software engineering being too boring for SciFi fans? Interestingly, there are also seem to be no restrooms in the spaceships. This is what they have in common with software engineers :-)

Thus, we could ask how software engineering especially the discipline of software architects will evolve in the future? Will we still use a successor of UML in hundred years? How could the role of software architects evolve?

The best approach to predict the future is identifying space of improvement. It is very likely that such areas will be addressed. How should the ideal process of software design look like? What is the software architect supposed to know and to do?

In most SciFi movies automatization/Model-Driven Software Development and AI are often the answer. Architects express their intent, smart systems then try clarify open issues, and eventually the result is generated.

Another approach could be an organization like the Borgs who strongly follow an agile approach with code ownership, collective programming and test-first.

Any opinions?

Keep in mind: Resistance is futile!

RRC Card

Most of you already know the CRC (depicting Class, Responsibilities, Collaborators) which is a low tech technique for discussing software artifacts in brainstorming sessions for software design.

I propose another kind of card for depicting the roles in a particular project: the RRC Card: Role Responsibilities Collaborators.

The idea is simple:

• The first R stands for role by which I mean a concrete person and her/his role
• The second R stands for Responsibilities: what responsibilities should this role have to meet its expectations
• The C stands for Collaborators: from which other roles does this role depend, e.g., with which other roles must it interact

If you start with a project, you could ask the parties involved to jointly fill out these RRC cards, thus making clear how responsibilities are partitioned and how roles are supposed to interact.

Example:

---------------------------------------------------

R: Michael / Architecture Consultant

---------------------------------------------------

Responsibilities                        Collaborators

Mentor and coach                    Lead Architect: Paul

Help design                             Architect: Tom

Implement

----------------------------------------------------

 

If you already in a project in deep trouble you could ask persons to fill out how they live their own role and likewise fill out RRC cards for other persons, thus identifying conflicts between expectation and reality.

Document these RRC cards in a document (e.g., paper-based or Wiki-based)

Risk-driven

In addition to requirements and business expectations, risks represent an important driver of architecture design activities. Not addressing risks appropriately is a major reason for project failure. But what exactly is a risk and how can I identify potential risks as an architect.

On a very coarse grained level, risk might be considered as not knowing the outcome of an event or activity. It is the threat or propability that “an action or event will adversely or beneficially affect an organisation's ability to achieve its objectives” [Wikipedia].

There are as many different types of risks in a  software development project as there are influential factors.

For example:

• Technical risk: Is EJB the right technology for achieving 100000 transactions per hour in our Web store?
• Organizational risk: Is the organization appropriate for this kind of development project?
• Political risk: Does this product manager follow her/his hidden agenda? Is this supplier really trying to support us or is he in fact competing with us?
• Design Risk: Is this design decision appropriate for dealing with the requirements?
• Requirements risk: Do we really understand the requirements of our organization or the customers?
• Process risk: Is this the right development process?
• …

For dealing with risks, there are lot of different ways. Technical risks and architecture risks can be reduced by a requirements-driven, test-driven approach with explicit assessment activities and refactoring activities. Political risks require the architect to communicate and lead efficiently and effectively.

An important point is this context: to be able to deal with risks, we must make them explicit. Implicit risks are often overseen, forgotten or ignored. You should really know your “enemy”.

Making risks explicit means to document AND communicate them.

I recommend to introduce a template for risk assessment comprising:

• Name of the risk
• Priority/Relevance of the risk
• Type of risk: organizational, technical, ….
• Rationale: explain why this is a risk
• Context: what else should we know when dealing with this risk
• Strategies: possible ways to deal with the risk
• Decision: the recommended strategy to deal with the risk for the concrete context, and the rationale for this decision
• See also: other risks this risk implies or similar risks that could be treated with the same strategy

Not all of these fields need to be available in the beginning. You could start with name, type, context, thus collecting the risks first. Then, you could prioritize the risks, and eventually extend the templates for this risks identified as high risks with missing information.

Let me give you an example:

• Name: Uncertainty Using EJB
• Priority: high (******)
• Type: technical
• Rationale: capability to meet the performance requirements of 100000 transactions/hour. We are not sure whether EJB will be appropriate
• Context: Our developers are not experienced with EJB
• Strategies:1) Build performance prototype first. 2) Hire external expert. 3) Use Enterprise OSGi instead. 4) Outsourcing.
• Decision: Hire external consultants. Let them build a performance prototype and then help implementing the system when EJB proves to be a reasonable choice.
• See also: we are not sure whether MySQL can cope with the throughput requirements.

It is worth mentioning that risk assessment is not exclusively provided by software architects. In fact, all roles in the project must help and closely cooperate for risk assessment. This way, everyone involved is aware of potential risks. And of course, some risks such as those implied by organization, test quality, or quality of requirements must be identified and clarified by other roles such as product and project management, requirements engineering or test and quality management.

Architecture and design decisions should then take risks into account.

Mind the risks: The highest risk is not systematically dealing with risks. 

Who is responsible

Suppose, you are working in a software development project as a software architect. You are still waiting for the requirements to be available in sufficient quality and quantity and priorization. Even after months almost nothing happens. Sure, the problem is caused by requirements engineering. But what is your responsibility as an architect? Your responsibility is to communicate and escalate which requires courage. For example, you could tell the stakeholders what implications the current situation might have - for example, you cannot start with architecture design which means development cannot start with implementation etc. You should never fall into the "this is not my job" trap. After all, software architects need to drive the project. Sometimes, it might be even possible to take responsibility for requirements as an architect but this should be the exception not the rule.
Similar issue for test & quality: if test coverage is only limited, bugs are seldomly found during testing, or tests themselves are faulty, then you could finger point to testers and developers. However, it is your job as an architect to deal with quality. Don't try to escape your responsibility in this area. Test First Design makes it very clear that testing requires the different roles to cooperate.
What about business aspects? I know, you got this product manager who is in charge of all business aspects. But software architects are responsible in helping develop the technology roadmap, identifying patents, estimating the business implication of architecture decisions. Even more, architects must base every architectural decision on requirements and business needs.
Hmmh, what about project management? Of course, an architect should not be forced to also act as a project manager. These two rules are very difficult to live at the same time in the same project. But what if project management needs to estimate costs and resources? Here the software architect needs to provide support. What skills do developers need? How many developers and how much time are necessary to develop subsystem A.
I don't claim that architects shuld be jack of all trades but masters of none. Their main focus should be on software architecture. However, they cannot ignore what is happening on the boundaries between software architecture and the rest of the project. They should feel responsible for project success.

Domain Models (continued)

Throughout my life as a software engineer I have participated in many different projects in various domains. Needless to say, I encountered some deja vu experiences in those projects, also known as war stories. One remarkable example always has been the problem domain itself. Even in projects with lots of experienced participants the problem domain had been often root of misunderstandings and misconceptions.  In large projects with outsourcing partners, external partners and other suppliers common knowledge of the problem domain is essential. A project  with lack of problem domain knowledge among the stakeholders is doomed to fail. Remember the tower of babel where lack of communication and insufficient common understanding lead to lethal catastrophe.

As I am mostly acting as software architecture mentor and coach or technology expert I am often master of the solution domain, but not of the problem domain. For me it is inevitable to obtain a detailed knowledge of the problem domain. This is even more important when developing product lines or platforms, because in that case lack of common understanding influences more than one product or solution.

How could we address that problem? My recommendation is to introduce a domain model. Of course, I am referring to the problem domain in this context.  A domain model is not just a glossary of terms – a common misconception I am often facing.

A domain model consists of all core concepts (actors, subsystems, components, services, …) relevant within that domain as well as the relationships and interactions between these core concepts.

 

A large domain typically consists of subdomains. For example, a telecommunications domain for Unified Communication might introduce subdomains such as basic communication or presence. As a consequence, the domain should be partitioned hierarchically into subdomains.  

There are different ways to descible a domain model, from informal approaches to strictly formal definitions such as a domain-specific language. In all cases, stakeholders might select textual or graphical representations, whatever is more appropriate.

Note that I have not assigned the responsibility of introducing a domain model to software engineers. Defining a common domain model should rather be a joint endeavor of all stakeholders, maybe driven by requirements engineers and architects.

If you have ever read definitions of software architecture like the one by ANSI/ISO, you might recognize that architecture definitions match closely with my definition of domain models. First of all, those architecture definitions are insufficient (due to their genericity). Secondly, a domain model is a good starting point for creating the architecture. Take the use cases (black box view) and specify their dynamics using the domain model entities (white box views). Supplement this architecture core with design tactics for non-functional requirements. This will lead to the architecture baseline. Using the domain model also helps you staying focused on the problem domain instead of diving to the solution concepts too early.

Eventually, a domain model defined, understood and agreed by the various stakeholders such as requirements engineers, developers, testers, architects, product managers, customers serves as a fundamental mean for architecture design and communication. 

Of course, the solution domain itself can also be expressed by a domain model. This is why the main task of an architect consists of mapping a problem domain to a solution domain.

Formalizing both to a sufficient extent helps automating this process using model-based software development approaches. Such automation, however, might be too expensive for one-off applications or smaller product lines.

Whenever you are participating in a development process in the presence or future: do not forget to specify the domain model if not already available.

Podcast on Software Architecture

For those of you capable of understanding German, I’d like to point you to our new Podcast on Software Architecture called Software ArchitekTOUR. You may also search in the ITunes Store for the podcast and find it – for example, search for ARCHITEKTOUR in the podcast category.

The Podcasters are

• Markus Völter (freelancer, associated with ITEMIS)
• Stevan Tilkov (innoQ)
• Christian Weyer (thinktecture)
• Michael Stal (Siemens)

To stay in touch with reality we have split up the podcast in three subcategories:

• General Architecture Topics (sponsored by IBM)
• Java and Software Architecture (sponsored by innQ, itemis, thinktecture and Siemens)
• Microsoft Technologies and Architecture (sponsored by Microsoft)

There are already 4 episodes available. Episode 0 is more intended to introduce ourselves. Episodes 1-3 cover patterns in general, as well as patterns in Java respectively .NET.

 

Listen, enjoy, and give us feedback

Simplicity rules

Many architectures are overwhelming. Not that they shine due to their excellent quality. Rather they appear as a big ball of mud. I already mentioned what I am calling design erosion in the context of architecture refactoring.

To keep an architecture small, expressive and simple we can do a couple of things as software architects.

For example:

• Each and every design decision must be based on an architecturally-relevant requirement. There is no decision without documented reason. Of course, this requires architects to feel responsible for the quality of these architecture requirements, but not for requirements engineering, of course! It is recommended practice to keep track on where in the architecture which requirements are “located”. Requirements traceability is an essential value, especially when assessing, refactoring or evolving a system. I am assuming here that architects already checked for the feasibility and essence of the requirements from a business perspective. Availability of 99.99999% is not reasonable in a project with a budget of 100k bucks. 
• Introduce symmetry and conceptual integrity. This includes the prescription of specific patterns, the introduction of design and coding guidelines, as well as the introduction of guidelines for (at least the most important) non-functional qualities. This prevents that the same problem is solved in various ways within the same solution space and it makes sure, we are applying best practices from experts instead of reinventing the wheel. War story: A system where designers have introduced dozens of ways for fault handling will be inexpressive and complex.
• Don’t consider technologies as toys. Many projects play with technologies. They base their decisions on a technology-first approach. This is human and we as software engineers like to experiment with the new hip operating system or SOA middleware. But in practice, we should first create an appropriate architecture meeting the requirements (see first bullet point) before we should start introducing new technologies. The business and the requirements drive the architecture, not technology.
• Test technology. The previous point does not mean we should not care about technology – only that we first address the requirements. Every technology must be checked for its appropriateness. If your task is to build a high performant messaging backbone, then you better check whether EJB is really the proper platform to use. Don’t trust any vendor’s promises here! They do not know your problem context and system environment. Building throw-away prototypes is the best way to check technologies.
• Apply a test-driven development approach: this is to introduce a kind of safety net. Emphasize testability in your architecture design, e.g., by proper modularization, interfaces, and service contracts. Test-driven in this sense also includes regular design and architecture assessment where you try to identify architecture smells.
• Regular architecture refactoring helps address architecture issues early. When a dependency cycle has just been introduced unintentionally, it is not a big issue to detect this kind of design erosion early with architecture reviews or architecture analysis tools. You are then able to get rid of the problem early by architecture refactoring. The more you wait, the more architecture entities will make it difficult to address the problem without heavy impact. Thus, do regular architecture reviews and refactorings.
• Strategic before tactical design. Don’t try to introduce over-generic solutions with myriads of configuration options. First address all strategic requirements (functional requirements and operational requirements) before dealing with tactical issues such as modifiability. Use the Open/Close principle to open your architecture for variation.
• Introduce priorities for all requirements/features. The priorities help decide which requirement to address first in architecture design. If security is more important than performance, you’ll end up in a completely different solution architecture, then wheny performance has the higher priority.
• Follow an approach of piecemeal growth. In all but the most trivial applications a big design upfront approach will be doomed to fail. Thus, assign requirements depending on their priorities to iterations and build your system incrementally.

 

These are only a handful of recommendations. Certainly, there are even more. Any tipps from your side?

Get off of my Cloud

It enters the stage as one of the great buzzwords of the last months and if we had not to deal with an economic crisis these days it would gain much more attention. Cloud Computing and its relatives such as SaaS are dominating the IT world. Ever heard about Windows Azure, Google Web Apps, Amazon E2C and S3? The idea behind this paradigm is rather old: why not provide resources transparently as services  in a hosting infrastructure? This would let IT departments and CIOs refrain from being infrastructure providers. Instead they could focus on their actual job: providing IT functionality to drive their company's business. 

By services I don't just mean applications, but all kinds of services.  An application such as a CRM app could be a service. Likewise, services could comprise a  piece of middleware such as a persistence solution or a workflow management system. And in the most extreme case, even the infrastructure itself, for example, operating systems, storage or network solutions might be deployed as services. Services where ever you look.

The idea behind cloud computing is a very promising one. Just take trusted infrastructure providers, let them host the applications (services) we need within an Internet-based cloud, and access these services from an inexpensive device. No need to buy or maintain your own data center solutions anymore. A data center on everyone's desktop if you like. Whenever you need more resources, just take them from the cloud and only pay for what you use. "Elastic computing" made real. Sounds like a good idea in an economic crisis, doesn't it? The strength and weakness of Cloud computing today is its dependence on external providers. Would you run your mission-critical applications with sensitive data on a 3rd party infrastructure that is not under your control and which supports multi-tenancy? What about services running in a remote country that you do not trust? How can a provider guarantee quality-of-service in an Internet-based infrastructure? Remember all the recent downtimes of Google Mail. What if all your company mail is running on such a network?

These problems imply a strategy where mission critical applications only run in a private cloud, while only a fraction of less critical applications may leverage public clouds. As soon as the Cloud Computing technology evolves and matures, more and more services might be moved to public clouds.  Then, private clouds could also be extended using public clouds. By the way, this extension approach is only feasible if standards foster interoperability of cloud solutions.

From 10000 feet Cloud Computing denotes sort of a business solution that introduces pay-per-use models for all types of services. Regarding technology this solution is based upon a bunch of existing paradigms such as virtualization, SOA, Multicore CPUs, NAS, or Web 2.0.

What all those people enthusiastic about Cloud computing tend to forget, is that there is no free lunch. It, indeed, represents a promising technology with a high coolness factor. However, from a software architecture perspective, it is rarely sufficient to just deploy application silos in virtual machines that are hosted somewhere else in the network. To really leverage clouds your applications need to be aware of the cloud infrastructure. You might split islands of functionality into interconnected services, and integrate application services with other platform or infrastructure services. What about issues such as collocation of services and data? In addition, engineers need to create desktop-like GUIs that communicate with backend services as promoted by Web 2.0. Last but not least, quality of service such as availability or responsiveness does not only depend on infrastructure and platform but is heavily influenced by the solution architectures.  The best infrastructure or platform does not help if the software architecture is a bunch of crap.

Cloud Computing offers fascinating new possibilities. But we shouldn't suffer from the hammer-and-nail syndrome. When applied inefficiently Cloud computing will inevitably lead to problems and disappointments. If your management or CIO department is all of a sudden promoting Cloud computing, this might be a good thing, but it also requires you as a software engineer to deal with expectation management. And finally, mind the architecture!

My Domain is my Castle

One of the most important (first) steps in architecture modeling encompasses the description of the domain model. This model introduces all entities relevant within the current domain as well as their relationships and interactions. It represents the main language all stakeholders should understand. All further activities in architecture modeling basically take the domain model and enrich it with additional infrastructure entities. 

Sounds very abstract, right? Let me give you a concrete example. Suppose, we are going to develop a Web store. What are the typical objects that appear in the problem space and are well known to all stakeholders?

For example, I'd expect entities such as:

• web store user: someone accessing the web store
• customer: someone interested to buy items
• shopping cart: used to add, remove, pay goods
• product catalog: presents all available products classified by categories and allows to search for these products using different types of query
• customer database: place where all customer information is stored
• order processing: responsible to process orders

If you think about this example further, you'll recognize there are some typical relationships between domain objects. For example: a web store user could be a customer or an administrator. A customer typically owns at most one shopping cart at the same time.

You also can easily defer some interactions between the entities. It is obvious there must be a relationship between the shopping cart and the product catalog, because the information about purchased items is read and updated from the product catalog. Of course, the shopping cart needs to interact with the order processing system after the customer has pressed the order button.

With other words: when thinking about how use cases would be mapped to sequence or interaction diagrams, the knowledge about the domain model is essential. It is the step taking you from a black box perspective to a gray box perspective.

There are different ways to express such a domain model. You could invent a graphical representation or use a textual language instead. If a domain object model gets formalized, we call this a DSL (Domain-Specific Language). In this case, engineers could even provide generators that map from the problem domain to the solution domain, i.e., that map problem domain objects and relations to solution domain objects and relations.

Using domain models offers huge advantages:

• it introduces a common vocabulary among all stakeholders which supports effective discussions and often prevents a steep learning curve for domain dummies
• it reduces the chance of missing to address important parts of a domain
• it eases architecture modeling significantly
• it helps focusing on the problem domain instead of always diving into the solution domain
• when enriched with domain patterns (analysis), it boosts productivity

It is neither necessary nor useful trying to come up with a complete domain model in the first place. In most cases, there will be an initial, maybe incomplete, domain model which engineers together with the other roles will evolve over time.

There are several examples, where the domain model is already available. Think about GUIs, compilers, and some kinds of healthcare domains. In these domains, it is useless and a complete waste of time to come up with your own domain model.

 

So, whenever you are being involved in a new software architecture, always mind the domain object model! It will be your best friend.

Software and System Architecture

Currently, I am involved in a large-scale project where Siemens Healthcare is building a couple of very innovative and exciting particle therapy systems. These systems are rather challenging as they consist of complex software, hardware  - for example, a circular particle accelerator - and all the other constituents such as the building, the CT devices. It is a typical project where software while being a major element is only contributing a small part of the system. However, within our company which yields revenues of over 70 billion euros a year, software engineering has become a major business factor. According to an estimation 60-70% of these revenues already depend on software.  Now imagine, 10% or even more of our projects would fail - what a nightmare! There are two implications: first of all, lead software architects must be well educated in software architecture and technology and show the right leadership skills. Secondly, system and software architecture must be treated as two sides of the same coin. So, what are the responsibilities  of software architects and system architects and how should they cooperate? I tried to find an appropriate definition in Wikipedia and failed. Especially, system architecture seems to be not well understood or ambiguous.

From my viewpoint an effective partitioning in software and system architecture depends on the characteristics of the project itself. If software constitutes the main part of the product or solution, then the lead software architect could and should also be system architect. Obviously, an additional IT architect  makes a lot of sense if the underlying infrastructure is sufficiently complex, but that's a different story.

In projects where hardware and software are of the same relevance or where hardware dominates, for example in most embedded systems, a system architect should be in the lead and co-operate with a hardware and a software architect. The system architect then should care a lot about testing. I have been involved in projects where I was in charge of the software system but got only rarely access to the target system. Thus, we had to use the development system where everything worked stunningly fine, but got deadlocks when finally moving to the target system - the OS of the target system had a faulty implementation of  the Sockets libraries. This implies, that the system architect should allow software and hardware development to happen almost independently, but introduce a lot of sync points where software/hardware integration is checked thoroughly. By the way, a similar problem occurs when partitioning a software system into modules which are then designed and implemented by different teams - think of outsourcing in the extreme case. If integration and system testing is neglected, you will encounter a lot of bad experiences. It is much more complex and challenging to check hardware/software integration than the integration of software modules - at least in theory :-;  One of the bad experiences you might need to cope with is that hardware requirements are often considered almost fixed, while software requirements are considered "soft". Would you ever change the main specifications of an airplane shortly before it is delivered? Definitely not! But this does not hold for software systems.  One of the consequences of this fact might be that the hardware team has already finished their part, being reluctant to change anything. As the software requirements have changed in the project, software development has fallen behind schedule. In these situations, software architects sometimes become the scapegoats of system development. A good system architect should be capable of handling these situations effectively and efficiently by better synchronizing the different teams.

A system architect should also make sure that the overall system requirements are mapped consistently and completely to software architecture and hardware related requirements. He or she should care about requirements traceability, especially for those requirements that have to be implemented in software and hardware as well.  If hardware comprised the core part of the system, software architects should have a sound understanding of the overall system, not just of the "soft" parts.

Basically, you could consider the partitioning of software and system architecture responsibilities in almost the same way like splitting responsibilities between software architects and the lead architect. The system architect is in charge of the whole system, while the software architect is in charge of a part of the system.

There is a lot more to say about this topic. But I am curious what you think and I am wondering about your experiences with software and software architecture as well as with the respective roles. Any comments highly appreciated.

News from the Engineering Frontier

I have been to the Canary Islands for a month. Since then I have been totally busy in two projects. One of them is dealing with medical particle therapy solutions, while the other one consists of setting up a certification program for software architects within Siemens. I am in charge of this program. Last week I was giving a seminar on software architecture with 12 guys from Siemens. I had a really great time and hope the same holds for the participants.

It is time to post more on this blog. Yes, I feel guilty for being so lazy. But you know, I am in love with Work-Life-Balance. To be honest, I needed a timeout from all software engineering business for a while, instead focusing on sports activities and digital photography. Now, I am really motivated to enter the software architecture bandwagon again. Last monday, we (= Markus Voelter,Stefan Tilkov, Christian Weyer, and myself) started with a new podcast series on software architecture. This will be in german with some company sponsors. I will keep you informed about the details when things mature. I am wondering whether a software architecture podcast in english would make sense.

In the medical particle therapy project we have just defined a possible partitioning between system architecture and software architecture. Seems to be a major source of misunderstandings in many projects. Same for differentiating problem and solution domain. Engineers seem to be addicted to the solution domain drug. Instead of thinking hard about the problem domain we always try entering safe ground, in particular by addressing solutions very early. The problem is: how can we define a solution when we don't understand the problem? On the other side, bubbles don't crash, as we also know. We could endlessly define UML diagrams without ever dealing with implementation. As Bertrand once said: all you need is code. Thus, we need to ensure the feasability of our solution very early. How can we satisfy both goals? Needless to say, agile development is the right answer.

Of course, there are a lot of additional issues when addressing the boundary between system and software architecture. This will be subject to upcoming posts, All feedback and comments as always highly appreciated

Views on a Cat

Suppose, you're going to design a new software system for a specific domain. One of the first steps is to introduce a domain object model which comprises the core entities in the domain as well as their relationships. That's easy, you might say. If it is that simple, why do so many  projects fail in establishing the right domain object model? And, of course, this challenge inevitably appears in all kinds of modeling activities.

But now for something completely different - as Monty Python would say. I got two nice cats at home. In order to model a cat, there are different possibilities:

• (ATOM) We could consider a cat as a mere collection of molecules.
• (GENE) Obviously, a cat can be uniquely identified using its DNA.
• (BODY) Another approach is modeling a cat as an aggregation of subsystems such as legs, joints, muscles, intestines.

Does it make sense to apply the first model (ATOM) in order to understand how a cat moves? No, because the detail level is overwhelming.

Is it useful to use the third model for understanding all chemical activities within a cat? Definitely not!

With other words, we can model the same physical concepts in different ways, each of these views strongly depending on the purpose of the model. Don't let yourself be confused. It is not as simple as having only one model for one purpose. For example, to understand the physical appearance of a cat, you could apply model ATOM as well as model BODY.

What are the implications of this observation?

• First of all, you might require different models (views) in order to define and understand a software system.
• Secondly, each model should be motivated by a concrete and useful purpose.
• Thirdly, all participants should agree on syntax and semantics of the model.
• And last but not least, all models should be documented.

Mind the word "useful" in the second bullet point. As organization drives architecture, the organization often directly maps to entities in the model. If your organization is badly structured, your architecture will reveal the same problem. Think about this in the context of your own organization.

As another consequence I'd like to emphasize that it is not sufficient just inventing a couple of sophisticated models. The set of models should also be simple, complete and consistent. All of the models should complement each other in a meaningful and appropriate way. That's the reason why we got view sets such as the UML 4+1 view.

Unfortunately, I often sit in project meetings where people strongly believe they are all sharing the same view of their domain, while in fact they are not. Endless discussions might be an indicator for this problem. Thus, it is essential to explicitly come up with a common domain model that is agreed among all participants. Basically, the design of such a model should be one of the early steps  in a development project (directly after scoping but before architecture design). Needless to say that software engineers are surrounded by models: they have specific views of the problem domain and the solution domain, use tools such as compilers or database systems that themselves are built on top of models.

If a software developer's life is about models and model transformations (e.g., the mapping from the problem domain to the solution domain) we should consider models as first class citizens and invest sufficient time for choosing the right viewpoints and establishing the right models.

Interestingly, even our view of the physical world such as the way we understand a cat is determined by models. But this is a different story.

Archidictatorship versus Develepocracy

In the software development projects where I am involved as an architect I  can typically observe different types of software architects. Let me illustrate two extremes:

• Some architects try hard not to decide anything or at least not too early. These architects typically integrate all other project participants in the decision process.  On one hand, this can be a very successful approach. If everyone agrees to important decisions, everyone remains motivated. On the other hand, the same habitude leads to the risk of decisions being postponed endlessly. Eventually, architects constantly drawing new UML diagrams or deep diving into technical details are a problem not a solution.  In most cases, wrong decisions are better than no decisions, because wrong decisions can be detected very early and all problems resolved. Think of refactoring as a tool!
• Other architects prefer a kind of tyranny. They got a lot of self esteem, believe to know everything, adore the Borgs for their "resistance is futile" strategy. This style of leadership can come very handy in critical situations where project success depends on strong leadership, for example when you require fast decisions. The downside of this dictatorship is that architects behaving this way may not get buy-in from other participants, thus leading to a high level of demotivation. And obviously, too much power in the hands of the wrong persons can lead to projects being doomed to fail miserably. A negative implication might be that those dictators want to get involved in every decision, even in unimportant ones.  As no one is able to understand all details in the case the project is large enough, you'll inevitably will experience lots of wrong decisions.

You will argue that these leadership styles are not specific to software architects, and, of course, you are absolutely right!

What I'd like to emphasize in this posting is the importance of leadership skills for software architects. Of course, there are situations where an architect should be dictator, and other situations where democracy might be the more appropriate choice.

Unfortunately, or should I say fortunately, all of us got different personalities. Your leadership style should fit with your personality, because otherwise everyone will recognize the discrepancy.

I won't cover leadership styles in too much detail because there are persons much more knowledgeable than I in this context. Just read this one or that one. A Web search will even yield more material.

What I consider really helpful is a kind of self analysis. For instance:

1. Personality: analyze your own personality, try to find your strengths and weaknesses. Ask other people how they assess you as an architect. How do you evaluate yourself acting as an architect? A very helpful tool might be to analyze previous projects: where did you succeed and why and where did you fail.
2. Leadership Styles: read about all possible leadership styles. Nothing more to add here.
3. Context: Think in your project context what leadership style should be applied in which situation. Of course, you can't act as a dictator in front of your head of R&D. Likewise, always striving for full agreement by every developer can result in endless recursion.

For a software architect, the time spent for communication might be as large as 50%. Communication means interacting with other people, informing them, guiding and mentoring them, or getting their buy-in. Without the right social skills and leadership style, a software architect is doomed to fail.

Panic mode

If you carefully observe the current economic crisis, you'll recognize that some market participants are acting in panic mode. This comes to no surprise if you ever have read books like those by Jose Ortega y Gasset. Unfortunately, panic mode is never helpful because it leads to spontaneous activities that are not motivated by thorough and systematic planning. Instead, people tend to think they have to react quickly and often draw the wrong conclusions which worsens the situation instead of improving it. If you look at some current stock values such as Intel or Siemens the situation is somehow out of control. With other words: it could be a could idea to buy stocks now. Maybe, Douglas Adams was wright when he once mentioned that nothing can travel faster than light with one exception: bad news!

If you look at software development projects, you'll often find the same issues. As soon as deadlines are getting delayed or dropped, participants are increasingly starting to get nervous. After a while the whole organization is typically entering panic mode. There is a possible alternative flow which I call Lemmings mode. This is when people refuse to recognize they are facing some severe problems and continue the project without any countermeasures which denotes a typical human behavior.
As an alien you'll recognize panic mode in an existing project by encountering different indicators. For example, if meeting frequency is high, and architects and developers spend more time for meetings than for productive work, this might either be a clue for developer's hell or for a project in panic mode. The same holds for situations where the organization is constantly generating new or enforcing uncontrolled and unplanned activities (actionism) in a higher rate than they can be processed. If you interview people in such a project context, you'll receive inconsistent and contradicting statements, because there is no synchronization or coordination in place.

Although it is obvious that panic mode worsens the situation and even leads to conflicts among the persons involved, projects fall into this trap pretty often. I believe, this is caused by the psychological fact that all participants in such a project are caught in the trap. In these situations, I generally recommend to involve an external person who is not part of the problem (aka as project) and acts more as an external observer and mediator. In addition, this mediator needs a peer in the organization's senior management able to enforce all recommendations (if accepted), because otherwise all suggestions would be ignored. To be honest, architecture reviews or other forms of assessments work exactly that way.

A way to address problems very early in projects is to use an agile approach, especially to elaborate what risks exist and how to address them systematically. In contrast to common believe such risks are non constrained to architecture and technology but also extend to politics, partner and vendor relationships, organization, quality, development processes. Risks analysis is not just about recognizing the risks but also about strategies and tactics for addressing them.

By the way, this is exactly what currently happens in the economic crisis. Since market participants don't seem to be able to address the challenges themselves, politicians and central banks are forced to bring countermeasures in place. All the risks in the market have been subject to constant ignorance.

The later risks are detected, the more likely the project will enter panic mode. This is why I always recommend regular flash reviews of architecture, code, test strategy, development process (for example, at the end of an iteration). They represent means for detecting problems very early and for getting rid of them (for example, by refactoring).

To cite Douglas Adams: DON'T PANIC!

Why the waterfall model does not work

Again and again I have met managers and software engineers who tried to convince me that agile processes while being appropriate for somebody else's project were simply not applicable to their problem contexts for several reasons. They came up with rather lame reasons like being used to waterfall models or the claim that agile processes do only work for small teams. Especially I am enjoying people's reaction when I am talking about really huge projects I had been involved and which followed the Scrum method.

From my viewpoint the question shouldn't be whether to use agile development processes instead of a waterfall approach. I would rather suggest to let all those waterfall addicted prove why they consider waterfall approaches more suitable in terms of a concrete project. 

It is true that an agile method is not superior in planning projects. Even in an agile context you will need to estimate resources, budgets, or time-frames in advance.  But in contrast to a waterfall approach you get an early feedback if your initial planing proves to be plain wrong: If after the third iteration you got an substantial delay you are able to react with appropriate countermeasures. In a waterfall model the old saying is that while 80% of the project have been completed, you still need to address the remaining 80%. You may be doomed to fail at the beginning of the project but a waterfall model lets you live with this illusion till the sad end. An agile method on the other hand just provides a whole range of safety nets. You may fall but contact with ground usually is much smoother.

Let me take it the other way round. What are the preconditions to make a waterfall model succeed?

• All requirements are available at project start and will remain unchanged.
• All of the requirements are consistent and complete.
• No errors will be introduced in any phase of software development.
• Likewise, all tools, legacy components, infrastructure parts fully adhere to their specifications and, of course, are consistent and complete themselves.
• All stakeholders start with a full understanding of the problem domain. Developers, testers, architects also have a full understanding of the solution domain.
• There will be no misunderstandings in any communication between stakeholders.

Obviously, most of these preconditions never hold in any real-life project. We have to cope with error and failure. In agile approaches, the principle of piecemeal growth helps to master complexity. Safety nets such as test-driven development or reviews enable quick detection of potential problems. Means like refactoring support us in addressing the problems we detect and to embrace change.  Agile communication and customer participation  help in keeping all stakeholders synchronized.

Two of the main principles of agile engineering might be called "embrace change" and "learning from failure".

And if we look at nature, evolution basically also represents an agile approach. For all creationists: evolution does not necessarily rule out that there could be a master plan behind everything.

 

What's your take on the agile versus waterfall dispute?

Integration

Integration represents one of the difficult issues when creating a software architecture. You need to integrate your software in an existing environment and you typically also integrate external components into your software system. Last but not least you often even integrate different home-built parts to form a consistent whole. Thus, integration basically means to plug two or more pieces together where the pieces were built independently and the activity of plugging requires some previous efforts to make the pieces fit together. The provided interface of component A must conform to the required interface of component B.  If these components run remotely, interoperability and integration are close neighbors.

For complex or large systems, especially those distributed across different network nodes, integration becomes one of the core issues.

Integration necessities of external components or environments should be treated as high-priority requirements, while internal integration often is caused by the top-down architecture creation process. Examples for external component integration comprise using a specific UI control, running on a specific operating system, requiring a concrete database, or prescribing the usage of a specific SOA service. Internal integration comes into play when an architect partitions the software systems into different subsystems that eventually need to be integrated with each other.

Unfortunately, integration is a multi-dimensional problem. It is necessary to integrate services, but also to integrate all the heterogeneous document and data formats. It is required to integrate vertically such as connecting the application layer with the enterprise layer and the enterprise layer with the EIS layer. Likewise, we need to integrate horizontally such as connecting different application silos using middleware stacks such as SOAP.  In addition, shallow integration is not always the best solution.  What is shallow integration in this context? Suppose, you have developed a graph structure such as a directory tree. Should you rather create one fat integration interface with complex navigation functionality or better provide many simple integration interfaces for all nodes in the tree?

What about UI integration where you need to embed a control into a layout?

What about process integration where you integrate different workflows such as a clinical workflow that combines HIS functionality (Hospital Information System) with RIS functionality (Radiology Information System)? Or workflows that combine machines with humans? 

What if you have built a management infrastructure for a large scale system and now try to integrate an addition component that wasn't built with this management infrastructure in mind?

Think about data integration where multiple components or applications need to access the same data such as accessing a common RDBMS. How should you structure the data to meet their requirements? And how should you combine data from different sources to glue data pieces to a complete transfer object?

Semantic integration is another dimension where semantic information is used to drive the integration such as automatically generating adapter interfaces for matching the service consumer with the service provider.

And finally, what about all these NFRs (non-functional requirements)? Suppose, you have built a totally secure system and now need to connect with an external component?

It is needless to say that integration is not as simple as it often appears in the beginning. Even worse, integration is often not addressed with the necessary intensity early in the project. Late integration might require refactoring activities and sometimes even complete reengineering if there are no built-in means for supporting "deferred" integration - think of  plug-in-architectures as a good example for this.

Architects should explicitly address integration issues from day 1. As already mentioned, I recommend to place integration issues as high-priority requirements into the backlog. Use cases and sequence diagrams help determining functional integration (the interfaces) as well as document/data transformation necessities (the parameters within these interfaces). During the later cycles in architecture development, operational issues can also be targeted with respect to the integrated parts. Last but not least, architects and developers need to decide which integration technologies (SOA, EAI, middleware, DBMS, ...) are appropriate for their problem context. Integration is an aspect crosscutting through all your system!

Agile processes consider integration as a first-class citizen. In a paradigm that promotes piecemeal growth, continuous integration becomes the natural metaphor. From my perspective, all software developments with high integration efforts must inevitably fail when not leveraging an agile approach.

To integrate is agile!

Trends in Software Engineering

It has been a pretty long time since my last posting. Unfortunately, I was very busy this summer due to some projects I am involved in. In addition, I enjoyed some of my hobbies such as biking, running and digital photography. For my work-life balance it is essential to have some periods in the year where I am totally disconnected from IT stuff. But now it is time for some new adventures in software architecture.

Next week I am invited to give a talk on new trends in software engineering. While a few people think, there is not more to discover and invent in software engineering, the truth is that we are still representatives of a somehow immature discipline. Why else are so many software projects causing trouble? I won't compare our discipline with other ones such as building construction  because I don't like comparing apples with oranges. For example, it is very unlikely that a few days before your new house is built, you're asking the engineer to move a room from the ground floor to the first floor. Unfortunately, those things actually happen in software engineering on a regular basis.

A few years ago I wrote an article for a wide spread  german IT magazine on exactly the same issue. I built my thoughts around the story of a future IT expert traveling back with a time machine to our century. When he materializes, a large stack of magazines falls on his head - unintentionally - and he cannot remember anything but his IT knowledge. How would such a person evaluate the current state of affairs in software engineering?  Do you think, Mr. Spock from spaceship Enterprise will use pair programming or AOP? Wouldn't look very cool in a SciFi series, would it?

To understand what software engineering is heading for in the future, it is helpful to understand that all revolutions always are rooted in continuous evolution. At one point in time, existing technology  cannot scale anymore with some new requirements which forces researchers to come up with new ideas. Mind also the competition challenge. The first one to address new requirements may be the market leader. Learning from failure is an important aspect in this context. So, what is currently missing or failing in software engineering? Where do we need some productivity boosts? An example for such new requirements might be new kinds of hardware or software. Take Multi Core CPUs as an example. How can we efficiently leverage these CPUs with new paradigms for parallel programming.

As another example I consider the creation of software architecture. In contrast to many assumptions, most projects develop their software architecture still in an ad-hoc manner. We need well educated and experienced software architects  as well as systematic approaches to address this problem of ad-hoc development. Especially, the agility issue is challenging. Software engineers need embracing change in order to survive. Architecture refactoring is a promising technology in this area, among many others, of course.

Another way of boosting productivity is to prevent unnecessary work. Here, product line engineering and model-based software development are safe bets. If the complexity of problem domains grow we cannot handle the increased complexity with our old tool set, We need better abstractions such as DSLs. And I simply cannot believe for the same reason that in a world with an increasing amount of problem domains, general purpose languages are the right solution. Wouldn't this be another instance of the hammer-nail syndrome? Believe me, dynamic and functional languages will have increasing impact. 

In a world where connectivity is the rule, integration problems are inevitable. In this context, SOA comes immediately to my mind. With SOA I refer to an architecture concept, not to today's technology which I still consider immature. New, better SOA concepts are likely to appear in the future. SOA will evolve such as CORBA and all other predecessors did. Forget all the current ESB crap which already promises to cure all world problems.

There is a lot more which will influence emerging software engineering concepts. Think of grids, clouds, virtualization, decentralized computing! Look at CMU SEI's document on Ultra-Large Scale Systems Linda Northrop was in charge of. You'll find there sufficient research topics we have to solve before ULS will be available.

For us software engineers the good news is that we are living in an exciting  (although challenging) period of time.

The Silver Bullet

As I just read in a comment about silver bullets, I couldn't resist to address this topic :-)
Frederick F. Brooks brought up the concept of silver bullets. Of course, Frederick was not refering to some weird persons who are turning into computer geeks during full moon and can only be stopped using these nice little silver bullets. Instead, he claimed that there never ever will be any software technology that can boost software development productivity by a factor of 10 or more. So far, he has been right. Object-Orientation? Really productivity-boosting, but not to an order of magnitude. Models? Nice try, but someone has to build the model and the generator.
But won't there be any technology in the future that can refute this thesis? Suppose, an alien lifeform with incredible technology is visiting planet earth. How would they design software systems? Do we really believe, they would leave their giant space ship with a notebook, start modeling their systems with a kind of UML tool, and program in Java?
Thus, could the Silver Bullet be a universal law, or just a kind of constraint only humans are facing due to their limited capabilities?
Frederick F. Brooks also brought up the concept of accidental versus inherent complexity. Basically, each problem has an inherent complexity. You can't specify a a linear algorithm for the general sorting problem. Forget it, there is really no way! But we ourselves are causing a lot of problems by accidental complexity. Using an array for storing a sparsely populated 1000x1000 matrix isn't a very smart approach, right?
No technology can get rid of inherent complexity, but it could hide this kind of complexity from the user. Hiding complexity is one way to increase productivity - just make it a SEP (Somebody Else's Problem). Helping to address accidental complexity is another. Consequently, the best productivity boost results from combining both.
According to philosopher Thomas Kuhn, evolution of technologies is more a linear process. A technology can only scale upon specific limits. If it reaches the limits, workarounds will help postponing the necessity for a new approach. But at some point in time, a completely new and revolutionary approach will appear (such as switching from structured programming to OOP) and the same kind of technology cycle will restart. It is a kind of spiral model if you will.
The only point where we could observe productivity boosts are the technology revolution events. However, new technology revolutions are mostly rooted in existing technologies. Thus, an immediate productivity gain is simply not possible.
But as a thought experiment: if you just compared the way of approaching software development challenges now and 40 years ago, you could recognize a tremendous increase of effectiveness. There is a long way from software engineering in the sixties to that of today. No person could have skipped all the intermediate results and challenges.
Thus, if an alien race would land on earth, it appears very likely they could introduce us to new ways of building software systems that offers magnitude of productivity increase.
Conclusion: Silver bullets are theoretical beasts only existing in the Star Trek Universe. As pragmatic architects we must leverage the tools and technologies existing today. And we shouldn't believe all those technology panaceas vendors and market analysts are constantly promoting such as SOA, SaaS, you name it.

Now it is proven - NESSI exists!

ok, but what is it?

According to its web site, "NESSI is the European Technology Platform dedicated to Software and Services. Its name stands for the Networked European Software and Services Initiative."

Yes, it is not very clear from this definition. So, let me try my own explanation.

Suppose, you'd like to leverage SOA to connect heterogeneous islands. Maybe, you are going to build a healthcare system that connects different international regions, allowing physicians to access your healthcare data if required and permitted. An underlying SOA infrastructure should provide:

• SOA based middleware and process modeling
• Enterprise Application Integration facilities
• a kind of internet bus that supports various protocol families
• registries and repositories
• horizontal services such as security
•  

Sure, you can obtain such ingredients, if you accept vendor lock--in. I don't list all the big SOA proponents here as they are quite obvious. The truth is that is almost impossible to cope with such heterogeneity. If you try, you will be soon overwhelmed by sheer complexity.

Unfortunately, there is another dimension to this problem:

• Even if you have all these facilities at hand, you need to standardize the vertical domains. What if the whole SOI (Service Oriented Infrastructure) is standardized but two banks don't agree in what a customer or bank account is?

So far, solutions have either attempted to address the technology issue or to address the domain standardization.

What NESSI tries is to address those challenges.

It is required to create a kind of standardized SOA operating system and to define domain-specific standards for how to create applications on top of this platform. From my personal viewpoint the combination of Open Source Software with standards would be a perfect fit. It is necessary to create both, standardized APIs and a reference implementation able to handle mission critical applications.

Is a platform sufficient? Absolutely not, because we also need to cope with service engineering and governance issues and non functional requirements and tools. Consequently, NESSI is also trying to address these points.

All of these aforementioned  problems are hard to solve but on the other hand we have to solve them anyway, because they root in inherent complexity. If we need to solve them to finally live the dream of "the network is the computer", then NESSI represents an excellent opportunity.

But that's my personal viewpoint. What is your's?

Michael's Event Calendar

 

• In September I will visit my colleagues in Bangalore and give a seminar on Distributed System Architecture.
• On 14th October I will give a keynote on COMPARCH 2008 in Karlsruhe. I was invited by Ralf Reussner and Clemens Szyperski, two well-known gurus on architecture and components. Guess, what topic I will cover? Yes, architecture refactoring, my preferred topic these days. 
• Originally, I planned to attend this year's OOPSLA conference in Nashville. I have two accepted tutorials, one on high quality software architecture, the other one on Software Architecture Refactoring. Unfortunately, I am so busy within Siemens that I won't make it to the OOPSLA. However, an excellent member of my staff will give the two tutorials instead. Marwan is not an emergency solution but an excellent architect himself, formerly working for Lufthansa Systems. You will have a lot of fun attending our two tutorials. Mentally, I will be there, of course :-)
• In November I will most likely be invited to the Microsoft TechED in Barcelona giving two talks.
• In December I will be available on a panel at the Service Wave Conference of the European NESSI initiative.
• In January 2009 there is my local event, the OOP 2009 in Munich. I don't yet know what topics I will address but be sure I will be around :-)
• I am planing to submit for next year's ACCU in Oxford (April).

Mojave - Episode II

Did you hear about Microsoft's Mojave Project where they told a bunch of selected people - most of them Vista critics -  of a cool new operating system called Mojave? Supposedly, most of them were kind of enthusiastic about Mojave. To their big surprise, Mojave turned out to be Vista - the devil in disguise if you will in remembrance of an old Elvis hit.

What can architects learn from this?

• First of all, we can hide all the mess of our systems behind a nice look and feel. The best software system will not be accepted if its usability sucks, and vice-versa.
• Secondly, if we got an excellent system that people for some strange reasons don't appreciate, we could try to put a new API or Look & Feel on top. Maybe, the interface is the only thing that sucks in our system.
• Last but not least, we could also learn that nice architecture and design diagrams might impress others so much that they consider the system as a neat piece of design while in fact the implementation is totally different from the design.

Regarding the original Mojave experiment I really doubt that the reviewers had the write level of expertise and mind set. Don't judge a software system by its cover!  Review and evaluate the design and the implementation thoroughly. 

Architects shouldn't just believe but rather evaluate and enforce.

But of course, in some stakeholder presentations such as presenting the system to your top management you can add a marketing layer while remaining politically correct and honest.  

Art, Craft or Science?

Is software design an art, a craft or a science? This question has been the starting point for many entertaining discussions. And many different opinions and answers exist.

So what is my take on this?

To build a software system requires engineers to come up with a mapping that takes the problem and maps it to a concrete solution. Requirements describe the problem and the desired properties of the solution. Unfortunately, the mapping typically is not unique in that there might be many solutions solving the same problem. With other words, architects and developers face a lot of freedom. The bad thing about freedom is that it always forces architects to take decisions. In other words, to architect is to decide.

Of course, we appreciate if we can immediately find an appropriate solution for all problems, just following a divide-and-conquer strategy of software development.  But this never happens in practice.

Two problems come to my mind in this context:

• Sometimes we encounter problems for which we neither know a solution nor can find a solution elsewhere (finding a solution means either knowing a direct solution or being able to map the actual problem to another problem for which you already know a solution - for example, if we are able to map our algorithmic problem to a sorting problem.)
• In contrast to mathematics, the problem itself is a moving target in software engineering. Pantha rhei.

The first issue implies that you may not find a pattern or some other kind of best practice for your concrete problem context. Instead, you need some intuition. Hopefully, you recognize that the problem at least resembles another kind of problem for which you already have a vague solution idea so that this can serve as a good starting point. By the way, this is the reason why it turns out to be so hard to build a general problem solver for software engineering. Freedom is very hard to deal with by a general problem solver.

The second issue, coping with change, requires some sort of agility. In contrast to logic, our axioms keep changing while we are applying the calculus. The only way to deal with change is to embrace change everywhere, in the process, in the tool chain, in designing, in testing, in refactoring, in programming, in communicating.

What makes software design so hard sometimes is when both problems appear in parallel.

When people ask whether software engineering is a craft, a science, or an art, the answer should be obvious now. It is a science as we try hard to follow formal approaches where feasible. It is a craft because solving problems often requires experience and best practices. And it is an art as software engineering leaves enough freedom for intuitive and even artistic solutions.

I guess this is what makes our job so interesting. 

Architect always implements

When I once invited Bertrand Meyer to our location in Munich, he gave an excellent talk on Software Engineering. His most memorable messages were "Bubbles don't crash" and "All you need is code".

With the first message "Bubbles don't crash" Bertrand referred to all those engineers who have turned into Powerpoint or UML freaks. A Powerpoint slide or UML diagram will never crash (if you don't use it as a model for code generation, the MDSD guys might argue). It is surprisingly easy to impress people with simple boxes and lines (sometimes dubbed software architecture). Especially it is rather simple to impress top management which is why the wrong consultants can potentially cause so much damage. Some might respond that high abstraction levels are the only way to understand and build complex systems. That's certainly true. On the other hand, you would not be able to build a ship if you never had any idea about water. The same holds for software systems. It is simply not possible to develop a software system without any idea on top of which runtime environment it is supposed to execute.

Even if you had in-depth knowledge of your software system's runtime environment and built an excellent software architecture, it is NOT a no-brainer to transform a software architecture to a working implementation. That's why Bertrand introduced the "All you need is code" attitude. What helps you the best theory, if you can't turn it into practice? As an architect you need to have implementation skills, either for implementing parts of the system yourself or for assessing the results of other team members. How could you ever build a good bicycle without biking yourself?

The problem arises how to ensure that we as software architects can meet these requirements. "Architect always implements" means that you should also implement in a project, but definitely not on a critical path. By implementing you will earn the respect of developers, while keeping in touch with the implementation at the same time. And you will have to eat your own dog food. Whenever something is not as expected you can detect the problem and help solving it. However, in some projects architects are busy due to communication, design, and organization efforts so that spending time for implementation will be almost impossible. Never say impossible in this context, do not even think it!  Even in such scenarios I recommend spending some of your time for code reviews and some of your spare time to get familiar with software technologies being used in your project context. Otherwise you will create kinds of Software Frankensteins: creatures you are not proud of and who tend to live forever unless not eliminated in an early stage.

Architects are not just masters of Microsoft Office and  of a bunch of drawing tools they leverage to create design pearls, throw them over the fence to the developers who ought to build a perfect implementation, while the architects enjoy their holidays on a nice Carribean island.

Be pragmatic! I guess, the Pragmatic Programmers book series was created for exactly the same reason; to make us aware of the relevance of practical experience.

Never ever forget: ARCHITECTS ALWAYS IMPLEMENT. There is no exception to this rule.

Pseudo Effectiveness

Do you know these co-workers who are always answering e-mail in the middle of the night, who are in their offices until midnight, who have so many contacts to customers, who have an infinite list of activities, who have their mobile phone directly integrated into their body, and who always tell you about the big work overload they have?  All those unknown heroes who always sacrifice their spare time on the altar of the god of infinite work?

Of course, many of us in the IT world are more busy than it proves to be healthy or effective. But occasionally people only claim to be active and involved, while in fact they are not. Unfortunately, it is not easy to differentiate between actual effectiveness and that kind of pseudo effectiveness. Sometimes, the one who does the job gets not promoted while the one presenting beautiful slides about the work results of others is.    Of course, you may also refer to the Peter principle in this context :-)

Why do I discuss pseudo effectiveness in this architecture blog? As an architect, you might experience high impact if there are people with high pseudo effectiveness involved in your projects. Think about outsourcing sites where you have the feeling that productivity is very low. Think about developers or architects who have the same skill level, but whose work results are completely different in terms of quantity and quality. Also think about job promotions where you got the gut feeling of being ignored while in fact you did an excellent job in contrast to this other guy.

But be aware of the fact that all of us are somehow guilty of pseudo effectiveness. A little chat with co-workers in the cafeteria, some daydreaming in front of the workstation, some Internet surfing. All this may count up to a significant amount of time during the day. But as you might remember from my previous posts breaks are important. I claim that productivity significantly drops when ignoring such kind of work-life balance.

How can you detect real pseudo effectiveness? The only way is by analyzing actual productivity. For which tasks has the person been responsible for and what are the actual outcomes? In this context, the goals should be defined in such a way so that there is no space for pseudo effectiveness. Someone who needs a week to write a simple letter may seem very active, while in fact being very ineffective and inefficient.

Design Pearls

If you ever have been to Florence, you surely have admired all those incredible ancient buildings fully covered with ornaments. In the past, constructing beautiful buildings has been en vogue. Today, we simply could not afford the costs required for such endeavors.

What holds true for building construction, is also applicable to software development. But software engineers don't consider themselves just as craftsmen, but also as artists. We strive for architectural beauty and aestethic design, not being satisfied with functional aspects.

While architecture beauty is an important achievement, improving qualities such as expressiveness, simplicity or symmetry, design pearls define the other extreme.
By design pearls I am referring to masterpieces of software design which don't stop just meeting the requirements, but also add unnecessary burden (or ornaments if you will) to the system.

While design pearls show the mastership of one single individual they are counterproductive for team development as they decrease simplicity (there are more artifacts than necessary for the given task) and symmetry (they don't follow the KiSS principle but try to constantly reinvent the wheel).

We can avoid design pearls by strictly following a requirements-driven approach. All architecture concepts in a system must be rooted in a concrete requirement. In addition, we should use patterns where useful. A pattern is a "complete disregard of originality". It just helps reusing common wisdom, instead of coming up with new solutions. And we should strive for architecture quality indicators such as simplicity, expressiveness, orthogonality and symmetry. One way to aim for symmetry is to introduce design guidelines from day 1. Instead of letting developers figure out how to cope with cross cutting concerns such as exception handling, it is better to give them a framework of rules and guidelines.

In summary: design pearls are really nice to look at but ugly to change and extend.

An Anecdote on Architecture Training

Recently, I gave an architecture training within Siemens. There was another Siemens seminar in the same hotel dealing with project teams.  In one of the breaks, a guy asked me whether I knew more details about this architecture seminar. He could not believe, that someone within Siemens was teaching on building architecture.  I explained to him that the seminar was actually covering software architecture, not building construction. He asked me whether I could recommend the seminar. "Yes, absolutely", I said to him, "to be honest, I am the speaker". What can we learn from this: be cautious if you tell someone you are an architect.

Stop those Control Freaks

Suppose, you are in a meeting with other people. You need more information on the Hubble constant and ask the audience for details. Someone (maybe a friendly physics professor) comes to the white board and explains the theory.

Hmmh, how does that relate to software architecture? If you think about the situation, you'll recognize that the solution is based on a decentralized approach.

Normally, we are more used to centralization. Maybe, we are even addicted to all those centralized concepts. There is a central directory server, a central hub in the EAI solution, a central database, ... But as the example illustrates, sometimes decentralization is the better choice.

In the real life example we have used a P2P concept by leveraging broadband communication - ok, you just spoke very loud so that everyone paid attention to your question. Is this the only way to implement decentralization? No, there are several other alternatives.

For example,  Swarm Intelligence: Suppose, I can't remember where I have parked my car. To increase search speed, all family members will visit different areas to locate the car. All searchers carry mobile phones to communicate with each other. After my wife found the car, she tells everyone where it is.

Another example could be Ad Hoc Networking: In the nearby forest, I find an interesting bird. I tell other people approaching my location about the bird who themselves tell other people who ... After a short period of time the scene is crowded with interested people.  

Decentralized thinking means getting rid of central nodes or central controls. This might increase fault tolerance, but decrease efficiency. Often centralized concepts are mixed with decentralized ones. For example, there might be registries where all possible peers in a P2P network are known.

From an architecture viewpoint, several patterns support decentralization. For instance, the Lookup/Locator pattern, Blackboard architecture pattern, or the Leader/Followers pattern.

One of the qualities we can get from a decentralized system is emergence. The system as a whole reveals smart behavior that is more than just the combination of all nodes. Think of swarm intelligence like in an ant population.

From my viewpoint, decentralization is still neglected in software engineering, especially in software architecture. And as the examples shows, real life offers an abundance of decentralized concepts.

Architecture Design - How deep should it go?

In my last posting I introduced the Onion model for architecture design. Often it is very difficult to determine when architecture design stops and fine-grained design begins. Thus, I am going to illustrate some of my thoughts.

Before you start the most important precondition for architecture design is the availability of

• the business goal of the software architecture
• a subset of all requirements (15-30%), but make sure the most important ones are already available
• a coarse-grained test strategy.

There are different recommendations for architecture design. First of all, I consider it essential to differentiate between strategic design and tactical design:

• Strategic design is about all those requirements that have a systemic impact on the whole software architecture. Basically, these are the inner layers of the onion. All architecture decisions that do not only reveal a local impact on a component together contribute to architecture design.
• Tactical design is simply about the rest: all decisions that show only local affects. Interestingly, variability issues often denote tactical design issues, even in product lines. Things like exchangeability of an algorithm only imply, we need to open our strategic architecture to support variation of the particular family of algorithms (a.k.a. strategies).

This definition comes close to what Grady Booch once meant by "software architecture is about the important things".

Another consideration consists of reducing the number of abstractions. This is what I call the Pyramid model:

• the top of the pyramid is the system itself,
• the middle layer represents all subsystems,
• the bottom layer comprises all components the subsystems consist of

For architecture design we should only follow the pyramid model. All abstractions underneath the bottom layer of the pyramid are part of fine-grained design. Note, that it might be subject to your own definition how the system, subsystems, or components of the pyramid model map to the concrete abstractions in your domain. This might look different between an embedded system and an Enterprise SOA system where subsystems might become services.

All architecture decisions should be completely rooted in requirements and risks. All extra add-ons increase complexity and thus decrease simplicity and expressiveness. 

Don't be caught by the trap that architecture is only about problem domain and not about solution domain although you always should strive for keeping your architecture as independent from specific technical issues as possible (always try to defer all technology decisions to a later phase). "Hard" requirements such as "cycle time must not exceed 1 msec" or "use of SQL Server 2003 is mandatory" will be treated as first class requirements. And those requirements may also have systemic impact, thus contributing to strategic design.

In the documentation of your architecture follow a newsletter style. This implies, each artifact (system, subsystem, component) should be explained using a newspaper-like article. Architecture design is just that and no more.

In addition to architecture design you should:

• specify design and architecture guidelines for all crosscutting concerns. This is best done before architecture design for all important systemic requirements. For other concerns you might create guidelines in parallel to architecture design.
• create technology prototypes for all important risks. For example, if you need to guarantee 2000 transactions per second, make sure your middleware of choice really supports this requirement.
• refine the test strategy you should have established after requirements engineering.

These rules of thumb are intentionally somewhat vague. Again, I'd like to refer to Grady Booch in this context: "there is no cookbook for software architecture".

Fortunately, there are some good practices to share.

The Onion Model

Whenever people ask me how to systematically create an architecture, I present them my Onion Model.
 
If you like to sow an onion, you need some preparation. It is important to take care of the right environment and also to think about the requirements an onion needs to grow and what risks you might encounter. From a business perspective, you should also consider what type of onion you like to harvest, but that's a different story.

The inner core of the onion will be the domain model including all core components, their relationships and interactions.

For the next layers you will need a prioritized list of strategic requirements. These requirements influence systemic architecture decisions. For example, reliability, availability, or security. All of them represent operational requirements that result in specific infrastructures. If performance is the most important operational requirement, you will build a performance layer around the domain logic core. With other words, the domain logic will be embedded into the performance infrastructure. Then you turn to the second most important operational requirement. If security is the number two requirement, your existing onion that consist of the functional core embedded to an performance infrastructure will get wrapped by a security layer. This continues until your architecture is stable.

At the end you need to address tactical issues such as flexibility or maintainability. These will become the outer layers of the onion. For some of them, you will need open up your strategic architecture, for instance, by providing hooks.

Eventually, you'll get an "onion" which was designed with a functional core in mind. All important requirements had an higher impact on the onion than the less important ones which was ensured by priorization of onion layers.

Focus matters

A lot of software systems do not meet some of their requirements and business objectives. Other systems have turned to design pearls only a few architects can understand. In either case you are in trouble. If a system does not meet its requirements or business goals customers will be dissatisfied. In addition, the system cannot be evolved to future versions before the problems are addressed, often causing large and expensive refactoring activities. If a system is a design pearl, it contains a lot of unnecessary architecture artifacts, making the system less expressible and simple. After short or medium time, these systems tend to become big balls of mud.
To prevent such problems there are different means. One of the most important one being requirements traceability. Each architecture change must address a requirement and/or a business goal. You should also clearly trace back the architecture decision to the requirement or business goal in your documentation. Architecture review methods such as ATAM use this information to figure out tradeoff points, sensitivity points, risks and chances of your architecture.
In the optimal case there will be patterns available to support your architecture design. For orthogonality you should apply the same patterns for the same problem contexts. Likewise, relevant cross-cutting concerns such as error management should be explicitly addressed by architects and enforced through guidelines.
Note, however, that architects may come up with the best architecture or the most consistent and complete guidelines, but will fail if the implemented architecture differs from the envisioned architecture. Thus, architects should walk around and communicate to enforce the architecture vision.
Just staying in office, throwing the architecture over the fence, and hoping everything will be in place after a while, is the best receipt for project failure.
As change is the rule and not the exception, architects must always ensure that any change in the set of business goals or requirements will lead to an appropriate architecture refactoring. With other words: if the focus may change and you need to stay in focus, then you'll have to change if the focus actually changes.

There is no business like office business

In the last weeks I enjoyed small talk with several software engineers from various companies. Topics included technology and architecture, but also office work, management, colleagues and career paths. And several times I heard from companies where a couple of characteristic metrics for measuring engineer productivity exist. Let me exaggerate a little bit:

• One of the wide spread metrics is office availability. I am speaking about engineers sitting in their offices or cubicles from early in the morning until late in the evening. This is something I fully understand for projects being in critical phases or delayed. Of course, I also did it and will do it in the future. But I also heard about engineers staying in their offices very late to send an implicit message to their management. Something, like "Look, I am working so hard for the company. If you don't promote me, you should suffer from remorse." However, the truth according to several studies is that you cannot be productive for 8 hours or more per day, at least not over a longer time span. Even worse, if you stay in office, your bug rate will increase significantly and the results will be not better than staying in office for less time. I know the stories about employees in Silicon Valley or Redmond sleeping near to their office PC. There is a reason why eXtreme Programming does recommend reducing work time to under 40 hours per week. And all of you believing in 16 hour work days, no matter wether you are good (engineer) or evil (manager): productivity in such overload situations is not possible. You either have many simple tasks or spend significant time twiddling your thumbs. Today, I read an article that an increasing number of IT workers suffers from the burn-out syndrome or other diseases which are caused by work overload.
• Another wide spread metrics is internal visibility. Do you remember Bertrand Meyer's "bubbles don't crash"? Management tends to rather promote engineers who spend significant time presenting and chatting to management. If you don't provide a steady stream of good news (even if you invented this stream of good news) to the right persons, they won't even notice your existence in the worst case. Often, not the real results count but management visibility which is measured by .. guess what .. Powerpoint presentations, Word documents and Excel sheets. Don't forget that e-mails and meetings represent additional, important means in this context. Considered from this perspective, virtual reality has already conquered the world.

There are a lot of lessons we can learn from these statements.

• First of all, Work/Life balance may be a buzzword, but from a useful one if you ask me. For example, if I recognize that my productivity is dropping to almost zero,  I am leaving office, go for running or biking, and then continue with my work, often until late in the evening. Such extended breaks provide me with a significant productivity boost. It might appear unbelievable, but I am definitely more productive that way than if I had spent the breakout time for office work.
• If I need to concentrate I will also work at home from time to time rather than spending the time in the office. With my phone constantly ringing, colleagues dropping by, and other continuous interrupts, offices don't really provide an adequate environment for creativity and brain workers. Ok, I am missing my colleagues and the social interactions after a short time. Thus, home office is the exception. Fortunately, there are so many opportunities (phone conferences, meetings) to interact :-)
• Effectiveness and efficiency are/should be the only things that count. It does not matter when or where you created your results. However, you need to make sure, managers are aware of your work and what you achieve. Marketing your own person is an important issue in a universe where managers can't afford tracking all work results from all engineers. If you coach some colleagues as a mentor and if you promote their visibility and career path, inform your managers. Improving skills of co-workers and other more indirect results offer high business value to a company.

Interestingly, all these points are also valid for architects. However, home days are often a bad idea in those hot project phases when creating or enforcing software architecture. For an architect, communication represents the most important instrument. Communication works best in Face-to-Face meetings. But, nonetheless, also in these situations you need to plan your recreational breaks to enforce Work/Life Balance.

Developer Usability

In the last months I've seen a lot of frameworks, platforms and libraries which revealed a substantial lack of usability. Developers reported to me that using those APIs was more a nightmare than a pleasure. Some of them just stopped using these artifacts, others built their own workarounds on top. When asking the API-developers for their comments, it turned out that most of those APIs have been built without any consideration of usability. In addition, adequate documentation was missing very often, which of course is also related to usability. Interestingly, I often hear the same complaints from end users. The UIs of most applications or devices are a pain. For instance, I am using a SAP-based system for ordering work items. What I need to fill in in all those fields is by no means self-explanatory. Help is missing or inappropriate. Error messages don't give any guidance what could be wrong. Suppose, shops like Amazon would come up with such UIs. How successful, do you think, would they be? Whenever we are building such systems we need two things:

• a clear understanding of the domain
• a tight interaction with users to ensure usability

For example, if we read the book by Krzysztov Kwalina and Brad Adams on how to design frameworks, you will find the recommendation on scenario-based design. Find out how end users could best use the system you are going to build in terms of user scenarios. Implement the API to support the most important and common scenarios. After the first prototype/iteration of the system is available, ask the target audience to use it and give feedback what they like or don't like. Then improve it!    

In contrast to common belief, architecture is not only about functional requirements and operational qualities. It is also about usability. A system that meets all requirements but is not usable, is of no value for its users. As an example: I owned Rio and Creative MP3 players which provided an incredible amount of features, but lacked usability. After a while I switched to Apple IPods which have less features and are significantly more expensive but offer a great user experience. 

At the end, you should strive for making the end user's life a pleasure. In this context, just remember that you yourself are using many more systems than you are developing.  Something similar is written in many public restrooms within Germany: Leave the room in such a way you'd like to discover it.

On vacation

If you already wondered, I am currently on vacation travelling abroads. That is the reason why I have only seldomly access to the Web. Now, I am sitting in front of an Internet PC in my hotel's lobby. I will be back on 28th March. Then, I will provide you with some new stuff. So far, I wish you some nice easter holidays.