Now for Something Completely Different- SIEMENS

Many of you may know that I am working for Siemens for almost 16 years now. I got employed back in 1991 by the Siemens R&D unit in Munich where I started my career as a software engineer and researcher. Some of you may also be aware of the fact that there was a high frequency of news about my company in the last months. Unfortunately, most of the news were rather on the negative spectrum. And even worse, as Douglas Adams once said, there is only one thing that can travel faster than light: bad news! My friends and relatives keep constantly asking all the time what is going on at Siemens.

To be honest, I am really unhappy with this kind of questions. Thus, It is time to shed some light on the many achievements and historical facts Siemens has provided over more than 160 years since it was founded in 1847 by the inventors Werner von Siemens and Johann Georg Halske. People seem to ignore all these great achievements which by far outnumber all the negative reports you may have heard recently. Some of the early innovations made by Werner von Siemens himself include a special telegraph, the trolley bus, or a dynamic transducer. But this is only the tip of the iceberg in terms of Siemens innovations. Thus, Siemens always has been a great place for research and engineering from day 1.

Werner von Siemens happened to be not only a great inventor but also a very kind person. He was the first entrepeneur to introduce a system of different social insurances for his employees.

Siemens constantly grew until today where we have reached a number of about 480000 employees working in almost all parts of the globe and creating a large range of high quality products such diverse as trains, plants, automation systems, medical systems, communication systems, automotive or lighting solutions as well as IT solutions. Did you know that scientists like Gustav Hertz or Walter Schottky were employed by Siemens?

What makes Siemens an excellent place for software engineers is the mixture of advanced and innovative products that require more and more IT centric constituents and a very creative setting biased towards innovations which is the reason why the company considers itself a global network of innovations. As the Siemens groups require a whole bunch of different software technologies and tools, I consider it a great opportunity to be part of the Corporate Research and Technology division of Siemens. Here, I am able to cooperate with colleagues from all Siemens-relevant applications domains. And as you might guess Software Architecture is one of the fundamental disciplines in such a context. All the books I co-authored and research results I achieved on architectural topics would't have been possible without such a creative and supportive environment where I am able to closely connect with people such as Frank Buschmann or cooperate with smart experts such as Doug Schmidt, Markus Völter and the like.

Why did I write this rather emotional posting? Because I consider it very unfair that in the last months public attention continously focused on only some illbehaving persons within the company ignoring all the aforementioned achievements. Siemens has already started to consequently investigate this behavior and establish a system of very rigid control mechanisms. On one hand, it is important to reveal any wrong doing within a company. But on the other hand, its is wrong to judge a book by his cover (as Franknfurter mentions in the Rocky Horror Picture Show). Of course, I am biased as I am a Siemens employee myself but that doesn't change my personal attitude and opinion.

I am very proud to be part of the Siemens family where I keep working on great products and software architectures.

For more information visit the following Siemens site.

Upcoming Events

I am in vacation for several weeks now. Nonetheless, I havent't been lazy :-)
One week ago I participated in a panel discussion at this year's ECOOP conference in Berlin. The panelists, among them celebrities such as Martin Odersky, Judith Bishop (moderator), Tiziana Margaria, Gilad Braha discussed "OOPS in the next five years - the hot topics". All of the panelists tried to argue from different perspectives introducing topics such as Dynanmic Languages or Service-Orientation, My key points were more driven by my industry background. I don't believe in Silver Bullets. Thus, I guess, OOP will experience some significant improvements integrating other paradigms such as functional programming and providing advanced features such as traits. However, I anticipate specialized OOPS solutions for specific domains. I also assume, we will have to live with other paradigms, due to the different kinds of impedance mismatches we are experiencing in every day projects. And I am sure, someday a new paradigm will evolve that is going to integrate and extend OOP.
In my vacation I also had to prepare for my two OOPSLA 2007 tutorials. The one on high quality software architecture covers process principles, architectural principles and quality properties relevant for achieving high quality in software architecture design. The preparation of my second tutorial on Software Architecture Refactoring was far more challenging. I thought, there should be a lot of material available on this issue. To my big surprise, there were only a few sources on that issue (which I already covered in a previous posting). Most sources refer to code refactoring such as introduced in Martin's excellent book. Architecture Refactoring deals with semantic preserving transformations of the software architecture itself. In my tutorial I will come up with some general concepts, a catalog of architecture refactorings as well as some words about how architecture refactoring relates to other disciplines. If you'd like further details, you might want to listen to the OOPSLA podcast (see episode 5) where Bernd Kolb from Software Engineering Radio interviewed me a few weeks ago.
In my OOP 2008 presentations I will also talk on Software Architecture Refactoring. Moreover, I will prepare a one day tutorial on .NET technologies for building SOA applications.
Note, that I won't be able to participate in this year's JAOO conference. I hadn't been invited as a track chair but Ted Neward asked me whether I'd be able to give a talk on Enterprise .NET. However, there is such a lot of private affairs and Siemens internal work that I cannot make it this year.
Thus, I am looking forward meeting some of you at OOPSLA.

Strategies and Tactics

Software architecture is about the components of a system and their relationships. It also comprises the guiding principles that lead to these components and relationships. Finally, a software architecture deals with multiple viewpoints on the same system, for instance structural and behavioral viewpoints. Hence, every software system reveals a software architecture. However, if you just enter the implementation phase skipping analysis and design you will follow an unsystematic approach which inevitably leads to an ad-hoc software architecture. For all but trivial systems a systematic approach is important to achieve software architecture quality. Quality in this context means to meet all requirements, the implicit and the explicit ones - of course, considering the viewpoints of all stakeholders. One of the most important disciplines in a project is mapping requirements to the software architecture using a systematic approach. This does not only imply drawing bubbles in graphical tools. At the end, software architecture is only a means for implementation, not less and not more. Reminds me of the famous Bertrand Meyer, who once said things like "Bubbles don't crash" and "All you need is code". All these considerations lead directly to what we call tactics and strategies. A strategy represents a guiding principle helpful to achieve a particular goal in the software system as a whole. For example, how should your application address issues such as availability or fault handling. To guarantee conceptual integrity and orthogonality strategies shouldn't overlap. Let me introduce an example for such an overlap. Assume, the architects in a project come up with the following strategies: (A) Whenever a security exception appears, a 32 bit error value must be returned. (B) For reporting and handling all kinds of errors and exceptions, structured exception handling shall be used. Both are strategies as they deal with guiding principles for the whole application. But now suppose, you are in the role of a developer. How should you deal with a security exception? Because the two strategies overlap, it is very likely that some of the developers will return error values while the others will apply exception handling. Why is this bad? Overlaps always have a negative impact on understandability and readability of the software architecture. What are tactics? In contrast to strategies, tactics are more fine-grained and closer to implementation. A tactic helps solving local problems instead of addressing a whole software system. Design patterns are very good examples for such tactics. Assume, you have implemented a temperature sensor component which needs to notify other components when specific temperature tresholds are reached. Of course, this is exactly a problem context where the GoF Observer pattern comes into play. Needless to say, that tactics shouldn't overlap either. In the ideal case tactics also help implementing strategies. By the way, architecture patterns such as Model-View-Controller are often applicable to define strategies, while tactical patterns such as Observer help implementing the architecture pattern. I guess, this gives you a better understanding what strategies and tactics are and how they relate with each other. For the job of a software architecture this classification implies a two-step, top-down approach. First define the base line architecture using the strategies that are derived from requirements and are supposed to drive the whole software architecture. In the next step, refine the software architecture base-line using tactics. Yes, but in which order should I integrate strategies and tactics? As strategies (and tactics) must be derived from requirements, let risks and priorities drive the order of architecture design. A system where performance is much more important than platform independence will reveal a completely different architecture than a system where platform independence is king. Thus, in the first case you'll first encounter strategies that deal with performance (caching, pooling, load balancing, concurrency strategies), while in the second case you will preferably apply appropriate strategies for platform independence (strict layering, wrapper-facades, reactors). A good exercise could be to look at your current or last project and figure out all the strategies and tactics applied there. Did you systematically consider and integrate strategies and tactics? In every project, you should regularly take some time and reflect about the project to keep it on the right track!

What the hell is an ESB?

When I am getting involved in discussions about SOA recently, ESB is one of those terms I often hear. ESB is another citizen of the SOA buzzword universe, comprising other TLAs such as BAM, BPM, EAI, or B2B. One of the most entertaining and fun exercises is asking senior management and other powerpoint engineers what exactly they mean by ESB. Guess, how many definitions of ESB you'll discover when asking this question? Once, business magazines such as the Wallstreet Journal start talking about SOA, you should be aware of the following facts:

• senior management may soon ask your IT department for a SOA strategy
• it may be the right time to sell shares of SOA vendors
• you'll find all those market analysts attempting like crazy to convince you of the business impact of the aforementioned TLAs

Another fun part is talking to techno nerds about SOA, especially those who believe CORBA and the like will cure all world problems. Did you ever encounter this "I may not know your problem, but I know that CORBA is the right solution" habit? On one hand, these techno nerds will point you to all performance penalties caused by SOA, because they think SOA is all about WS-* standards only. On the other, they will prove that CORBA is SOA, because it already provides the same solution concepts such as separating interface from implementations and leveraging standardized protocols.

Nonetheless, SOA makes a lot of sense from a technology and from a business perspective. When integration of heterogeneous environments is a key issue in your product and solution development, SOA may be the appropriate choice. SOA is neither a ubiquitous solution for all networked systems as business managers often think, nor is it that inefficient as techno nerds believe. The reality lies somehow in the middle between these extremes. In fact, SOA is an architectural paradigm that may even be applicable for CORBA based solutions (but fits much better with message-oriented middleware, to be honest). Of course, SOA in its WS-* instantiation is a perfect alternative to EAI and BPM approaches.

When SOA started to evolve, it was considered a set of architectural concepts that must be implemented using a stack of low-level protocols (such as SOAP, WSDL, UDDI). This resembles how CORBA and DCOM started more than 15 years ago. For instance, the first CORBA products didn't even provide interoperability. They just offered the same standardized, uniform programming interface. To guarantee interoperability, the OMG introduced the IIOP (Internet Inter-ORB Protocol). Shortly after real world projects started integrating CORBA ORBs, they required advanced services to deal with events, security, discovery or transactions. Hence, the OMG initiated efforts to come up with the so-called COSS (Common Object Services Specification). The availability of COSS, however, did not solve all issues either, because engineers had to manually deal with CORBA ORBs as well as with a whole bunch of rather complex COSS services. In addition, not all vendors provided all the required services. Hence, the CCM (CORBA Component Model) was introduced that decouples developers from lifecycle management and service issues.

In the SOA/WS-* universe this storyline re-appears. Think of SOAP and WS-I (IIOP)! And think of WS-Coordination, WS-Attachments, WS-Security (COSS)! Finally, ask yourself which part of SOA is playing the role of advanced technologies such as CCM. This is exactly where ESB enters the stage. ESB is one of the core assets of any SOA strategy. It provides a container that decouples implementation details such as communication protocols from developers. It also adds (or should add) lifecycle management, routing, transformation, security, management and discovery services. Without an ESB, engineers have to face the challenge of dealing with a bunch of different (mostly low-level) technologies they need to integrate. Unfortunately, big players such as SAP or Microsoft do currently not provide integrated (!) ESB products. And those companies selling ESB solutions often provide "heavy-weight" containers with a high degree of vendor lock-in.

What we actually need are light-weight ESB solutions that are based upon standards such as SCA or OSGi and interoperate with other platforms (such as WCF/WF/BizTalk). Such ESB solutions should cover the whole range of application domains and support enterprise applications as well as embedded applications.

We have started our journey but it will take a long time to reach the destination.

Post Mortem

Did you ever observe small children when they are learning new capabilities such as walking or biking? They are not masters from the beginning, but step by step they are able to improve their skills until they reach a level of mastership. Do they give up in the case of failure? Only in rare cases! Instead, they mostly learn from their failures.
Did you ever observe software architects in projects when they are developing new products or solutions? How do they deal with failures? Interestingly, whole organizations and individual architects only seldomly learn from failure. Failures are not really impressive in any performance review or project meeting. That's the reason why they are often ignored or even forgotten. How often in your career de-ja-vouz situations appeared where you couldn't believe that the same errors were made again and again even by smart people?
Now, think about famous inventions such as the light bulb? Inventors and scientists such as Thomas Alva Edison succeed because they continuously reflect about their own activities and leverage failures in a constructive way.
So what conclusions should be drawn? If you like to find the right direction in a labyrinth use an Ariadne thread. Whenever you encounter a dead end reflect and then try a new direction. Or shortly: learn from failure!
Software architecture design is complex as you need to balance multiple and sometimes even interdependent forces. Thus, failure is a fact and not a seldom accident with respect to software development activities. There is a whole range of possible failures. In the small, some design decisions might be wrong. In the large, the whole architecture respectively project might fail. The best way to deal with such situations is analysing when and why wrong decisions were made that lead to the failure and which alternative decisions would have been the right ones. Agile processes explicitly consider the possibility of failure introducing practices such as testing and refactoring.
I made the experience that a post mortem analysis of any project (also known as architecture review) is really helpful to learn and improve my own skills and experiences as a software architect. Of course, this also holds for projects where you were not involved. Unfortunately, western culture does not really motivate writing articles about projects that failed.
The same "learning from failure" pattern does not only apply for software artifacts but also for software development activities. In the ideal case we would reflect over our results and activities daily. Correcting a wrong direction early is rather inexpensive. Correcting it lately will require substantial corrections. Shouldn't a culture of learning from failure be an important part of any (development) process?
Software architecture is the core asset of any software product and determines the quality of the result. Failure in software architecture activities has a significant impact on the implementation. Needless to say that failures must be handled with care, explicitly analysed and addressed by architects.

ESB

In the recent time I have been involved in investigations whether SOA represents an appropriate paradigm for buidling highly distributed enterprise and embedded systems. What makes such efforts really challenging is the fact that the SOA universe is polluted with all kinds of acronyms. If you really like to confuse innocent colleagues or customers, I'd suggest to throw terms such as BAM, BPM, WSIT, WSDL, BPEL, BI and the like at your communication partners. The either will consider you a smart guy or a professional IT pretender.

Work Life Balance

This time I like to address a completely different topic which seems to be completely unrelated to software architecture and software engineering, at least if you only scratch the surface. People like you and me working in a software engineering company or department tend to spend a lot of time for their job. When I started my career, project work was so exciting that I often continued to work in my spare time. Guess, what I did during vacation? If someone asked me for assistance or provided me an additional challenge, I kept saying yes to every offer. After some years I experienced total work overload and even got some health problems. I was forced to spend more time for other activities such as sports, arranging music, photography, or literature to recover from these overload situations. At the beginning, I thought and feared, spending more time for such leisure activities would significantly reduce my productivity. But to my surprise I made exactly the opposite experience. Spending more time for non-work related activities, significantly increased my productivity. That is also the reason why extreme programming suggests to limit work time to 40 hours per week. If you ever admired those colleagues who stayed in office for more than 10 hours day by day, you should consider the fact that after a specific amount of work time bug rates increase dramatically and work results tend to reach lowest levels. Thus, working too hard for a long period of time could even have a negative effect. Reminds me of Dilbert's boss who always keeps saying "work smarter not harder". I really enjoy running or biking after work and during weekends. My favourite place is the Isar river in Munich which offers beautiful trails and helps me first forgetting and then successfully addressing all those work-related troubles and worries. If you ever had to cope with a complex problem in a project and couldn't find an appropriate solution despite of your hard efforts, just stop working and go running, biking or what else you prefer. In many cases, this will also help finding an appropriate solution for your project later on. Over the years, I spoke to many famous friends in our software engineering industry and they made exactly the same experience. Most of these well-known experts spend significant time for additional non-work related activities.
If you want to become am accepted expert in your field, take care of an appropriate work-life-balance, or in terms of work activities: sometimes, less is more.

Release It!

A lot of Java and .NET developers are involved in Enterprise and Web development projects. According to official statements by senior management or marketing of software development companies or departments, no one ever seems to have faced major problems in such projects, not to mention catastrophes. However, if you meet other developers at all those software engineering conferences worldwide, you will hear some war stories in personal communications - maybe after a few bottles of beer or wine. According to Henry Petroski "learning from failure" is one of the most important issues in software development. We all have made faults in the projects we were involved. Thus, one way could be to forget those errors in the next project and cause the same problems again and again and .... Do you know the de ja vu feeling when you think in a concrete project situation "why do they make exactly the same mistakes like in previous projects". The other way to address these project failures or problems in a constructive way is to learn from them without trying to identify a scape goat, because attacking people won't be of any help here. It is the whole organization including all stakeholders that should be ready to learn from failure in a positive way. Recently, I read a book on this issue which was written by Michael T. Nygaard and published by "The Pragmatic Programmers" - I really love the books from "The Pragmatic Programmers", to be honest. It is called "Release It! Design and Deploy Production-Ready Software" and covers problems related to software production and deployment. The author illustrates potential risks, he derived from real life projects, as well as countermeasures to cope with these risks. It is exactly the kind of book that is helpful for practioners like me. The book is partioned in four parts: part I deals with stability problems, part II with capacity problems, part III with general issues and finally part IV with operations of software systems. I really enjoyed reading this book and learned a lot. I am too lazy to go into all the details, because it is really better for you to read the book yourself. I'd like to see more books of this kind. Of course, I understand that no company would be willing to share all its failures with the rest of the world, especially with its competitors. But unfortunately, only a few companies have a philosophy of learning from their own failures. Thus, books that abstract such problems from the real projects, and show potential pitfalls and their solutions in a more general way could be a treasure chest for all software engineers. Believe me! Documenting patterns is much easier as they represent proved solutions. No one will complain about documenting patterns for this reason. But I won't give up trying to convince everyone that a culture of learning from failure would help us being more productive and successful. At the end, most technical (r)evolutions are caused by learning from previous solutions that reached their limits. Maybe, someone has read such books and can give some recommendations. More information on the excellent book by Michael T. Nygard is available here.

Product Lines and Platforms

In Software Product Line Engineering one of the core activities deals with determining the central assets common to all programs in the scope of the product line. These assets are not constrained to implementation artifacts but might contain models, aspects, documents, test plans, to name a few. Jan Bosch once proved that ad hoc re-use of assets does not work which leads to the conclusion that re-use of assets in product line engineering requires a more systematic kind of approach. It also implies that re-usable core assets do NOT represent a set of unrelated components, but rather should be combined in a predefined and more tightly coupled aggregation. Now, we are entering the universe of software platforms which denote the central fundament of software product lines. The glue of a platform might consist of different elements such as configuration files or DSLs & generators. It is noteworthy that the platform represents the central core of the software product line including all common components but also supporting the range of variabilites within the program family (BTW, I've already explained in previous postings how architects may analyze commonalities and variabilites.) From the platform all instances of a product line should be derivable by using configuration mechanisms, APIs, a DSL, or other means. If you think, you only seldomly face platforms and product lines, you are plain wrong. The Java JDK and the .NET Framework Classes are examples for such platforms such as all other commonly used APIs. From my viewpoint, all SOA based applications also represent examples of product line engineering.
Why this platform-based approach sounds very reasonable it also reveals some challenges. Suppose, different product line instances, e.g., various mobil phones, leverage a common platform, e.g., a software platform that includes all required communication stacks such as the UI, persistence features, power management, etc. What if one of these application projects requires a modification of one of the common core assets within the platform? There are different alternatives in this context. As precondition, I assume that the modification could not be handled by variation (using a kind of variability hook in the platform), but requires the modification of a common platform component. In a naive world, we would just derive a new branch of the platform without any impact on the other family members. It is a no-brainer that this alternative is unacceptable in the real world as further change requests could emerge each of them leading to a new specialized platform version. Thus, we would end up in a sea of platforms. This is of the same dimension such as asking Microsoft to provide a customized Windows for each customer. Why is this such a big problem? Suppose, that you need to evolve the platform in the future. How can you manage evolution facing a sea of platforms? In the best case, it will work with large efforts and costs, but lead to design erosion forcing the platform team to come up with a complete new platform every second or third product line generation. Needless to say, that the RoI through re-usability which was the primary reason to establish a product line won't be too impressive in such a scenario. Thus, we need another way of handling modifications! Any change request should lead to a modification of the platform. Yes, this will require all application projects to leverage the modified platform. When the platform is a significant part of the application, then we should be ready to pay this price for the sake of platform evolution. In such an approach the platform will be evolved in a systematic way, thus reducing the risk of design erosion. However, successful platform lifecycle engineering is not only a matter of implementation. The development organization must establish a team that is in charge of platform development with a clear interface (and firewall) between the platform team and the application projects which are building the product line members. Otherwise, we will end up in exactly the same approach as illustrated in the first alternative. Of course, a platform team is responsible to meet its customers' requirements. At the same time, a platform team should be empowered to reject all change requests that have a negative impact on the majority of application projects. Moreover, application projects need to synchronize with platform development, and also send change requests and bug reports to the platform team. For these co-operation aspects, an appropriate product line engineering process and tool chain must handle all synchronization and coordination issues. MDSD techniques (Model-Drived Software Development) help to decouple platform from application developers, while increasing productivity. This is the reason why MDA (Model Driven Architecture) introduces PIMs (Platform Independent Models) and PSMs (Platform Specific Models).
To summarize, product lines and platforms denote twins, because product line engineering requires the development of a platform that combines core assets in a common way, while integrating variability management. Unfortunately, platform development is one of the underestimated activities in product line engineering, which is why many projects fail. The reason for this is obvious. Each platform modification has an impact on ALL program family members and is not constrained to a single one-off-application. In addition, the situation gets even worse and messy if you consider the fact that a platform is not only a set of implementation artifacts but also includes tests, documents, models, aspects, among many other entities. Thus, only a systematic platform lifecycle engineering approach will help successfully develop and evolve program families.
If done right, product lines can save time and money. If done wrong, they will make developer life like hell.

Doomed to fail

Suppose, an organization offers you a large amount of money if you agree on the following arrangement. You'll apply for a management position offered by a competiting company that is presumably going to develop a new innovative and software-intensive product. And it will be your task to sabotage their project. I know, that you never ever would accept such an unethical offer. Nonetheless, I am asking you for the moment to think of the unlikely case that you would agree. The main question in this context is how can you manage this task successfully. As you might have expected, I got ten recommendations for you:

• Decide to use a waterfall model. If engineers complain, just tell them that you can't live with the uncertainties and missing predictability of agile approaches. Convince engineers that computer science is a science which implies that they could easily collect all input at once in the beginning, and then systematically design an appropriate software system. If they object to this statement, ask them whether this means they are not able to follow the same kind of approach as other disciplines. Make clear, that from your standpoint agile processes are only for pussies and that you don't respect proponents of this paradigm. If they tell you that agility is about addressing risks, respond to them that you consider agile developers the highest risk in any project. For the same reason, reject any test-first activities from the beginning. As a revenge, tell the development team that in order to reduce budget, multiple engineers will have to share one telephone and one workstation. This is what you understand by Pair Programming.
• Don't introduce any roles. This makes your staff clueless and helps abide to the "Jack of all trades but master of none" approach. People should feel either responsible for everything or for nothing. Using this setting, project members won't be able to ever know whom to address for clarifications. Everyone will be developer and architect. Don't assign the role of a lead architect as this would unnecessarily improve productivity. There is only one predefined role. Make the most unorganized person in your team the project manager.
• Requirements are essential drivers for any project. Thus, a perfect means is to collect a large set of requirements that are as vague as possible ("our application should be flexible"), and in the best case are contradicting ("system should be hard realtime but offer runtime flexibility") or not feasible ("we need 100% availability without any additional costs"). If architects ask you to assign priorities, just tell them that all requirements are equally critical and important. Add to this recipe another dimension by constantly adding, changing or redefining requirements over space and time, especially in later phases of the project. Needless to say, you should never remove any requirements!
• Communication should be leveraged in a proper way. On one hand, try to minimize effective communication between engineers. On the other hand, fill the schedules of your engineers with an infinite sequence of unproductive meetings. These meetings should not have any agenda, be open-ended, mandatory for all engineers, and unstructured. In addition, embrace a culture of e-mail communication with dozens of mails regularly spreading around. Tom DeMarco has labeled this kind of paradigm "Corporate Spam". Obviously, continuous demotivation is also helpful here. Never give developers any positive feedback. Only refer to the incredible and stupid mistakes they made. Introduce the "moron of the month" award for this purpose. Be a control freak and check and discuss every decision in the project, even the unimportant things. This will drive people crazy (e.g., in meetings) and will enlarge your level of joy.
• Outsourcing represents an additional means to worsen the situation. Thus, you should follow an offshoring and outsourcing strategy to play the communication game introduced in the previous discussion. Especially, selecting outsourcing teams with completely different culture is of high importance in this context. Don't let anyone figure out in the project team nor in the outsourcing locations who is in charge of what. One face to the customer won't work very well here, because your goal is to play the game of engineer torture. In the optimal case, language skills of involved persons should be low as otherwise communication would be more effective.
• Force the development team to address all technical challenges with unproven, upcoming technologies where no one in the team could obtain any experiences so far. The application should be based on MDSD, AOSD, Web 2.0. LINQ, WCF, Multicore systems, Semantic Web, REST - add your own favourites here. Tell the people in the team that they are allowed to use whatever tools, technologies, or languages they prefer. This way, the resulting heterogenous tool chain will help you decreasing productivity, while making developers think they got freedom of choice. Another, alternative approach is to force developers to uss inappropriate tools such as the C#-command line compiler, Notepad, Microsoft Word and Visio.
• At the same time don't give people any time for competence ramp-up. Motivate this by telling everyone that you consider "training on the job" the most effective way to learn new technologies. People not capable of learning on the job, are concept-addicted idiots who could be easily substituted by a bunch of chimpanzees typing on a notebook. This behavior will keep morale high. Note: I am refering to YOUR morale here.
• Architecture should be addressed inappropriately. In most cases it is sufficient to prescribe the exclusive usage of UML. As another option consider to develop dozens of different DSLs for even the smallest subdomains. Enjoy when engineers then try to come up with integrated solutions. Architecture documents should have no specific structure, never include all relevant parts, only motivate the "what" and never the rationale. Don't allow any domain modeling during start-up to make sure that stakeholders DO have a DIFFERENT understanding of the underlying domain (concepts). Never introduce architecture guidelines for crosscutting concerns such as fault management, security, or logging. Don't assign the responsibility for subsystems or such cross-cutting concerns to single individuals. This strategy supports mixing domain-specific functionality with infrastructural and non-functional issues which is a constant source of all kinds of errors (-; Applying patterns should be considered a tabu as all other kinds of best practices. To be even more effective, introduce anti-patterns and pretend they are patterns. Last but not least, don't prescribe any locations where to put all those documents. Recall your mantra of IT sadism "Searching is better than finding".
• Introduce re-use by copy-and-paste only. Argue that you are expecting developers to think in terms of open source and that from your point of view this implies a copy-and-paste philosophy. In this context, it would be a mistake to enforce any configuration management tooling. After a while, the application code will be flooded by re-used code fragements that - in case of luck - were changed in the meantime. As soon as an error pops up in only one of these fragments, all developers will feel the pain. Re-using binary components should be considered a bad idea as it will constrain your control. Moreover, enforce and motivate your team to develop everything from scratch. Don't trust any 3rd party middleware, tools, operating systems, but let developers develop these things. Using this approach, the project will focus more on infrastructure than on domain which supports your sabotage stategy.
• Make your system a SOA system. Come up with a great number of potential integration problems and distribute the command to solve all these issues with loose coupling, i.e. using a service-oriented architecture. Invite consultants from market analysts and other companies to tell your team about emerging business and technology directions. Your team will then fall into the SOA trap, forced to consider topics such as Governance, BPM, SOX, EAI, BAM, ESB, REST, WSDL, BI, SAP, etc. This recommendation is a guaranteed success factor as people tend to stop thinking whenever they discuss the SOA panacea. Maybe, they'll recognize that none of this technologies has any relevance for the project, but then it will be too late, anyway. And if the project fails, there are lots of project members you could easily assign the scape goat role. Here, the "I am the Wizard of Oz surrounded by evil demons and incompetent engineers" argument will convince senior management. Take a financial reward from management for opening their eyes and then leave the company. Now, it will be a clear SEP (somebody else's problem).

Note, that knowing these recommendations does not imply that I have personally seen or applied any of them. Nor do I claim that the list above is complete. You may tell me about your personal experiences by adding a comment to this blog. I am open to any feedback :-)

Danger Seekers - A day in the life of an anonymous software architect

• 08.00h: Scanning the e-mail for any recent project catastrophes.
• 08.15h: Asking the product manager and the requirements engineer what the customer means with "system should never fail but there is no extra budget for replication hardware and software".
• 09.00h: Meeting with subsystem developers, because they didn't follow the architecture design nor any guidelines. Told me that they understood every detail from the strategic architecture document I have conceived and written. However, they assume, I were not able to understand my own architecture which is why they tried to illustrate what they think I should have meant.
• 10.25h: Telephone conference with outsourcing site. They didn't know what interfaces to provide and what interfaces to use. Discussed the same issue over and over again with different people in the past but obviously they didn't share the information in the team.
• 12.00h: Having lunch with the project manager and a software process consultant for iteration planning. We are behind schedule and aleady exceeded the budget. Our table was surrounded by staff from other projects where they kept complaining about architects, senior management and the rest of the universe. What a motivating lunch.
• 13.15h: Talking to the testers. They are a little bit unhappy because some managers told them that they don't need to test as this would endanger the release date. According to this manager, the project got high quality developers who provide high-quality code. Thus, testing is overkill. Could raise their motivation after showing them the newest Dilbert cartoons.
• 13.35h: Talking to an external consultant. He is not familar with the domain, nor with the technologies. To be honest, I am not sure whether he got any expertise. Only recently he got his university degree. But senior management is relying on his statements as he is employed at one of those famous consulting companies. "They never make any mistakes", one of my senior managers mentioned.
• 14.30h: A middleware vendor is explaining why her product could solve all the project's problems. It can even solve problems we don't have. What a cool technology!
• 15.56h: One of the developers is calling me. He asks me to explain again why patterns are so important. He fears, applying patterns is reducing his creativity. Moreover, his productivity will also suffer as he must read all those pattern descriptions.
• 16.20h: The phone is ringing. It is our senior manager. He urgently needs some slides on the ROI of software architecture with some concrete numbers. He obviously knew that this presentation would be required for weeks, but one day before the meeting he is asking me for help. I must keep calm.
• 16.45h: Speaking with some developers. I just found out they have never built any Java EE application before. Our management thought, they are so smart that they could learn Java EE in a few days. That is what they call training on the job. I've decided to schedule some seminar days where I will provide them a Java EE hands-on-training.
• 17.20h: Some of the tests have failed. The product managers sent me some vague new customer requirements - all of them with highest priority. Software engineering drives me crazy.
• 18.20h: Reading the news. The new operating system will be postponed to next year. It is the goal to add some new features in the meantime and to further improve the quality. Going where no man has gone before, as it seems. A friend of mine working on the OS had told me some other background stories. Something about severe quality problems and the like. For heaven's sake, we are not alone!
• 19.45h: Drinking beer with some project members. Interesting to hear some war stories from earlier projects that failed. After several drinks my colleagues become very informative and open-minded about what they really think about the project. Should I quit tomorrow and search for a better job? No, that wouldn't make any difference.
• 23.30h: Reading some new technology stuff and one of the cool architecture web sites. This way I am learning a lot about new technologies. But, of course, the best technologies in the hands of some insane software developers or, even worse, managers can cause a lot of harm.
• 00.30h: Very exhausted now. It is time to go to bed.
• 06.00h: The alarm clock is ringing. What a nightmare. I had a dream of a meeting with managers who presented their favourite software architecture to me. Then I got the task to improve their architecture. Whenever I was done, a new meeting was arranged where they confronted me with yet another architecture and the whole pain started again. I could feel Sysyphos' pain.
• 08.00h: A new working day full of joy, honour and interesting conversations is waiting.

On Apples and Oranges

In almost all discussions about the maturity of software engineering, people refer to other disciplines claiming that software engineering is relatively immature. One of the arguments popping up all the time addresses the inability of software engineers to systematically map requirements to specifications. In addition, software components and their composition rather resemble primitive tools such as the ones used in the stone age according to those critical voices. Software enginering happens in an ad-hoc manner while other disciplines have learned how to apply a scientific formalism or a systematic engineering approach. What a shame to be a software developer. A point often made is the industrialization experienced in other domains such as hardware manufacturing where re-use is the norm and not the exception. It is my basic claim that most of these comparisons are more of the apples and oranges kind of style. So let me state some thoughts:

• While it is certainly true that industrial hardware production happens with a huge amount of re-use of standardized components, software engineering is more like hardware engineering. At design time hardware developers face exactly the same kind of problems when identifying requirements and mapping these to concrete circuit plans. This is the same mixture of craft, art and science software engineers rely on.
• Software systems experience a lot of more changes during design and implementation than their non-software cousins. Would you ever expect that shortly before a Boeing 767 is ready to be delivered someone will ask the engineers to change the body or engines. But that is exactly what happens in software engineering. Changing an airplane engine during flight? Impossible for aircraft engineers but expected from software systems. Add to this that customer requirements are much more unspecific, dynamic and unreliable in software development projects.
• Software must run in a lot of different contexts. Take operating systems such as Mac OS X, Linux, Vista as an example. They must support different kinds of hardware systems, different kinds of drivers, different kinds of user preferences. Such flexibility is not provided by any other product. For the same reason, large-scale software re-use such as the component or service market places we often dream of will remain dreams despite of standardization.
• Some people tend to make humorous comments about software engineering and rocket science. The problem is that in terms of complexity many software developent projects reveal a large degree of complexity. Software engineering sometimes IS like rocket science. Costomers and developers often ignore this complexity which is caused by operational and developmental qualities. The largest software projects are way larger than any comparable non-software projects. Keep in mind that the same holds for all the software tools and infrastructures.
• An incredible amount of bugs lurk in software systems due to a lack of quality in software developemt. True or false? Have you ever seen the error list of hardware systems such as that of new CPUs? Believe it or not, but I claim that the error rate in software systems is not higher - at least not significantly - than the one in comparable non-software systems. In addition, we should keep in mind testing is much more difficult in software due to the higher diversity of the software ecosystem.

I propose the following experiment: take a bunch of hardware developers who have become software engineers and see whether they are more productive and offer higher quality when developing software than their colleages without that background. From my viewpoint the immaturity of software engineering is a myth. Wrong statements repeated often will not become correct statements. I am wondering what other people think about this.

Contracts - continued

There are a lot of potential use cases for contracts:

• In distributed application environments such as SOA (to add one of the inevitable buzzwords here) the interfaces provided by services and components have to be described by contracts, because generator tools require these contract descriptions to generate proxies and stubs.
• In a container (no matter whether it is a lightweight container or a heavier one) contracts are required so that a container can handle component deployment in a proper way.
• Developers may follow the guidelines of contract-based programming as introduced by Bertrand Meyer in Eiffel by leveraging preconditions, postconditions or invariants for their abstract data types.
• All kinds of unit tests or integration tests require at least implicit knowledge of contracts as contract violations are one potential source of quality problems.
• Contracts are the means to implement and validate Service Level Agreements.
• In future scenarios of service compositions quality annotations available in contracts might help to determine if a particular workflow can meet quality requirements such as response times.
• Security feautures and measures re based on contracts that specify who is allowed to do what in a system.

But how can we describe contracts? This is an important question that has to be addressed in this context. Again, we should differentiate between implicit and explicit contracts.

An implicit contract basically says: here is the implementation and whatever it provides to its consumers or expects from its consumers is part of the contract. Implicit does not necessarily imply that there is no documentation of the contract. There might be comments in the source code that describe pre- and postconditions checked or guaranteed by the implementation. Alternatively, the contract might be documented in an API document.

An explicit contract is a contract that is expressed in a kind of contract DSL. A WSDL description is a typical example of an explicit contract description. Of course, the contract DSL can also be expressed with the means of the underlying programming language such as using annotations in Java or attributes in .NET.Another question in this context is whether contract violations are checked and whey they are checked. If a generator takes a contract DSL and generates implementation artifacts that are compliant with the contract, this helps with automating otherwise tedious and error-prone tasks. However, this is far from being sufficient. What about people taking and extending or changing generated code? Compile time checking of contracts by a compiler reveals the same issues. Runtime checking of contracts is the most secure approach while also causing potential performance penalties. Take security breaches such as buffer overflows as an example for the consuequences of not checking contracts. Or consider Java class loading which is checked at runtime. Another example for runtime contract checking is the .NET CLR security architecture where the security contracts may either be expressed programmatically or descriptively.

In architecture modelling languages such as UML you may apply language features like OCL to describe contracts. Obviously, in software architecture design all contracts must be described explicitely without prescribing how these contracts are eventually mapped to implementation artifacts. How do we determine the contracts of architectural entities? External entities we are integrating should already provide contracts. For all other constituents we should take functional and non-functional requirements as the base to defer contracts. For discussing design using CRC cards we could add contract information either on extra cards annotated to CRC cards or introduce a contract part at the bottom of the CRC card. With other words, contract-based programming should be an important discipline for software engineering in general.

What is (in) a Contract?

In component-based and service-oriented systems interfaces represent the core means for describing the boundary of the provider and consumer of a particular, semantically-related aggregation of functionality. In object-oriented systems these interfaces are typically message-based or method-based. Contracts help specifying the properties of such boundaries. Basically, they define constraints, properties, rules and other details that have to be considered when something is crossing the boundary. The most relevant part of the contract are rules defining which messages are permitted to be sent to a message-based interface or what methods might be called on a method-based interface. Obviously, everything returned from the provider (results or faults) to the consumer must also be expressed in terms of the contract. Constraints may define how messages are structured and what parts they contain. It is noteworthy that a contract does not just comprise syntactic sugar but may also specify semantics such as the order of message exchange or method invocations, pre- and postconditions, or even operational qualities like security or performance hehavior. For example, a contract may specify that calling a specific interface method only takes a specific amount of time. The order of method invocations could also be an ingredient of the contract such as the rule: "before invoking read(), you need to invoke open()". Same holds for messsage-based interface where a state machine might define the order and kind of message exchanges. In addition, a contract could contain infrastructural information. E.g.: Which kind of protocol may be used for accessing the interface? Or the location of the implementation artifact! A contract may also include relationships between interfaces such as the required interfaces a component needs or expects from other components.
There is another dimension of contracts. Contracts might be explicit or implicit. In the former case we need description languages to express the contract, while in the latter case contracts are either part of the infrastructure (such as a programming language) or implicitly hard-coded into application functionality.
In contracts we also may find different restrictions for different stakeholders. Think of a C++ class with private, protected and public sections. Depending on whether you subclass or use a class, you might experience more or less access restrictions.
In distributed systems contracts are ubiquitious. Think of EJB with its deployment contracts, component/container contracts, or client/component contracts and think of SOA platforms!
As we recognize, contracts are much more complex and important than we tend to believe.
Why - in the hell - are contracts relevant for softwae architecture? They are essential as they define various essential aspects when dealing with the boundaries of a particular entity such as a service or component. Contracts define the interactions that may take place between the service or components and their clients as well as the qualities and constraints associated with these interactions. Only if there are formal contract descriptions, tools can help with the (automatic) composition of artifacts (e.g., MDSD) as well as with the validation of architectures and implementations.
If you are going to model a new system, keep in mind to describe the contracts of the interfaces introduced. Otherwise, developers or architects won't be able to understand how to access this interface in a proper way. A surprisingly large amount of problems in software engineering is caused by missing or incomplete or inconsistent contracts. Hardware developers won't ever build any system without knowing the contracts of the system constituents a.k.a. specifications. Why should we as software engineers? Don't believe in statements like "contracts are for pussies". Only contract-based development leads to high-quality architectures and implementations.

The Complexity Of Interface Extraction

From day 1 when remoting middleware appeared on the horizon, one of the main challenges for application developers consisted of extracting interfaces from application functionality to allow decomposing applications into a network of objects. The same issue is still relevant to appease the gods of Service-Oriented-Architecture. Unfortunately, many developers have been fooled by middleware proponents constantly praising the transparency caused by middleware. This transparency makes developers believe building a distributed system is exactly the same like building a non-distributed one. Just take your application classes and make them service interfaces. That's all what you need when migrating from a traditional to a distributed universe. Believe this and your project is doomed to fail. What, the hell, is the right approach to extract interfaces for a distributed software system and what are common pitfalls? Want some examples? Here we go:

• Granularity: Suppose, you have developed a Java class that implements a graph-like structure such a s a tree. You got classes such as Tree and Node. As you are a smart programmer, you already have extracted interfaces such ITree, INode instead of bloating the class interfaces with oceans of methods. When you navigate to a subtree, a method such as navigate(theChildINeed)returns the right child tree. If you now naively convert all of your classes, i.e. the aforementioned interfaces, into remote interfaces, things will rapidly get messy. Each invocation of navigate will now return another remote object. All of a sudden, network traffic will explode. You can't simply develop remote classes as POJOs/POJIs or PONOs/PONIs. For the same reason, it does not make sense to transfer fine-grained data in service invocations. Sequences such as node.setName("Michael"), node.setStreet("Marketstreet"); node.setCity("SFO"); are DONOTs in any SOA or Remoting context. Firstly, they imply a lot of traffic. Secondly, the calls are related and might force the server to keep session state across method calls.
• Impedance Mismatch: In heterogeneous networked environments (such as SOA) you'll always face the problem of mapping between the object model of your language and the object model of the remotung technology, and vice versa. Whenever you try to transmit a HashTable between a .NET application and a Java EE application over a SOAP connection, things will turn out to be like hell. This does not work without significant work arounds. As a relational database programmer or O/R victim you will now what I am talking about. Most remoting solutions will only provide the least common denominator approach in their object model. As soon as you are trying to integrate the richness of your preferred programming language into that middleware, you are calling for trouble.
• Operational Qualities: Add to all this the complexity of operational qualities such as security or performance. Basically, you are now forced to combine the security infrastructure of your application with the security infrastructure of the middleware. And you have to take performance of your application objects into account, but also the costs of communication. And you have to integrate distributied transaction monitors with local transaction APIs. And ...
• Contracts: An interface is not just a matter of syntax. Nor is it just a set of signatures. It is also about semantics such as preconditions and postconditions. A finite state automaton could specify in which order which methods are allowed to be called. The contract might also deal with local policies and define the protocols with which clients can bind to the service. Note, that this is much more than what you would expect from an interface in a non-networked environment.
• Architectural Balance: Let us suppose, we have magically solved all these issues. How should we compose our application to remote components or services. We might end up in an ocean of components. Or we might end up in an ocean of service interfaces. Architectural patterns such as Layers or Extension Interface are helpful in this context. I must confess that no rule of thumb exists for this architectual structuring. It really depends on the concrete problem context how to address this issue. One really helpful way is to think in terms of roles and responsibilities.

Basically, what all this means is that when you start to develop your distributed system from scratch, you need to thoroughly consider the issues explained above. However, when you are in the situation to integrate legacy code, things are much more complex: you'll need a whole bunch of refactoring activities to reorganize your application in such a way that is will consist of classes and interfaces that easily fit into the middleware ecosystem. In the best case, it might be sufficient to add some wrapper {facade} objects. However, in the case of bad luck you will even need re-engineering efforts.

I tried to shed some light on the complexities of application integration and provisioning of remote interfaces. What appeared so obvious and simple in the beginning, in fact is one of the most complex tasks engineers have to cope with. Efficiently building efficient distributed systems implies thorough treatment of all these issues. Believe me, prevention is better than cure!

Oh, Happy Day !

Suppose, you are an architect of a new web portal framework. For this purpose, you came up with a prioritized list of requirements and use cases. Now, you are starting to design the framework. First use case of consideration is login of a user who should then redirected to his personal portal. But now you are starting to consider more details:

• what if the user provides the wrong credentials?
• what if you encounter memory outeage when instantiating new objects?
• what if the communication breaks down?
• what if you can't read the user's preferences from the file system?
• what if data already has been locked by someone else?
• what if the HTTP request does provide a user agent you don't support?
• ...

You are a smart architect who never forgets any detail, a perfectionist in terms of software engineering. Thus, you are starting to cover all these exceptions in your architectural design. A few use cases later you are lost in design erosion, trapped in a web of entities and relationships.

On one hand, it is essential to consider all of these exceptional cases and list them in the use cases. Most of the exceptional issues are cross-cutting, re-apperaring in different places of your system. On the other hand, if you start to consider them from day 1, you will be doomed to fail in any non-trivial development project.

The recommendation in architectural design thus reads as follows:

• In the beginning only start with the main flows (the happy day scenarios). Those should be the ones driving and guiding your core architectural design.
• In addition, you should come up with project-wide strategies how to cope with exceptional cases. Strategic documentation should be made available and mandatory for all developers with description how to deal with exceptions, transactions, synchronization issues, and the like.
• The exceptional cases will together with non-functional requirements help extending or changing the domain-specific architecture and infrastructural parts.

Of course, it would be fatal to rely on happy day scenarios only. How often, did I encounter finite state machines with all "happy day nodes and edges" where no one thought about the error states, because errors are for pussies. It is not the matter of ignoring exceptional cases. It is more on doing happy day scenarios first to obtain a stable strategic base-line and then thoroughly integrating the exceptional cases.

From now on, you should always think of "happy days" when modeling your architecture :-)

Brothers in arms - SOA and Product Line Engineering

How would you engineer a SOA based system? First of all, you'd need to define all the composite applications and processes which become part of your infrastructure. Then, you are able to defer common services. Obviously, a separate team should be in charge of service engineering and maintain all services as well as their implementations in central repositories. For example, Amazon organizes its development this way. Developers of applications then build upon these services as well as other common parts. They give feedback to the service developers so that existing services might be reorganized, changed, extended or supplemented. Consequently, we introduce a separation of service engineering and application engineering.
Now, compare all this to Software Product Line Engineering (SPLE, see my special posting on that issue):

• Services and other common parts (such as legacy applications or platforms) are core assets
• Applications (composite applications and business processes) denote the members of a product line
• Service engineering resembles domain engineering and application engineering is identical.

Sooner or later you'll reconginze that SOA engineering is best organized as a SPLE process.

Scoping might be a little more difficult but you should read my last posting how we could cope with this challenge. You might also consider to provide a generative architecture as described in another recent posting as well as in what I wrote on ULS.

From my viewpoint the only conclusion to draw is that SPLE and SOA engineering should be considered as two sides of the same coin.

The Scope of Product Line Engineering

In Software Product Line Engineering the first activity comprises scoping which basically means to anticipate all upcoming products that should be in the scope. I was asked by another engineer whether this implies that we need to know all applications in advance which will be part of the product line and if yes, what if the number of applications would be VERY large. So, what does scoping actually require? Scoping does not mean that you really know everything in the beginning. Otherwise it would be very easy to define the commonalities and variabilities. As always, architects have to deal with vagueness and uncertainty in this context. My understandimg of scoping is as follows. We need to have an idea of the products in the program family we are going to build. Thus, the first step would be to define use cases / scenarios that are typically expected by an application in the program family. If use cases differ or if they appear in some applications but not in others they will help discovering variabilities. If they reveal common (generic) parts or are identical for all applications they will reveal the commonalities. But how do we obtain those use cases? Either, the domain itself inherently defines such use cases. Take ATMs as an example. Or, we could take a representative sample of applications from which we derive the use cases.
This brings me to another important point. I was also asked whether product lines are always a simple partitioning in domain engineering and application engineering. No, they are not. Take car manufacturing as an example which many consider as the prototype of Product Line Enginering. If a car type is manufactured as part of a product family, it will contain common parts all members of the family share as well as variabilities such as different colors, chassis, and so forth. Thus, we have domain engineering where we produce the common core assets and application engineering where we build and customize the individual cars. So far, so good. If we think this a little bit further, some of the assets we use will themselves be parts of other product lines. Maybe, the supplier of the entertainment and navigation system will also leverage product line engineering. Or maybe, the car manufacturer will provide other assembly lines where core parts such as the doors of the car are produced in a product line approach. In other words, a product line can be a combination of different product lines intertwined with each other in a pipeline.
But it is also possible to find the other way round? Suppose, we are going to build a generic web shop application for different customers. We are setting up two separate product lines, one for SOHO and another one for enterprise customers. After a while we find that both product lines can be based upon an additional product line where the common parts for the SOHO as well as for the enterprise web shop are developed. In the SOHO and enterprise product lines we are then just adding the deltas. Obviously, such a setting only makes sense if the SOHO and Enterprise Web shop solutions reveal a lot of differences. Otherwise we would better establish one single product line and handle the differences of both shop variants by variabilities. By the way, finding the appropriate approach is also an important outcome of scoping.
In one of my next postings I will address the issue of why SOA engineering can be considered Product Line Engineering. So, stay tuned!

Generative Software Architecture

When designing really huge and dynamic software and system infrastructures such as the Internet or cellular networks it is very unlikely to come up with a complete structural design in the beginning. You cannot simply draw some architectural viewpoints that cover those systems as a whole. The reason for this is straightforward: such systems are designed to grow all the time and in unanticipated ways. On one hand these systems represent product line architectures. On the other hand they cannot be treated as such in terms of engineering because they have virtually infinite scope. But how can we successfully design and implement systems of that scale or complexity? You might have encountered the term Ultra Large Scale Systems (ULS) in the past months which exactly deals with this problem. As I already explained in another posting a ULS is a system of systems as Linda Northrop and her peers pointed out. An appropriate metaphor often mentioned in this context is building cities. So far, we have built software systems that resemble building single houses or city blocks. In the next step, we are going to build complete cities and even countries. Complexity represents one of the core problems in developing such large systems, because inherent complexity won't disappear. Hence a possible solution must master complexity instead of getting rid of it. The key strategy as in many other disciplines is emergent behavior. In other words: How can we partition our large problem into a set of simple problems and ways to combine these problems to an overall solution? In the world of software architecture this could basically mean different things depending on the scale we encounter:

• we could define the basic infrastructure and let complex parts be integrated into this infrastructure
• we could define a set of rules to dynamically build and evolve such an infrastructure as well decribe integration constraints for parts
  interacting with and over the infrastructure or
• we could combine the aforementioned approaches by defining a small set of rules and smaller predefined infrastructural parts

Those rules are not only axiomatic rules as in a language grammar, but also define constraints and requirements for the parts that are integrated into the infrastructure. In such an approach, the non-functional properties must be derived from those rules and constraints.
They cannot be evaluated in advance for the reasons already introduced. Obviuosly, to cope with future challenges, such a system must also provide extension rules and hooks to even adapt the infrastructure to changing requirements.If you think about it: Why could we come up with a technology such as the Internet with all of its complexity? Internet technologies already
fulfil the properties mentioned above. The Internet consists of smaller infrastructures, e.g., Internet backbones. Technologies such as IP, DHCP, or DNS define a rather simple (but not simplistic!) rule set for extending and evolving the infrastructure as well as for integrating servers and clients into the infrastructure. All those very simple technologies are combined to a highly evolvable and dynamic, large-scale system which is open for extension. Basically, HTTP and the Web are only one of those extensions. As a real-life example take the brain which consists of rather simple constituents and combinations between those constituents, but reveals the most complex emergent behavior we have ever dealt with. From my viewpoint, we should direct our research activities to those generative architectures, especially when dealing with large-scale software systems. Emergent behavior is the key to master complexity in such systems.

Non-functional requirements

Many software projects don't fail because of functional aspects. In general, experts know very well how to build a system in their domain. The number one technical reason for failure are non-functional requirements. Issues such as reliability, availability or performance have to be addressed in addition to all the domain logic. Unfortunately, many domain experts are not necessarily experts in these non-functional topics. And the efforts to get consistency or availability right can easily outnumber those for functionality. Ask what the CTO of Amazon thinks about this challenge. Thus, my first rule could be described as "always define key developers for the requirements with the highest risks and relevance". These key developers are in charge of teams that explicitely address one or a small number of these issues. The size of these teams and the number of teams depends on your overall project size.
Another important point in this context is desribed as requirements elicitation and requirements engineering. It is interesting how unusable many requirement specifications are. Statements such as "System should provide flexibility" are worthless. What exactly do these stakeholders mean by flexibility? Which parts of the systems should be flexible and when? Of course, you are also doomed to fail if you get a bunch of requirements with no priorization. The same application would look completely different in case security is the number one requirement compared to the same application when performance had the highest priority. So, my second rule simply says "Architects and key developers are in charge to clarify all requirements including their priorities. For each pair of requirements, it should be clear which one has precedence in concrete design decisions". As many requirements might contradict each other, this is inevitable to survive. If your management does not agree to detail requirements or prioritize them, either refuse to be in the project or, more realistic, document this fact and possible implications so that all stakeholders know they might encounter project failure. Unfortunately, non-functional requirements are incredibly diverse. Performance is totally different to security. Even security might mean a lot of various things such as authentication, authorization, confidentiality, name it. Thus, my rule 3 is: "Detail all non-functional requirements in the beginning of the project, maybe establish checklists, and use these information to clarify requirements".
Developmental requirements such as flexibility (e.g., changeability) have to be addressed early only if you are going to design a product line where a Commonality/Variability analysis is of foremost importance. If this is not your primary setting, developmental issues should be treated as a tactical refinement of your software architecture. Thus, the strategic base line architecture is primary influenced by the domain logic and by operational requirements. The integration of developmental issues is a further refinement activity. Considering issues such as extensibility in early phases typically leads to generic and inperformant architectures with instantiations of patterns such as Observer, Interceptor or Strategy. In addition, your job as an architect tends to continue forever in such environments. This is where I'd like to add my fourth rule "Domain logic dominates infrastructure; infrastructure dominates operational requirements; operational requirements dominate developmental requirements". Of course, it is also important to specify when operational requirements appear in the running system (run-time, install time, compile time, ...) and who is in charge to inject them (operator, user, developer, ...).
As of the huge diversity of requirements it is not easy to come up with general rules of thumb for all of them. Some observations might be interesting in tbis context: A given requirement might be invasive (i.e., cross cutting through other parts of your system) or non-invasive (i.e., it is sufficient to add some additional infrastructural components to the architecture). The same requirement might be invasive for one system (security in all its aspects must be supported everywhere) or non-invasive (we just can add a firewall and don't have to care about security at all). A requirement might me measurable such as the response time to a method request or not (such as removability). Requirements that can be measured can be covered by testing, all other requirements must be covered by architecture and code analysis. Last, but not least, my rule 5: "Classify your requirements and provide strategies how to deal with them in architecture design activities".

Of course the overall goal of software engineering must be to

• Let engineers focus on their domain by strong separation of domain aspects from other aspects.
• Automate where possible and useful.

There are several efforts to support these goals such as Model-Driven-Software Development, Product Line Engineering, Aspect-Oriented Software Development, SOA.

One way could be to provide DSLs for the domain, additional domains for the most important non-functional requirements, a technology that easily lets project teams integrate these DSLs under one umbrella (e.g., by leveraging AOSD techniques, Semantic Integration). A Product Line Engineering approach could define core assets which become then part of platforms or frameworks. With SOA, platform dependencies could be overcome. Is this just Science Fiction or a reasonable future vision? Let's work on this :-;