- 08.00h: Scanning the e-mail for any recent project catastrophes.
- 08.15h: Asking the product manager and the requirements engineer what the customer means with "system should never fail but there is no extra budget for replication hardware and software".
- 09.00h: Meeting with subsystem developers, because they didn't follow the architecture design nor any guidelines. Told me that they understood every detail from the strategic architecture document I have conceived and written. However, they assume, I were not able to understand my own architecture which is why they tried to illustrate what they think I should have meant.
- 10.25h: Telephone conference with outsourcing site. They didn't know what interfaces to provide and what interfaces to use. Discussed the same issue over and over again with different people in the past but obviously they didn't share the information in the team.
- 12.00h: Having lunch with the project manager and a software process consultant for iteration planning. We are behind schedule and aleady exceeded the budget. Our table was surrounded by staff from other projects where they kept complaining about architects, senior management and the rest of the universe. What a motivating lunch.
- 13.15h: Talking to the testers. They are a little bit unhappy because some managers told them that they don't need to test as this would endanger the release date. According to this manager, the project got high quality developers who provide high-quality code. Thus, testing is overkill. Could raise their motivation after showing them the newest Dilbert cartoons.
- 13.35h: Talking to an external consultant. He is not familar with the domain, nor with the technologies. To be honest, I am not sure whether he got any expertise. Only recently he got his university degree. But senior management is relying on his statements as he is employed at one of those famous consulting companies. "They never make any mistakes", one of my senior managers mentioned.
- 14.30h: A middleware vendor is explaining why her product could solve all the project's problems. It can even solve problems we don't have. What a cool technology!
- 15.56h: One of the developers is calling me. He asks me to explain again why patterns are so important. He fears, applying patterns is reducing his creativity. Moreover, his productivity will also suffer as he must read all those pattern descriptions.
- 16.20h: The phone is ringing. It is our senior manager. He urgently needs some slides on the ROI of software architecture with some concrete numbers. He obviously knew that this presentation would be required for weeks, but one day before the meeting he is asking me for help. I must keep calm.
- 16.45h: Speaking with some developers. I just found out they have never built any Java EE application before. Our management thought, they are so smart that they could learn Java EE in a few days. That is what they call training on the job. I've decided to schedule some seminar days where I will provide them a Java EE hands-on-training.
- 17.20h: Some of the tests have failed. The product managers sent me some vague new customer requirements - all of them with highest priority. Software engineering drives me crazy.
- 18.20h: Reading the news. The new operating system will be postponed to next year. It is the goal to add some new features in the meantime and to further improve the quality. Going where no man has gone before, as it seems. A friend of mine working on the OS had told me some other background stories. Something about severe quality problems and the like. For heaven's sake, we are not alone!
- 19.45h: Drinking beer with some project members. Interesting to hear some war stories from earlier projects that failed. After several drinks my colleagues become very informative and open-minded about what they really think about the project. Should I quit tomorrow and search for a better job? No, that wouldn't make any difference.
- 23.30h: Reading some new technology stuff and one of the cool architecture web sites. This way I am learning a lot about new technologies. But, of course, the best technologies in the hands of some insane software developers or, even worse, managers can cause a lot of harm.
- 00.30h: Very exhausted now. It is time to go to bed.
- 06.00h: The alarm clock is ringing. What a nightmare. I had a dream of a meeting with managers who presented their favourite software architecture to me. Then I got the task to improve their architecture. Whenever I was done, a new meeting was arranged where they confronted me with yet another architecture and the whole pain started again. I could feel Sysyphos' pain.
- 08.00h: A new working day full of joy, honour and interesting conversations is waiting.
If you are a software engineer: DON'T PANIC! This blog is my place to beam thoughts on the universe of Artificial Intelligence and Software Architecture right to your screen. On my infinite mission to boldly go where (almost) no one has gone before I will provide in-depth coverage of architectural and AI topics, personal opinions, humor, philosophical discussions, interesting news and technology evaluations. (c) Prof. Dr. Michael Stal
Thursday, April 26, 2007
Danger Seekers - A day in the life of an anonymous software architect
Monday, April 23, 2007
On Apples and Oranges
- While it is certainly true that industrial hardware production happens with a huge amount of re-use of standardized components, software engineering is more like hardware engineering. At design time hardware developers face exactly the same kind of problems when identifying requirements and mapping these to concrete circuit plans. This is the same mixture of craft, art and science software engineers rely on.
- Software systems experience a lot of more changes during design and implementation than their non-software cousins. Would you ever expect that shortly before a Boeing 767 is ready to be delivered someone will ask the engineers to change the body or engines. But that is exactly what happens in software engineering. Changing an airplane engine during flight? Impossible for aircraft engineers but expected from software systems. Add to this that customer requirements are much more unspecific, dynamic and unreliable in software development projects.
- Software must run in a lot of different contexts. Take operating systems such as Mac OS X, Linux, Vista as an example. They must support different kinds of hardware systems, different kinds of drivers, different kinds of user preferences. Such flexibility is not provided by any other product. For the same reason, large-scale software re-use such as the component or service market places we often dream of will remain dreams despite of standardization.
- Some people tend to make humorous comments about software engineering and rocket science. The problem is that in terms of complexity many software developent projects reveal a large degree of complexity. Software engineering sometimes IS like rocket science. Costomers and developers often ignore this complexity which is caused by operational and developmental qualities. The largest software projects are way larger than any comparable non-software projects. Keep in mind that the same holds for all the software tools and infrastructures.
- An incredible amount of bugs lurk in software systems due to a lack of quality in software developemt. True or false? Have you ever seen the error list of hardware systems such as that of new CPUs? Believe it or not, but I claim that the error rate in software systems is not higher - at least not significantly - than the one in comparable non-software systems. In addition, we should keep in mind testing is much more difficult in software due to the higher diversity of the software ecosystem.
I propose the following experiment: take a bunch of hardware developers who have become software engineers and see whether they are more productive and offer higher quality when developing software than their colleages without that background. From my viewpoint the immaturity of software engineering is a myth. Wrong statements repeated often will not become correct statements. I am wondering what other people think about this.
Saturday, April 21, 2007
Contracts - continued
- In distributed application environments such as SOA (to add one of the inevitable buzzwords here) the interfaces provided by services and components have to be described by contracts, because generator tools require these contract descriptions to generate proxies and stubs.
- In a container (no matter whether it is a lightweight container or a heavier one) contracts are required so that a container can handle component deployment in a proper way.
- Developers may follow the guidelines of contract-based programming as introduced by Bertrand Meyer in Eiffel by leveraging preconditions, postconditions or invariants for their abstract data types.
- All kinds of unit tests or integration tests require at least implicit knowledge of contracts as contract violations are one potential source of quality problems.
- Contracts are the means to implement and validate Service Level Agreements.
- In future scenarios of service compositions quality annotations available in contracts might help to determine if a particular workflow can meet quality requirements such as response times.
- Security feautures and measures re based on contracts that specify who is allowed to do what in a system.
But how can we describe contracts? This is an important question that has to be addressed in this context. Again, we should differentiate between implicit and explicit contracts.
An implicit contract basically says: here is the implementation and whatever it provides to its consumers or expects from its consumers is part of the contract. Implicit does not necessarily imply that there is no documentation of the contract. There might be comments in the source code that describe pre- and postconditions checked or guaranteed by the implementation. Alternatively, the contract might be documented in an API document.
An explicit contract is a contract that is expressed in a kind of contract DSL. A WSDL description is a typical example of an explicit contract description. Of course, the contract DSL can also be expressed with the means of the underlying programming language such as using annotations in Java or attributes in .NET.Another question in this context is whether contract violations are checked and whey they are checked. If a generator takes a contract DSL and generates implementation artifacts that are compliant with the contract, this helps with automating otherwise tedious and error-prone tasks. However, this is far from being sufficient. What about people taking and extending or changing generated code? Compile time checking of contracts by a compiler reveals the same issues. Runtime checking of contracts is the most secure approach while also causing potential performance penalties. Take security breaches such as buffer overflows as an example for the consuequences of not checking contracts. Or consider Java class loading which is checked at runtime. Another example for runtime contract checking is the .NET CLR security architecture where the security contracts may either be expressed programmatically or descriptively.
In architecture modelling languages such as UML you may apply language features like OCL to describe contracts. Obviously, in software architecture design all contracts must be described explicitely without prescribing how these contracts are eventually mapped to implementation artifacts. How do we determine the contracts of architectural entities? External entities we are integrating should already provide contracts. For all other constituents we should take functional and non-functional requirements as the base to defer contracts. For discussing design using CRC cards we could add contract information either on extra cards annotated to CRC cards or introduce a contract part at the bottom of the CRC card. With other words, contract-based programming should be an important discipline for software engineering in general.
Friday, April 20, 2007
What is (in) a Contract?
There is another dimension of contracts. Contracts might be explicit or implicit. In the former case we need description languages to express the contract, while in the latter case contracts are either part of the infrastructure (such as a programming language) or implicitly hard-coded into application functionality.
In contracts we also may find different restrictions for different stakeholders. Think of a C++ class with private, protected and public sections. Depending on whether you subclass or use a class, you might experience more or less access restrictions.
In distributed systems contracts are ubiquitious. Think of EJB with its deployment contracts, component/container contracts, or client/component contracts and think of SOA platforms!
As we recognize, contracts are much more complex and important than we tend to believe.
Why - in the hell - are contracts relevant for softwae architecture? They are essential as they define various essential aspects when dealing with the boundaries of a particular entity such as a service or component. Contracts define the interactions that may take place between the service or components and their clients as well as the qualities and constraints associated with these interactions. Only if there are formal contract descriptions, tools can help with the (automatic) composition of artifacts (e.g., MDSD) as well as with the validation of architectures and implementations.
If you are going to model a new system, keep in mind to describe the contracts of the interfaces introduced. Otherwise, developers or architects won't be able to understand how to access this interface in a proper way. A surprisingly large amount of problems in software engineering is caused by missing or incomplete or inconsistent contracts. Hardware developers won't ever build any system without knowing the contracts of the system constituents a.k.a. specifications. Why should we as software engineers? Don't believe in statements like "contracts are for pussies". Only contract-based development leads to high-quality architectures and implementations.
Friday, April 13, 2007
The Complexity Of Interface Extraction
- Granularity: Suppose, you have developed a Java class that implements a graph-like structure such a s a tree. You got classes such as Tree and Node. As you are a smart programmer, you already have extracted interfaces such ITree, INode instead of bloating the class interfaces with oceans of methods. When you navigate to a subtree, a method such as navigate(theChildINeed)returns the right child tree. If you now naively convert all of your classes, i.e. the aforementioned interfaces, into remote interfaces, things will rapidly get messy. Each invocation of navigate will now return another remote object. All of a sudden, network traffic will explode. You can't simply develop remote classes as POJOs/POJIs or PONOs/PONIs. For the same reason, it does not make sense to transfer fine-grained data in service invocations. Sequences such as node.setName("Michael"), node.setStreet("Marketstreet"); node.setCity("SFO"); are DONOTs in any SOA or Remoting context. Firstly, they imply a lot of traffic. Secondly, the calls are related and might force the server to keep session state across method calls.
- Impedance Mismatch: In heterogeneous networked environments (such as SOA) you'll always face the problem of mapping between the object model of your language and the object model of the remotung technology, and vice versa. Whenever you try to transmit a HashTable between a .NET application and a Java EE application over a SOAP connection, things will turn out to be like hell. This does not work without significant work arounds. As a relational database programmer or O/R victim you will now what I am talking about. Most remoting solutions will only provide the least common denominator approach in their object model. As soon as you are trying to integrate the richness of your preferred programming language into that middleware, you are calling for trouble.
- Operational Qualities: Add to all this the complexity of operational qualities such as security or performance. Basically, you are now forced to combine the security infrastructure of your application with the security infrastructure of the middleware. And you have to take performance of your application objects into account, but also the costs of communication. And you have to integrate distributied transaction monitors with local transaction APIs. And ...
- Contracts: An interface is not just a matter of syntax. Nor is it just a set of signatures. It is also about semantics such as preconditions and postconditions. A finite state automaton could specify in which order which methods are allowed to be called. The contract might also deal with local policies and define the protocols with which clients can bind to the service. Note, that this is much more than what you would expect from an interface in a non-networked environment.
- Architectural Balance: Let us suppose, we have magically solved all these issues. How should we compose our application to remote components or services. We might end up in an ocean of components. Or we might end up in an ocean of service interfaces. Architectural patterns such as Layers or Extension Interface are helpful in this context. I must confess that no rule of thumb exists for this architectual structuring. It really depends on the concrete problem context how to address this issue. One really helpful way is to think in terms of roles and responsibilities.
Basically, what all this means is that when you start to develop your distributed system from scratch, you need to thoroughly consider the issues explained above. However, when you are in the situation to integrate legacy code, things are much more complex: you'll need a whole bunch of refactoring activities to reorganize your application in such a way that is will consist of classes and interfaces that easily fit into the middleware ecosystem. In the best case, it might be sufficient to add some wrapper {facade} objects. However, in the case of bad luck you will even need re-engineering efforts.
I tried to shed some light on the complexities of application integration and provisioning of remote interfaces. What appeared so obvious and simple in the beginning, in fact is one of the most complex tasks engineers have to cope with. Efficiently building efficient distributed systems implies thorough treatment of all these issues. Believe me, prevention is better than cure!
Wednesday, April 11, 2007
Oh, Happy Day !
- what if the user provides the wrong credentials?
- what if you encounter memory outeage when instantiating new objects?
- what if the communication breaks down?
- what if you can't read the user's preferences from the file system?
- what if data already has been locked by someone else?
- what if the HTTP request does provide a user agent you don't support?
- ...
You are a smart architect who never forgets any detail, a perfectionist in terms of software engineering. Thus, you are starting to cover all these exceptions in your architectural design. A few use cases later you are lost in design erosion, trapped in a web of entities and relationships.
On one hand, it is essential to consider all of these exceptional cases and list them in the use cases. Most of the exceptional issues are cross-cutting, re-apperaring in different places of your system. On the other hand, if you start to consider them from day 1, you will be doomed to fail in any non-trivial development project.
The recommendation in architectural design thus reads as follows:
- In the beginning only start with the main flows (the happy day scenarios). Those should be the ones driving and guiding your core architectural design.
- In addition, you should come up with project-wide strategies how to cope with exceptional cases. Strategic documentation should be made available and mandatory for all developers with description how to deal with exceptions, transactions, synchronization issues, and the like.
- The exceptional cases will together with non-functional requirements help extending or changing the domain-specific architecture and infrastructural parts.
Of course, it would be fatal to rely on happy day scenarios only. How often, did I encounter finite state machines with all "happy day nodes and edges" where no one thought about the error states, because errors are for pussies. It is not the matter of ignoring exceptional cases. It is more on doing happy day scenarios first to obtain a stable strategic base-line and then thoroughly integrating the exceptional cases.
From now on, you should always think of "happy days" when modeling your architecture :-)
Monday, April 09, 2007
Brothers in arms - SOA and Product Line Engineering
Now, compare all this to Software Product Line Engineering (SPLE, see my special posting on that issue):
- Services and other common parts (such as legacy applications or platforms) are core assets
- Applications (composite applications and business processes) denote the members of a product line
- Service engineering resembles domain engineering and application engineering is identical.
Sooner or later you'll reconginze that SOA engineering is best organized as a SPLE process.
Scoping might be a little more difficult but you should read my last posting how we could cope with this challenge. You might also consider to provide a generative architecture as described in another recent posting as well as in what I wrote on ULS.
From my viewpoint the only conclusion to draw is that SPLE and SOA engineering should be considered as two sides of the same coin.
The Scope of Product Line Engineering
This brings me to another important point. I was also asked whether product lines are always a simple partitioning in domain engineering and application engineering. No, they are not. Take car manufacturing as an example which many consider as the prototype of Product Line Enginering. If a car type is manufactured as part of a product family, it will contain common parts all members of the family share as well as variabilities such as different colors, chassis, and so forth. Thus, we have domain engineering where we produce the common core assets and application engineering where we build and customize the individual cars. So far, so good. If we think this a little bit further, some of the assets we use will themselves be parts of other product lines. Maybe, the supplier of the entertainment and navigation system will also leverage product line engineering. Or maybe, the car manufacturer will provide other assembly lines where core parts such as the doors of the car are produced in a product line approach. In other words, a product line can be a combination of different product lines intertwined with each other in a pipeline.
But it is also possible to find the other way round? Suppose, we are going to build a generic web shop application for different customers. We are setting up two separate product lines, one for SOHO and another one for enterprise customers. After a while we find that both product lines can be based upon an additional product line where the common parts for the SOHO as well as for the enterprise web shop are developed. In the SOHO and enterprise product lines we are then just adding the deltas. Obviously, such a setting only makes sense if the SOHO and Enterprise Web shop solutions reveal a lot of differences. Otherwise we would better establish one single product line and handle the differences of both shop variants by variabilities. By the way, finding the appropriate approach is also an important outcome of scoping.
In one of my next postings I will address the issue of why SOA engineering can be considered Product Line Engineering. So, stay tuned!
Sunday, April 08, 2007
Generative Software Architecture
- we could define the basic infrastructure and let complex parts be integrated into this infrastructure
- we could define a set of rules to dynamically build and evolve such an infrastructure as well decribe integration constraints for parts
interacting with and over the infrastructure or - we could combine the aforementioned approaches by defining a small set of rules and smaller predefined infrastructural parts
Those rules are not only axiomatic rules as in a language grammar, but also define constraints and requirements for the parts that are integrated into the infrastructure. In such an approach, the non-functional properties must be derived from those rules and constraints.
They cannot be evaluated in advance for the reasons already introduced. Obviuosly, to cope with future challenges, such a system must also provide extension rules and hooks to even adapt the infrastructure to changing requirements.If you think about it: Why could we come up with a technology such as the Internet with all of its complexity? Internet technologies already
fulfil the properties mentioned above. The Internet consists of smaller infrastructures, e.g., Internet backbones. Technologies such as IP, DHCP, or DNS define a rather simple (but not simplistic!) rule set for extending and evolving the infrastructure as well as for integrating servers and clients into the infrastructure. All those very simple technologies are combined to a highly evolvable and dynamic, large-scale system which is open for extension. Basically, HTTP and the Web are only one of those extensions. As a real-life example take the brain which consists of rather simple constituents and combinations between those constituents, but reveals the most complex emergent behavior we have ever dealt with. From my viewpoint, we should direct our research activities to those generative architectures, especially when dealing with large-scale software systems. Emergent behavior is the key to master complexity in such systems.
Saturday, April 07, 2007
Non-functional requirements
Another important point in this context is desribed as requirements elicitation and requirements engineering. It is interesting how unusable many requirement specifications are. Statements such as "System should provide flexibility" are worthless. What exactly do these stakeholders mean by flexibility? Which parts of the systems should be flexible and when? Of course, you are also doomed to fail if you get a bunch of requirements with no priorization. The same application would look completely different in case security is the number one requirement compared to the same application when performance had the highest priority. So, my second rule simply says "Architects and key developers are in charge to clarify all requirements including their priorities. For each pair of requirements, it should be clear which one has precedence in concrete design decisions". As many requirements might contradict each other, this is inevitable to survive. If your management does not agree to detail requirements or prioritize them, either refuse to be in the project or, more realistic, document this fact and possible implications so that all stakeholders know they might encounter project failure. Unfortunately, non-functional requirements are incredibly diverse. Performance is totally different to security. Even security might mean a lot of various things such as authentication, authorization, confidentiality, name it. Thus, my rule 3 is: "Detail all non-functional requirements in the beginning of the project, maybe establish checklists, and use these information to clarify requirements".
Developmental requirements such as flexibility (e.g., changeability) have to be addressed early only if you are going to design a product line where a Commonality/Variability analysis is of foremost importance. If this is not your primary setting, developmental issues should be treated as a tactical refinement of your software architecture. Thus, the strategic base line architecture is primary influenced by the domain logic and by operational requirements. The integration of developmental issues is a further refinement activity. Considering issues such as extensibility in early phases typically leads to generic and inperformant architectures with instantiations of patterns such as Observer, Interceptor or Strategy. In addition, your job as an architect tends to continue forever in such environments. This is where I'd like to add my fourth rule "Domain logic dominates infrastructure; infrastructure dominates operational requirements; operational requirements dominate developmental requirements". Of course, it is also important to specify when operational requirements appear in the running system (run-time, install time, compile time, ...) and who is in charge to inject them (operator, user, developer, ...).
As of the huge diversity of requirements it is not easy to come up with general rules of thumb for all of them. Some observations might be interesting in tbis context: A given requirement might be invasive (i.e., cross cutting through other parts of your system) or non-invasive (i.e., it is sufficient to add some additional infrastructural components to the architecture). The same requirement might be invasive for one system (security in all its aspects must be supported everywhere) or non-invasive (we just can add a firewall and don't have to care about security at all). A requirement might me measurable such as the response time to a method request or not (such as removability). Requirements that can be measured can be covered by testing, all other requirements must be covered by architecture and code analysis. Last, but not least, my rule 5: "Classify your requirements and provide strategies how to deal with them in architecture design activities".
Of course the overall goal of software engineering must be to
- Let engineers focus on their domain by strong separation of domain aspects from other aspects.
- Automate where possible and useful.
There are several efforts to support these goals such as Model-Driven-Software Development, Product Line Engineering, Aspect-Oriented Software Development, SOA.
One way could be to provide DSLs for the domain, additional domains for the most important non-functional requirements, a technology that easily lets project teams integrate these DSLs under one umbrella (e.g., by leveraging AOSD techniques, Semantic Integration). A Product Line Engineering approach could define core assets which become then part of platforms or frameworks. With SOA, platform dependencies could be overcome. Is this just Science Fiction or a reasonable future vision? Let's work on this :-;
Thursday, March 29, 2007
Back in Town
If you like to read the Ph.D., then obtain a PDF version from the university library. Since the defense ceremony was very impressing (in an old-fashioned and nice setting), you can also look at my flickr photo set. By the way, Groningen is a lovely town in the Netherlands. You should definitely visit it whenever you are in the area. In addition, all Dutch people I met are extremely nice. I am glad that everything is done although I am already missing the great co-operation with Jan. Now I am ready again for future blog postings.
Sunday, February 18, 2007
Simple is Beautiful
The fundamental properties of of these examples are as follows:
- The entities and their interactions are simple
- If any of these entities and interactions are removed, full coverage of the problem space is not possible anymore
- The same design principles are applied orthogonally throughout the system. Such design principles cover functional as well as non-functional properties
- The system design provides atomic components on top of which higher abstractions (e.g., subsystems) are built, thus resulting in a hierarchical instead of a flat structure
If you think about the properties above, you'll see that a violation of one of them immediately results in a non-simple (i.e., complex or simplistic) system. For instance, you might have simple entities and relations, but when the system consists of a giantic flat structure, you might be lost. Trying to understand Windows Vista by looking at its assembly code, simply does not work. Simple systems offer what I call architectural beauty. Just by using a small set of easy-to-understand concepts, you can even cover complex problems in a domain.
What architects tend to forget is the fact that humans are not able to consider and understand more than 7 entities at the same time.
Thus, upcoming Ultra Large Scale systems will only succeed if they rely on a simple fundament consisting only of a few core concepts that can be combined to a broad spectrum of emergent behavior.
Simple does not mean simplistic. A system is simplistic when its core elements are not capable of covering the whole problem space.
How can we achieve simplicity? Simple software architectures often reveal a high pattern coverage, use abstraction concepts such as components or services, and use the same solutions for the same problem contexts. Evolving a simple system might happen in two ways, by evolving the core concepts or by applying the available abstraction mechanisms.
By the way, in case you wonder how to check whether an architecture is simple: one test for simplicity could be the following. An architecture is simple when is can be explained to other experts in a few minutes without using visualizations.
It is important to mention that in a software project it is not sufficient to deal with simplity in the software architecture. Simplicity should also be the leading principle for other aspects such as the process, the tools, the technologies, the programming languages, the models, the DSLs, and so forth. Also keep in mind that it is generally insufficient if each of these aspects is simple. You might use a simple but powerful language such as Lisp and will be completely doomed to fail when trying to build a multi-tier enterprise system using Lisp even if the architecture is simple. This implies, that the combination of simple aspects must be simple itself. Therefore, consider simplicity a holistic approach.
Mind the "Keep it simple, stupid" principle to make your professional life simpler but not simplistic :-)
Sunday, January 28, 2007
The Singleton Patterns and The Concept of Identity
Basically, what the Singleton does is to guarantee the existence of one single instance of a class.
Example from JavaEE:
// get ConnectionFactory via JNDI
ConnectionFactory cf = ... // single instance
Connection c = cf.CreateConnection(...);
The factory uses the Singleton pattern such as in:
class ConnectionFactory {
static private ConnectionFactory _instance = null;
protected ConnectionFactory() { ... };
static public ConnectionFactory instance() {
if (null == _instance)
_instance = new ConnectionFactory();
return _instance;
}
}
For sake of brevity I didn't care about thread safety. For our purpose you can just ignore this fact. In the code you will recognize that no external user of the class can access the protected constructor so that the only way to obtain an instance is calling the static factory method instance().
This way, there will be exactly one instance whenever one or more clients access the class.
There are many problems with this approach. The most important one is that a Singleton defines a global variable such as ConnectionFactory.instance().
Suppose, we need exactly one object instance. Do we really need to rely on Singleton?
Let me introduce a simple example.
class ThePrinter {
private String printerAddress;
public void print(String fileName) { ... }
public ThePrinter() { printerAddress = MY_CONSTANT_VALUE; }
}
In the class above, we can provide multiple instances and each of these instances will semantically define the same object (thus we should provide appropriate equals and hashCode contracts).
From an identity viewpoint, we need to constrain our class to provide objects with the same identity. The way a Singleton tries to guarantee this is by only providing one single instance. But it is legal and much more appropriate (in terms of OO pureness) to achieve the same result using an immutable class as shown in the printer example above.
Another option of course would be if we need to constrain the numbers or kinds of instances created by a class such as in a Manager or Pool class. These only allow creating specific instances or a maximum number of instances. Thread and connection pools are typical examples for such behavior. In this context, a Singleton is a Pool with capacity == 1. I won't show you more code here. You can google for Peter Sommerlad's Manager pattern.
Another "inverse" kind of approach in terms of identity is the following. Suppose, you got thousands of users on your Web site. You started with providing one volatile user object for each online user which made your system less fault-tolerant and more resource consuming. How could you change this? In other words, how could you provide a limited number of instances of user objects (# user objects << # online users)?
a) implement a pool of fixed user objects that wait for incoming requests (Active Object pattern)
b) when a request comes in, it is enqueued
c) one of the idle objects dequeues the request
d) it retrieves the user id from the request and looks up the user's data in the database
e) it operates on the user data and stores it back to the database
f) the result is returned to the client
g) the user object is ready to take another request and thus another identity
This approach does even work if multiple user objects run under the same user identity. In this context, each pool object can temporarily take any identity because the identity's data itself is persisted somewhere else. Thus, identity is not the same as instance.
Keep this in mind when you next time feel addicted to solve an identity problem with the Singleton pattern or other weak approaches.
It is surprising how often architects and engineers forget to think about identity and how to design and implement it.
Thursday, January 25, 2007
Architecture Refactoring
As you might remember, Joshua Kerievsky came up with "Refactoring To Patterns" in 2004. The basic idea behind his excellent book is to identify places in an implementation that use proprietary solutions for problems which should be better solved using patterns. I also want to give you an example for this approach. Suppose, you have implemented a class ShareValues that contains a list of consumers. Whenever a share value changes, you iterate through the list and call event handler methods on each of these objects. While simple to implement, the flip side of this approach is the tight coupling unnecessarily introduced. As soon as new consumers are added, we have to manually modify the ShareValues's list of consumers. Obviously, applying the Observer pattern comes to our rescue here. The advantage of refactoring to patterns is that we are able to introduce a higher abstraction level.
If you think that even further, we enter the world of what I call Architecture Refactoring. In Architecture Refactoring we refactor the architecture itself.
Possible Examples:
- Partition Responsibilities: If a component or subsystem got too many responsibilities, partition the component or subsystem into multiple parts, each of which with semantically related functionality.
- Extract Service: If a subsystem does not provide any interfaces to its environment but is subject of external integration, extract service interface.
- Introduce decoupling layer: If components directly depend on system details, introduce decoupling layer(s).
- Rename Entity: If entities got unintuitive names, introduce appropriate naming scheme.
- Break Cycle: When encountering a cycle on subsystem level, break it.
- Merge functionality: If there is broad coheshion between two modules, merge them.
- Orthogonalize: If two parts of an architecture introduce different solutions for the same problem, choose one preferred solution and eliminate the other.
- Introduce strict layering: If in a layered system, a layer accesses lower layers without necessity (relaxed layering), enforce strict layering.
- Introduce hierarchies: If several entities are only variants of a particular entity, introduce a hierarchy.
- Introduce Interceptor hooks: If we have to open an architecture for out-of-band functionality according to the Open/Close principle interceptors should be introduced.
- Eliminate dependencies by dependency injection: Reduce direct and wide-spread dependencies of Parts in a Whole/Part setting by introducing a central runtime component (Whole´) that centralizes dependency handling with dependency injection.
So far, these are only suggestions of potential candidates. But you can see the possibilites now. Obviously, we should collect a whole catalog of such architecture refactorings. Note, that higher-level refactorings might incur lower-level refactorings the same way architecture patterns often refer to design patterns. Extract Service will very likely lead to an Extract Interface refactoring if the implementation already exists. Orthogonalize is heavily leveraging refactoring to patterns.
From my viewpoint, we have just started to understand and apply refactorings in a more holistic context. As the old saying goes, we have just seen the tip of the iceberg. Architecture Refactorings will guide architects to identify potential problems in a software architecture and also provide them with refactorings to solve those issues.
Sunday, January 21, 2007
How to start with requirements
So, let us assume we are starting a new project. For simplicity, suppose we are in a company that produces healtcare products. Some senior managers in our company found out that it is time to build a web-based framework for physicians where patients can obtain information, ask for prescriptions online and arrange meeting schedules. Product management asked several customers for theirs required and desired features and it turned out that customers would buy such kind of product. Thus, we got our business case.
As a software architect, the first step is to interview stakeholders about their requirements. The problem here is that senior managers and product managers are often far away from software architecture and technology. Hence, it is our task to clarify requirements with them and translate management level requirements to technical requirements. For example, I often hear sentences such as "this must be performant and scalable and secure and flexible." Yes, but what exactly do they mean by "flexible" or by "performant"? What if both requirements contradict each other? Another issue here is that different stakeholders often have different perspectives. A project manager in a multi-site software development project might be more interested to partition subsystems in such a way that each subsystem is developed by only one site, while a software architect thinks more problem-space oriented. Political issues might also be lurking such as sites which are striving for being responsible for specific parts of the product. Obviously, outsourcing and offshoring denote important factors as well. All of these issues need to be addressed by software architects. In order to deal with them, architects must disclose the relevant issues from day one. Here, it turns out how important communication skills are for architects if you ever doubted it.
My first recommendation is: software architects should arrange meetings where they interview all relevant stakeholders. Thus, identify stakeholders with senior management. Then take the time to interview all of these stakeholders. Prepare yourself with checklists where you list questions you want to ask the stakeholders. Otherwise, it might be a problem to get full coverage or to obtain comparable answers. Note, that these checklists differ depending on the personal backgrounds of interviewees. It is a common rule that senior managers only seldomly can give technology or architecture advice (at least not explicitely). In the interviews clarify requirements. Ask interviewees what they exactly mean with the requirements. You should also be prepared to ask more concrete questions and guide the interviews. For example: our online patient system framework should be flexible. What does flexible mean - parts can be removed, added, exchanged, removed or does it refer to modularity? How fast must changes be committed? How many changes are anticipated? Can these changes be added at runtime or at compile time or at maintenance time? What kind of changes do typically occur? Don't forget to ask interviewees about the priorities they give these requirements. If they tell you, all requirements are of identical importance, ask them questions like: "If you had to choose one of these requirements as the most importan one what would it be?" or "What are the features that must be available in the beginning and what do you consider the highest risk?".
After the interviews set up a kick off meeting where all architects and stakeholders are invited. Present your findings to them and also prepare a rough proposal of the project setup (milestones, work packages, activities, requuirements with priorities and risks). After the meeting all the attendees should commit to a final project plan, have a common understandings of terms, and what exactly the requirements mean, and what their risks and priorities are.
In this context, it is important to mention that there are different types of requirements:
- Functional requirements define the concrete domain-specific problem (in terms of the problem space not of the solution space!). Here a DSL is a good start. Or at least, should be one of the results of domain modelling.
- Operational requirements such as performance or fault tolerance relate to the operation of the final product.
- Developmental requirements define architectural issues such as flexibility.
Customers are often more interested in functional and operational qualities. These qualities will also have a huge impact on the strategic core of your product. Developmental qualities are mostly tactical issues. They are of high relevance for the development team, especially when we are going to build a product line such as in our example project.
Often, it helps a lot when defining the boundaries of the system to be built using context diagrams. Additionally, a use case analysis with participation by customers is essential.
It is important to mention that a software product line such as the one introduced in our running example requires special focus on commonalities and variabilities of the products we are going to build. For example, some of our customers might need the patient meetings to be synchronized with Exchange Server, while others might use IBM/Lotus, or even exotic platforms. Thus, our patient framework must be built in a way that allows this kind of variability. Another variability may be the different looks and feels of the Web sites.
After you got final agreement and committment on the requirements you can start to proceed to the next step and set up the project which won't be covered in my postings. However, it turns out to be important to asign important subsystems to key developers and also assign the most important non-functional issues also to key developers. Software architects are then responsible for the whole picture, but key developers for their own topic of concern. In the beginning, architects and (and key developers) will design a base line architecture and communicate this architecture to the stakeholders. The design should reflect functional aspects but also at least the core non-functional concerns.In our example, security and usability will be most likely the most important requirements. Thus, we should organize two teams, one responsible for security and the other one for usability.
In the whole project activities is essential to ensure requirement traceability. Each design or implementation decision should be motivated by requirements and follow the architectural baseline. This is where architects must act as supervisors.
This concludes the first posting in my "series".
Friday, December 29, 2006
Architect's Toolset - CRC Cards

I was always wondering why I still can find developers not familiar with this tool.
- The class name is the name of the component being described. In the example, we are introducing a Shopping Cart.
- The responsibility section contains all responsibilitiesfor which this component is in charge such as adding or removing items in our example.
- In the collaboration section we define on which other components our component depends, the so-called collaborators. In the example, the shopping cart must be able to co-operate with the product catalog.
All of this is simply drawn on a sheet of paper and then visibly put on a white board, poster, whatever is available. After a while the board will be filled up with CRC cards. In order to make the collaborations visible I often use a string for connecting the CRC card to its collaborator(s) using pins.
Because this is a flexible, paper-based solution it is easy to change CRC cards, remove them again, add new ones, add or remove responsibilities or collaborators. Everyone in the team is able to participate in the whole brain storming process.
Obviously, in an agile setting the CRC cards will be incomplete in the beginning and then keep growing over time.
Later on in the design process we might add additional information to the cards such as superclasses.
There is a lot more to say about CRC cards. Ward Cunningham , the "father" of CRC cards, and Kent Beck provided an excellent paper for OOPSLA 1989 that you should definitely read (http://c2.com/doc/oopsla89/paper.html).
Wednesday, December 27, 2006
Product Lines
Ok, what is the second approach? In our second attempt, we start developing the Ruby IDE taking special consideration to implement all subsystems such as the GUI, the debugger, you name it, in a re-usable way. After the Ruby IDE is completed, we can re-use existing components for further IDEs. Basically, we got a set of independent lego units which we are just might compose in different ways. Obviously, this is a much better approach in terms of productivity through re-use, but it turns out to add other challenges. For example, we have to come up with different application architectures that use the existing components in a bottom-up way. All of these architecture require a lot of effort themselves. In addition, it turns out that some of the components developed for the Ruby IDE are not appropriate for the Fortress IDE. Thus, we need to refactor them to be more generally applicable. But after that work we need also to refactor the Ruby IDE to integrate these refactored components. Of course, we might choose to end up with different versions of components. This is exactly one of the reasons why some projects experience death by unsystematic re-use.
Ok, what is the third approach? In the third approach we set up a development team that first analyzes which concrete IDEs we are going to develop. The team will take all requirements and define an general architectural framework for all of these IDEs. It will come up with commonalities which are all assets that should be part of the general framework. E.g., components such as the project and configuration management subsystem that are basically identical no matter which programming language you need to support. The team will also define the variabilities , i.e., things that vary among different IDE editions. For example, the abstract syntax editor framework might be a commonality, but the different concrete syntax editor implementations will vary. As we expect users to be interested in adding their own components to the IDE (for example tools that support programming conventions), we integrate a plug-in manager. This is an example for a component that supports variation over time. So the general team will set up an architectural framework with explicit variation points as well as other commonly used assets. As we expect these assets and even the architecture framework to evolve, we need to set up an environment where common assets and their evolvement can be supported. By the way, assets in this context can be different kinds of things: implementation artifacts, documents, models, test plans, and so forth. The setup of such a project will be very expensive but the Return On Investment will rock. As soon as everything is in place, the Ruby IDE team will be able to systematically instantiate the architectural framework with all commonalities, and also bind the variabilities according to their own needs. They will save a lot of time because they don't have to come up with their own, new software architecture from scratch and they can even use existing assets. What we have defined here, is Software Product Line Engineering. All products instantiated from the product line are often called Software Program Families. I hope, you got the basic idea. It is like in other industries such as car manufacturing where the car design team comes up with a new series of cars, defines what parts all of the models could share, and also defines the variabilities. Obviously, a development team must carefully identify whether is worthwhile to set up product line engineering. It is probably a bad idea when you only have two different products.
So far, the teams that develop concrete products use the architectural framework and the common assets but the rest will be plain-vanilla software engineering. In most cases, there is a lot of space for automating many activities. Take a production line in a car manufacturing plant as an example. In Software Factories the idea is to define a systematic plan of how a product is configured or implemented using the common architectural framework and common assets. The Software Factory will then allow automating the process of developing a concrete product by leveraging model-driven software development. In order to make this experience less challenging, the idea is to come up with a domain-specific language (instead of forcing everyone to use UML) and provide a generator that includes all knowledge about the architectural frameworks, as well as about commonalities and variabilities. The generator will take a model as an input and generate (most parts of) the concrete product.
An issue we must consider in this context is how to deal with evolution and change of core assets It is unlikely the architectural framework or other assets will remain unchanged forever. Thus, a systematic approach is required to give feedback from product instantiations to the product line itself. This also implies, you should always have separate teams for the product line itself and for the products. Otherwise, organizational issues might me very challenging. In other words: even, if you provide a product line engineering approach, you might be doomed to fail due to organizational problems. Needless to say, that the development process is also significantly different which you should take into consideration.
An interesting issue is the differentiation between a platform and a product line. In many cases, the most important core asset of a product line will be a complete platform. Eclipse is a good example for this. In this case, the platform will be the critical part of your whole product line. If it suffers from design erosion (see my last blog posting), this will have an impact on ALL products. That's why systematic evolution of common assets is so essential. Note, that a software product line does not necessarily have to introduce a platform. This is only one extreme, where we have prefabricated implementation of the common architecture and other common subsystems and components. Another extreme might be an application framework. Or we might only have a set of loosely-coupled artifacts and a software architecture. Thus, there are lot of options to implement software product lines.
In future postings I will address architectural issues related to product lines in much more detail.
Friday, December 22, 2006
Design Erosion
Sunday, December 10, 2006
Teaching the Architects
This time I'd like to address one specific issue. How can we effectively teach other persons everything they need to know about designing a high quality software architecture? For the sake of brevity, I won't cover what a software architect should know in this posting. In the seminars Frank (Buschmann) and I give, we mostly offer a mixture of Powerpoint presentations, discussions, and group exercises. In the presentation sections we also address things we've learned in projects. In the group exercise 3-5 attendees are then asked to design a small example application such as a Web Store or Chat Server. This setting works very nicely, but is far from being sufficient. Thus, another and additional approach is to teach and consult small project teams in real-world projects. In this scenario we start with a series of seminars and then participate in real-world projects as architecture consultants. However, this isn't sufficient too as we also need a kind of certification for software architects. Of course, small companies might not afford to spend all that time and budget for educating their staff in architecture concerns. But that is another topic.
I'd like to know what you think. I'd also like to know about your experiences. What did work for you and what didn't? How did you become a software architect? I am looking forward to all comments.
Thursday, November 09, 2006
Ultra Large Scale Systems (ULS)
Swarm intelligence was a sort of precursor in that direction. Define behavior of entities as well as their interactions locally and then let the system act globally. The killer application were ant populations.