In the eighties I bought s small pogramming system called Borland Turbo Pascal for around 100 bucks. The whole language system documentation included a tutorial as well as a reference manual with more than 100 pages. I managed to read through the whole documentation in a few days and from that day on I could call myself Turbo Pascal expert.
In the beginning of the nineties I got familiar with a new middleware standard coined CORBA. The first implementation from a small Irish company IONA Technologies was extremely easy to use.
In the mid of the nineties a programming language called Java conquered the world. As it looked amazing, I learned Java in a few days reading through the seminal book "Java in a Nutshell" written by the well-known David Flanagan.
What do all of these stories have in common? Well, you will find their commonalities if you consider what happened after these technologies were introduced. Borland extended Turbo Pascal until it became what we now know as Delphi. CORBA version 3.0 turned into a giant middleware platform no single expert is able to understand. And I can't believe that anyone will ever be capable of knowing Java the language and all the different platforms such as Java EE in full detail. Obviously, this is not an observation only applicable to software products. When did you buy the last gadget (mobile phone, MP3 player, TV set, ...) where you knew all of its features? When did you read the whole instructions manual or user guide? And how many of these features do you actually use?
The main challenge for software engineering is death by complexity. Even if you were expert in some of the technologies such as Java and CORBA and Eclipse and AspectJ, you wouldn't be able to oversee all implications of using deliberate combinations of these technologies. "Yes, you definitely need to use the new version of Hibernate as well as EJB 3.0, WSDL 2.0, Java 1.6, Spring 2.0 which, of course, must support the new Windows Vista SP1. Not to mention all the new tools such as Eclipse 3.3, Maven2, Ruby 1.9. I forgot to mention that our documentation will use the new Share Point Server 2007 as well as Office 2007 applications. And did I already tell you about the new Microsoft Silverlight 1.1. we need to support?" Today, we are expected to be masters of many technologies each of them being extremely powerful. As this mastership doesn't work except for a handful of geniuses among us, we often become jacks of all trades but masters of none. Obviously, in a perfect world we would be experts in all technologies we are using, but in the real world we simply can't reach this level of wisdom. And often technologies even change or new technologies appear within the product lifetime. In concrete projects we don't have sufficient time to check and understand all these new technologies before applying them.
The problem is that almost everything in the world of software engineering has become a fast moving target. We are surrounded by these moving targets in our daily work. Some might argue that technologies reached new levels of power in the last years. We definitely achieved a high level of abstraction using OOP concepts, SOA, Model-Driven-Software Development, patterns, new runtime platforms such as Java and .NET, you name it. Today, we can implement software systems that we were not able to think of a few years ago.Unfortunately, the complexity of the domains has even more increased. With other words, all these cool and smart technologies can not cope with the rising complexity introduced by the increasing requirements in real projects.
I think, that most technologies are simply over-engineered offering functionalities no one really needs. Consider Office software systems as an example. I also believe that product development cycles are much too fast to achieve the required level of quality. Quality requires to gain in-depth knowledge of all technologies which is not very realistic in current projects. Another experience is that many projects neglect software architecture efforts. Why else must I constantly explain to engineers and managers that architectural efforts are not a waste of time and money? The software architecture base must be stable. It needs to be developed as well as evolved in a systematic manner. Otherwise, the whole system will collapse or implode. Many systems start with a nice architecture which then begins suffering from design erosion after engineers have added more and more workarounds under increasing time pressure. This problem reaches new dimensions when facing product line engineering and platforms. Now, platforms are developed that do not serve a single one-off application but a whole sequence of products and solutions. All problems lurking in the guts of such platforms, will inevitably have an impact on all applications relying on these platforms.
Maybe, automating most engineering tasks can be one of the solutions to cope with this challenge. Software engineering is mostly about introducing indirection layers so that adding new abstraction levels and hiding more and more details from the programmer might be another approach. Nonetheless, a third idea might be to stop for a while, reflect over what we are currently considering as the appropriate approaches in software engineering, and then maybe come up with a different way of thinking. Otherwise, our whole discipline will suffer in the future.
