One of the problems of the software development industry is non-software people make decisions about software builds. Many of these people believe that building software is like building houses, which it is not. People see the software business, approach it like building a house, specifying upfront and asking when it is finished. Assigning project managers as general contractors and hiring cheap labor to execute the build. I can hear some managers who are reading this think: Wow! can we do that? That is the answer! No, it is not, we tried it for more than a decade and it failed spectacularly. They even came up with a name for it: “out-sourcing”.
But even though we move from Big Design Up Front (BDUF) to Agile the analogy of building houses is hard to get rid of. Yes, we know we do Agile with sprints, no deadlines just sprint goals and a backlog with user stories and a road-map with epics. But we are still building a house. Well.. no.. we are actually not building a house. The whole house analogy is wrong. We are building a LEGO house. Yes, something like this:
Was it a coincidence that Paul Hammant used a LEGO picture in his post about software architecture? I think not. Here are some of the points that show why building LEGO house is a good analogy for building product software:
- Innovation: Best builds come from experimenting constantly and thinking out-of-the-box.
- Flexible: LEGO house foundation can be replaced carefully. Anything can be replaced any time.
- Looks: Building a pretty LEGO house is very hard and requires special skills.
- Agility: There is no such thing as a “done” LEGO house. It is alive and constantly changing.
- Requirements: You don’t plan building a LEGO house. You know what you want when it evolves.
- Specs: LEGO is enjoyed best by building whatever comes to mind, not by following specs.
- Compatibility: Everything connects to everything, with hardly any effort.
- Freedom: LEGO can be build without having to comply with all kinds of regulation.
- Creative joy: Creating software is as much fun as creating with LEGO.
- Value: People pay for the promise of easy integration and high flexibility.
- Validation: A real-life test by the target audience tells you things you could not imagine.
- Mindset: Grown-ups do not understand LEGO, like business does not understand software development.
- Out-sourcing: If somebody else builds your LEGO house, you will get something you don’t like.
- Standards: Build with non-standard blocks and you’ll get stuck quickly.
Can you think of other things in which the analogy is right? Use the comments! Don’t comment saying that I do not take software development serious. I believe that software development is a serious business, but it is also seriously fun. To put it stronger: If it does not remind you of playing with LEGO was you were still a kid, you are doing it wrong.
Managers in software teams are often confronted with highly intelligent programmers; Programmers that all suffer in some extent from Asperger’s syndrome. These people are very intelligent, question authority and do not do what is told unless they are satisfied with the reasons why. They have a low social sensibility and love endless discussions about principles. They like to stick to rules onces they are settled, can focus for hours on single pieces of code, love to be fully focused and hate it to be disturbed.
Am I generalizing? Yes, you bet I am, but I am sure: If you are a programmer you will recognize some of it.
“Managing programmers is like herding cats” – Meilir Page-Jones
I love that quote, because it holds so much value. Some say it only applies to senior programmers. But that leads to the following dilemma: hire easy-to-steer junior programmers that may have little or negative contribution to the software you are building or hire hard-to-steer seniors that can bring big value. Whether it is true or not, I think the latter is preferable.
Dealing with senior programmers is hard and I believe it is a challenge that does not have its match in other industries. One of the factors that is making matters worse is the global lack of senior programmers. These programmers can afford to behave arrogant and non-compliant, because the software business needs them so badly. They can always find another well paying job. All they have to do is mentioning on their social media profile that they are “available” and it will rain job offers on them.
One of the methods I see that people use to get grip on their developers is to pay them above average or get them secondary benefits that are so good they cannot be matched elsewhere. But then again, you must be careful this does not lead to arrogance, lack of self-criticism and a feeling of superiority. These are definitely enemies of a good working team, so this cure may be worse than the disease.
Further reading on Asperger’s syndrome
The following links give some more insight into Asperger’s syndrome:
Browsers have become the most important application you run on your computer. Google Chrome, Firefox, and Internet Explorer seem to be the most commonly used browsers today. But there are other options. The following things are worth considering when choosing a browser:
- Support for media
- Development tools
If you are on the Linux platform, you can install a variety of browsers, but Firefox and Chromium are probably the most popular choices. This post will introduce two other browsers for Linux that you might not know yet: (Gnome) Web and Midori.
Gnome Web 3.10.3
Gnome Web is an elegant polished and minimalistic browser that only runs on Linux and feels much like Safari. It used to be called Epiphany and was based on the Gecko engine (currently WebKit). It has a good set of development tools hidden under the right-click (Inspect Element).
- Engine: Webkit
- Search: DuckDuckGo
- AdBlocker: Yes
- Platforms: Linux only
Installation command for Debian-based Linux systems:
sudo apt-get install epiphany-browser
User-agent for this browser on Xubuntu Linux 14.04:
Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/538.15 (KHTML, like Gecko) Safari/538.15 Version/6.0 Ubuntu/14.04 (3.10.3-0ubuntu2) Epiphany/3.10.3
Midori is a fast and full-featured browser with nice development tools. The only thing I found is that it does not have as many plugins as Chrome or Firefox. Other than that I think the browser is just as good (with some better defaults). It also runs on Windows, but as far as I know it is not ported to OSX.
- Engine: WebKit
- Search: DuckDuckGo
- AdBlocker: Yes
- Platforms: Linux and Windows
Installation command for Debian-based Linux systems:
sudo apt-get install midori
User-agent for this browser on Xubuntu Linux 14.04:
Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-us) AppleWebKit/537+ (KHTML, like Gecko) Version/5.0 Safari/537.6+ Midori/0.4
This post is about real world software products that make money and have required multiple man-years of development time to build. This is an industry in which quality, costs and thus professional software development matters. The pragmatism and realism of Joel Spolsky’s blog on this type of software development is refreshing. He is also not afraid to speak up when he believes he is right, like on the “Big Rewrite” subject. In this post, I will argue why Joel Spolsky is right. Next I will show the real reasons software developers want to do a Big Rewrite and how they justify it. Finally, Neil Gunton has a quote that will help you convince software developers that there is a different path that should be taken.
Big Rewrite: The worst strategic mistake?
Whenever a developer says a software product needs a complete rewrite, I always think of Joel Spolsky saying:
… the single worst strategic mistake that any software company can make: (They decided to) rewrite the code from scratch. – Joel Spolsky
You should definitely read the complete article, because it holds a lot of strong arguments to back the statement up, which I will not repeat here. He made this statement in the context of the Big Rewrite Netscape did that led to Mozilla Firefox. In an interesting very well written counter-post Adam Turoff writes:
Joel Spolsky is arguing that the Great Mozilla rewrite was a horrible decision in the short term, while Adam Wiggins is arguing that the same project was a wild success in the long term. Note that these positions do not contradict each other.
Indeed! I fully agree that these positions do not contradict. So the result was not bad, but this was the worst mistake the software company could make. Then he continues to say that:
Joel’s logic has got more holes in it than a fishing net. If you’re dealing with a big here and a long now, whatever work you do right now is completely inconsequential compared to where the project will be five years from today or five million users from now. – Adam Turoff
Wait, what? Now he chooses Netscape’s side?! And this argument makes absolutely no sense to me. Who knows what the software will require five years or five million users from now? For this to be true, the guys at Netscape must have been able to look into the future. If so, then why did they not buy Apple stock? In my opinion the observation that one cannot predict the future is enough reason to argue that deciding to do a Big Rewrite is always a mistake.
But what if you don’t want to make a leap into the future, but you are trying to catch up? What if your software product has gathered so much technical debt that a Big Rewrite is necessary? While this argument also feels logical, I will argue why it is not. Let us look at the different technical causes of technical debt and what should be done to counter them:
- Lack of test suite, which can easily be countered by adding tests
- Lack of documentation, writing it is not a popular task, but it can be done
- Lack of building loosely coupled components, dependency injection can be introduced one software component at a time; your test suite will guarantee there is no regression
- Parallel development, do not rewrite big pieces of code, keep the change sets small and merge often
- Delayed refactoring, is refactoring much more expensive than rewriting? It may seem so due to the 80/20 rule, but it probably is not; just start doing it
And then we immediately get back to the reality, which normally prevents us from doing a big rewrite – we need to tend the shop. We need to keep the current software from breaking down and we need to implement critical bug fixes and features. If this takes up all our time, because there is so much technical debt, then that debt may become a hurdle that seems too big to overcome ever. So realize that not being able to reserve time (or people) to get rid of technical debt can be the real reason to ask for a Big Rewrite.
To conclude: a Big Rewrite is always a mistake, since we cannot look into the future and if there is technical debt then that should be acknowledged and countered the normal way.
The lies to justify a Big Rewrite
When a developer suggests a “complete rewrite” this should be a red flag to you. The developer is most probably lying about the justification. The real reasons the developer is suggesting Big Rewrite or “build from scratch” are:
- Hard-to-solve bugs (which are not fun working on)
- Technical debt, including debt caused by missing tests and documentation (which are not fun working on)
- The developer wants to work on a different technology (which is more fun working on)
The lie is that the bugs and technical debt are presented as structural/fundamental changes to the software that cannot realistically be achieved without a Big Rewrite. Five other typical lies (according to Chad Fowler) that the developer will promise in return of a Big Rewrite include:
- The system will be more maintainable (less bugs)
- It will be easier to add features (more development speed)
- The system will be more scalable (lower computation time)
- System response time will improve for our customers (less on-demand computation)
- We will have greater uptime (better high availability strategy)
Any code can be replaced incrementally and all code must be replaced incrementally. Just like bugs need to be solved and technical debt needs to be removed. Even when technology migrations are needed, they need to be done incrementally, one part or component at a time and not with a Big Bang.
Joel Spolsky is right; You don’t need a Big Rewrite. Doing a Big Rewrite is the worst mistake a software company can make. Or as Neil Gunton puts it more gentle and positive:
If you have a very successful application, don’t look at all that old, messy code as being “stale”. Look at it as a living organism that can perhaps be healed, and can evolve. – Neil Gunton
If a software developer is arguing that a Big Rewrite is needed, then remind him that the software is alive and he is responsible for keeping it healthy and growing it up to become fully matured.
In 1998, I started my career as a functional designer in a large company, designing all kinds of new functionality for large and complex systems. We used MS Word to document the customer requirements. We were free to decide how to do this.
A simple case tool (named SDW) was used to document the functional designs. These designs contained a description of the system functionality, as well as data structure/relationships from a logical point of view. The projects were based on the Waterfall method.
Nowadays, designers use all kind of sophisticated methods and techniques to improve speed, quality, maintainability, etc. I’ve worked in many organizations using methods like RUP, Agile and Scrum, to name a few!
All kinds of expensive tools were used, such as Enterprise Architect, Rational Rose, alongside use cases to document the functional designs. But never have I experienced the level of professionalism as during the ‘Waterfall and MS Office’ period at my first employment.
Why? I’ve been asking myself that question.
Why is it nowadays that Waterfall is often considered as a ‘bad method’? Why do functional designers use these newer design techniques like use cases? Were the classic ones not sufficient? Are these newer techniques better?
Waterfall vs. iterative methods
Waterfall is a relatively simple method for system development. It is a sequential method, that starts with defining all the requirements, then creating the functional design, which is then followed by development and testing. Each stage handles all requirements and has to be finished/approved before the next stage can be started.
The main disadvantage is that the requirements that arise at a later stage are often difficult to include in the scope. Sometimes it simply doesn’t fit the architecture. Or sometimes a chosen solution that is already programmed should be reverted before the new solution can be implemented. So adding or changing requirements at a later stage can be difficult.
A solution for this is to use an iterative method, like RUP. These methods can handle ‘progressive insight’ better than Waterfall because the changes are divided into chunks, which are implemented in phases. So the requirements do not need to be defined in an extensive way, as is done in the Waterfall method at the very beginning of the project.
But iterative methods are usually harder to manage because you have to do parallel planning, manage multiple disciplines at the same time, (re)define the stages and priorities during the whole project. Also iterative methods can lead to constantly changing requirements during the project, because the stakeholders are involved throughout the whole process. New requirements are being added regularly, or changed due to ‘progressive insight’, which makes it very hard for an inexperienced project manager to manage properly.
To sum things up…
Waterfall is relatively easy to use but is too rigid to deal with new insights. This could be an issue in fast changing markets/environments, especially when dealing with larger projects where these risks are higher. But if the requirements at the beginning are solid and usable in the long-term (compared to the project term) then Waterfall could be very suitable.
Iterative methods are harder to manage, but provide flexibility for the stakeholders to (re)define the requirements.
Office tools vs. sophisticated case tools
Case tools are very effective for performing impact analyses, drawing mock-ups, describing functionalities, and creating diagrams. But when the quality from functional design is poor, then a case tool will not improve it, if you would compare it with a text editor like MS Word, for instance.
A functional design needs to be correct, complete, and clear (CCC). As an analyst, developer, or tester, one needs to trust the information it contains. But for this CCC, you do not need a case tool, however, it might help.
So a case tool could be very beneficial when it is used properly.
But sometimes as a designer you are expected to be pragmatic (because of deadlines, for example). In this case, a designer won’t be able to deliver a full blown design. And sometime no designs are made at all, when the changes are communicated orally. This will result in the already existing design being incomplete and even more inconsistent. In this case, the benefit for case tools will not be significantly greater, if compared to a good text editor. On the other hand, an existing and complete design in a case tool will make analysis and maintenance much easier.
However, case tools will not deliver the most visually creative designs as designers are limited in their creativity. This is because of the rules upon the shared use that are agreed with other colleges. The case tool itself also demands that the design are created a certain way (according to a specific structure). But this could also be considered as an advantage because the case tool will guide a designer, making it easier to create the respective design(s).
So a case tool can deliver many advantages, but should be used properly to justify the investment (money, learning time, set up, integration, etc.).
Traditional designing vs. use case
With traditional design techniques you describe the functionalities of a system. You describe what a system does from a system perspective, as well as the logic to achieve this.
Nowadays, use cases have become a very popular method to replace the traditional designing method. There is only one problem. Use cases do not describe the system functionality. But they are used to describe the interaction between systems and actors. So you often still need to design the system logic. But the ‘problem’ is that many designers aren’t aware of this and consider the use cases as the final stage for designing. It is very helpful for work-flow and/or a scenario driven system, and in many cases offer enough information for the developers. But it is not practical to always use use cases. Try to design Excel by using use cases, for instance. That just doesn’t work.
There is also another problem. Readers need to be familiar with use cases, so you need to learn to read them. Designers often undergo training to learn to this method. The readers (customers, testers, developers) do not. There are also many interpretations about how to create a use case properly. For instance, include and exclude relations between use cases. These are very much debated (also on the Internet). This can lead to ambiguous designs. One can guess the consequences.
But under the right circumstances, it is a very powerful method for designers to describe the functionalities (from actor’s perspective), especially when combined with a case tool.
All these new methods, techniques, and tools can be helpful. But this doesn’t mean the old ones aren’t sufficient anymore. Depending on the situation, the right method has to be chosen.
Some things to consider…
Define the requirements for the requirements. In other words, what do you need to gather and document the requirements in the most efficient way.
Investigate the environment:
- Is the environment fast changing or not?
- Consider the level of IT expertise within the company. Do the developers/testers/stakeholders understand use cases. How experienced are the project managers?
- Consider the company culture. How much does the business control the IT department? Does the company demand that everything is documented or do they prefer not to document too much?
- What are the needs? Speed above quality or vice versa.
Then make the right decision. BUT use the advanced one properly, or otherwise keep it simple.