Agile or Some other Methodology. Are we still thinking?

We are still debating in most projects whether to use Agile or use traditional approach.
I find at least two reasons as to why its difficult in moving from traditional water fall / other approaches to Agile (1) our customers are unaware of how agile can help them build something they actually want.
(2) Some of us have made traditional software development approach as our software development religion. We feel threatened by the fact that what ever we have learned in our schools and during our past experiences of developing software applications using traditional approach is suddenly not the right approach.
In traditional approach the design won’t start till the requirements is finished. Coding won’t start till design is finished and the testing won’t start till you are done coding. The reason which was given was that you must gather all and correct requirements before you design because if a requirement is wrong then everything from design to test is essentially a waste. I guess in doing that we missed one major thing and that is human factor. First of all we should stop the over use of the word “Requirement”, and try to find out what are the preferences of our customers. Preferences change more frequently than requirements. (An ideal software solution would be one where there are no requirements but only preferences.)
Let me ask you this, if you plan to build a custom home for your self, can you write each and every detail of your house in a piece of paper, give it to builder and then come back when the house is completely build. Do you think that whatever you were imagining while writing your so called requirements is exactly what you got? You might be getting something close to what you were thinking but not exactly what you want and sometimes it could be totally different then what you have thought. That’s where the difference lies; there is a difference between what you think you want and what you actually want.
Now instead of building the whole house and then showing it to you, if the builder shows you every week or so what he has build so far and what he will be building in next week and each time asks you if you are satisfied? Now let’s say you are not satisfied with what ever was built last week or so and you want the builder to redo or change something. If the change is simple builder would probably do it for free. But lets say doing that change would cost some extra $ . Now you will have three choices (1) Not to do the change (2) Pay extra $ (3) Trade in some other feature of your house. (E.g. may be not to build the deck and instead do the change which is more important to you). The outcome would be totally different if the builder used this approach. At least there won’t be any surprises when the builder finally delivers you the house. Similarly if there are no surprises when the software is put to production then you have a satisfied customer and hence a successful project.
Water fall or other approaches works fine if there is no change in the requirements. No change means you were able to document exactly whatever your customer was thinking (not saying) when you were documenting the requirement. Another issue is that sometimes we don’t know exactly what we want until we see it. Take the example of buying a car. How do you decide that you need a particular car? Well, simply put, you will actually look at a car and if you like it, you will go ahead verify other specifications and buy it. You requirement is that you need a car but which particular car buy is your preference.

We all have seen that most of the time when software is delivered it does not meet our customer’s needs and preferences and then we start blaming our poor requirements gathering, wrong software design and may be bad technology selection. But I guess the blame goes to the fact that we were not ready to accept that fact that requirements could be poorly documented. Human beings to some extent are afraid of change, simply because changes are unpredictable and so we do not welcome changes easily. I guess time has come to solve this software problem and instead of resisting change why not welcome it.
Traditional approach resists change but Agile welcomes change. Agile software development methodology eliminates element of surprise by releasing software in short iteration (2-4 weeks). In each iteration you plan, design, code, test and then release your software. Every 2 week your customers get a chance to look at your software. With each iteration, they will have more insight in to what they will be getting in the end. They are welcome to change their preferences and even after 2-week, if what ever you have developed, is a waste then let it be. It’s far better and more productive than keep working on developing software for a year and then realize that more than 50% of the software is waste. At least we all know that changes done in early stages of software development are less costly compared to changes done in later stages of software development.
In the end I would say that I am not sure whether Agile will work for real-time and satellite/rocket software’s where whole system must be designed and tested before it can be put to use but I assure this approach works for rest of the application software.