Before the agile community swarms my home with pitchforks and torches, let me get one thing straight right now: I have nothing against agile development practices (ADP) or theory and do not dispute their efficacy when applied reasonably and pragmatically. I have learned much from, and have a deep respect for many agile evangelists such as Martin Fowler and Kent Beck, and more recently Mike Cohn of Mountain Goat Software. In fact, I believe that many of the engineering practices purported as “agile” are simply a handful of best practices that have emerged from modern software engineering. I stand behind, and use many of these practices in my own work and continue to learn from many of the aforementioned visionaries (although I so wish we could banish the term "agile" as it has become such a polarizing term in software process community).
But while I believe that the tenets of the Agile Manifesto are noble and well intended, I can’t help but wonder if these tenets are being somewhat misinterpreted by the community at large and should either be clarified or ratified.
One questionable trend that I believe we are witnessing in the agile gold rush is the wholesaling of agile methodologies into canned process frameworks that are peddled like snake oil to desperate and unsuspecting souls. These frameworks are often deployed very prescriptively, as if to “cleanse the masses” and often result in a carte blanche implementation where previous learnings are encouraged to be forgotten or disregarded.
In fact, I have seen teams adopt agile methodologies with such zealotry that they cling to every letter of the tenets as if the words were inscribed on a marble tome. Worse, they then go beyond the written guidelines to form their own interpretations which are then applied conveniently to meet their needs. This is a dangerous practice that is common to all religions, and the fact that I am even describing a software methodology in this context suggests that things are getting a little ridiculous. The industry doesn't need to be born again, it just needs a unified representation of how to accomplish the job by doing more of the right things and a reminder that no one is smart enough to figure the whole puzzle out before getting their hands dirty. This doesn't mean you scrap everything for this new snake oil- it means you identify some key areas of opportunity and lend some good old fashioned process improvement.
There is an exception to this, where wholesaling ADP does make sense, and that is in organizations that have very little understanding or appreciation for the engineering aspects that drive product development. These are the companies that are 100% date and sale driven and blame IT for all of the failed projects because they lack the transparency to identify the real problems keeping them from meeting their objectives. But few of these organizations will actually stick it out because the cultural transformation is often insurmountable (then again, one would have to ask themselves why they would stick around in this environment to begin with)?
But the biggest problem I see developing within the agile movement -that has gone largely unchecked- is the de-emphasis on so called up-front design and architecture, a key component that I believe is fundamental to (successful) software engineering. Although it is not stated anywhere in the Agile Manifesto to abandon architecture and design, the dismissal of up-front architecture and design as draconian casualties of “traditional” or “waterfall” methods is a common thread within the agile community. As an architect, when I challenge this point (and trust me, it happens all the time), too often tenet # 3 (“Working Software over comprehensive documentation”) is used to support this belief that up front design is the modern leprosy of agile development. Martin Fowler has done some work to help dispel this myth, but there is still much work to be done around educating the community on this topic, and I am confident that many reading this actually firmly believe that all so called up front design (BDUF) is evil.
The reality is that this tenet is being used as a fig leaf to cover up the real reasons for not doing the right things, and this has nothing to do with being agile or not agile, but instead points to what I think boils down to a lack of experience and maturity.
Some up-front, pragmatic architecture and design work which sets the foundation for the project's conceptual integrity and then is consistently refined is a sound engineering practice which is the responsibility of a competent architect to perform. Such critical design aspects include layer and corresponding artifiact definitions, assembly allocation, messaging patterns, process allocation, identity management, transaction boundaries and deployment plans- these are fundamental to software projects that you don't need a crystal ball to figure out, just some good old fashioned experience.
Of course, all architecture and design is to some extent evolutionary, but modern software projects of any significance apply this technique as a critical step for gaining consensus, forming conceptual integrity and orienting teams on the work that lies ahead. In my opinion, this step is just as critical to the success of any non-trivial endeavor as writing unit tests and automating the build process. Skipping it will almost always yield lost time, productivity and result in a fragile product, regardless of how agile or motivated the team may be.
There is nothing written in stone that says that all BDUF is bad, and I believe that there are a fair amount of Agile projects that do put this critical step into an architecture/core-foundation iteration, but I just wish that the Agile community at large would distinguish this a bit better so that the lay person interpreting the impact of ADP has a more educated perspective. This is certainly an optimistic perspective, and for as many moderates on this subject, I am sure there are radicals, but the I think that the kind of thinking that rationalizes the omission of this practice threatens the software industry’s ability to evolve as an engineering body of knowledge and will further demarcate the gap between enterprise software and toys.
Worse, I believe it will continue to prevent the agile movement from ever graduating to a mainstream, disciplined SDLC process, instead being forever relegated as a fringe software development process. This is unfortunate too, because I think ADP has a real potential for forming the bedrock of what modern software engineering practices look like and mobilizing the industry to do more of the "right things", lest we skip the critical design aspects.