I don’t have a problem with the tenets of the Agile Manifesto and honestly believe that they are noble and well intended. However, I can’t help but wonder if the misinterpretation, and more importantly the flawed execution by naïve practitioners, of “agile software development” as it is called is creating somewhat of a bastardization of modern software development resulting in a setback in the software industry that is striving to move the industry into an engineering discipline.
Is it necessarily the right approach to abandon traditional methods (that do work, given the right teams and organizational structure) in favor of an approach that deals in giving the customer the least amount of functionality possible within a fixed box of time? Have we really given up in our ability to understand the customer's big picture and deliver on it in a highly skilled and professional manner?
The question doesn't just apply to engineering. That is, the question must be asked not only in relation to software engineers (developers), software architects, testers, and infrastructure engineers, but also in relation to product managers, product owners, business analysts, stakeholders, testers, and customers- pretty much anyone that interacts with the creation of software products.
I believe that regardless of the answer to this question, in the end, the intention of agile methods to humanize software engineering is a noble one and is probably the closest thing to fixing the current state of software. I have read of tremendous success in these efforts and am optimistic that valuing people over process and interactions over contracts is a move in the right direction. This said, one of the main problems I see with agile methods is that it tends to cloud the waters between these noble goals and software engineering as a body of knowledge, especially given agile's aversion to up front design and architecture. Does one really give up what she has learned with regard to the need for architecture and design when they go “agile“? Is all this really that extreme? Of course not. No one says you can't do modeling that describes architectural and design decisions, but like taking on the false belief that refactoring is an enabler of code & fix development, the tenets in the manifesto are easily misinterpreted and mis-applied.
What follows is a collection of thoughts, sprinkled with some real world experience that have come together to mold my current thoughts on agile software development and what impacts, both positive and negative it may have on the software industry. An informal collection of thoughts, I may take these ideas, backed by some more formal reasearch and publish a comprehensive paper on the topic, but for now, I welcome your thoughts, counterarguments as well as any ideas you might have.
Software Engineering is not for Everyone
It is my feeling that software engineering will always be an esoteric practice. This is why it isn't everyone that can be a programmer and it is for the same reason that not all programmers can evolve into software engineers (we need look no further back in history than CASE). By the same token, not all programmers with software engineering titles are necessarily engineers, nor do they learn to think as designers and become architects. I know these titles or roles may sound quite hierarchical and limiting, especially in the flattening circles of the agile community which prefers generalization to specialization, but the fact that there are practitioners of varying proficiencies and only a few that attain true mastery of their particular specialty within a trade is a reality of any industry, and software engineering is no exception. Sorry. Please accept this and let's move on.
So, we have programmers, and we have software engineers, some of the latter are, or will one day become software architects if this is the career path they choose. There is absolutely nothing wrong with being a career software engineer, but if your goal is to get rich (hey, it's not everything there is to life, but it's not a bad goal) unless you go out on your own and develop a product that sells, remaining within the corporate workforce will ultimately lead to a compensatory glass ceiling (licenses scale much better than hours) or a decision to move into a C-level role. This phenomenon is what Juval Lowy refers to as an “Inflation in Terms”, suggesting that titles or roles in software engineering don’t necessarily align with that of other engineering disciplines due to the white collar nature of software engineering1. This is an imbalance that software professionals, through organizations such as the IEEE Computer Society (Institute of Electrical and Electronics Engineers) are attempting to address through comprehensive education, training and licensing regimens including university training (which focuses more on problem solving than computer science), while placing a greater emphasis on recurring and continuing education so that software professionals might truly one day join the ranks of mechanical or electrical engineers.
This Inflation in Terms is quite benign compared to the bigger problem that plagues the software industry: There is an alarmingly high rate of generalization bordering on incompetence in the field and this is a problem that agile approaches to software seem to embrace when left unchecked. It turns out that in most organizations, internal support teams (for which CIOs are in a constant "build vs. buy" turmoil) are relatively innefective overall, and tremendously disspraportionate to some of the better shops out there (ISVs and boutique consulting firms for example). This is a problem that is attributed to the relative immaturity of software engineering as an engineering discipline, and thus, a lack of professional accountability. As a result, the lack of standard licensing, training and general education requirements for software engineers is a problem of pandemic proportions. Can increased communication between developer and customer solve this crisis? Of course not.
Certification programs from vendors such as Microsoft is a step in the right direction, but there is still much work to be done. One answer, typical in agile spaces to this dilemma is that software should be adaptive and evolutionary, with the customer participating in the solution. This is a somewhat impractical solution to the entire problem, because it implicitly suggests that the customer should pay for, or subsidize this OJT. This philosophy certainly works in organizations with relatively unskilled (or at least disproportionably skilled) employees where generalization of technical skills is fashionable, but not all organizations and projects can afford this rosy picture unless they are producing simple froms with buttons that do neat things like query a database (in other words, toys). My point here is that the best and brightest software professionals will both generalize and specialize, a concep summed up quite well as "generalizing specialist" or "Jack (or Jill) of all trades, master of some".
Software as an Engineering Discipline
If we consider the software industry's progression into an engineering discipline as a step in the right direction, some will argue that the idea of Agile practices is a response to the fact that software engineering is still significantly behind the curve as compared with other engineering disciplines, specifically, mechanical engineering. Some might assess that software, as an engineering discipline today is where mechanical engineering was at the turn of the 20th century. This isn’t a bad thing, it is simply a frank assessment of where software engineering is compared to other engineering disciplines which also experienced the same struggles along the way. This seems like a reasonable concept, since mechanical engineering, for example, has a couple hundred years on us.
If the above concept seems palatable, then it should also be noted that predictive, plan-driven approaches (commonly regarded as “traditional” approaches by the agile community) to software product development represents the attempt to bring a new level of maturity to an otherwise still immature engineering discipline. Agelists will assert that software is malleable and thus cannot be approached in the same way as mechanical engineering. This is probably true for most projects, but it is not for the malleability of the software as much as the people that are working on the project that introduces many of the challenges of working with software. I say most, because it is quite possible to apply a highly predictive (and accurate) manufacturing approach to even large system level projects. The Software Capability Maturing Model (or SW-CMM), and companies that have successfuly implemented and prospered from it would probably agree. NASA, Honeywell, Motorolla, Microsoft, and the U.S. Airforce are some examples of organizations that have been succesful, but I would bet my career that most of these organizations have also been highly effective with agile techniques such as Scrum wrapped XP, leading me to believe that it is no so much a process issue, but a human capital problem.
Mature Software Engineering Practices are Agile by Nature
Given the right teams, I think this manufacturing, or mechanization is quite tenable, again provided you have the right people. Architect(s), software engineers, analysts, etc. I know this because I successfully led a team that delivered an alpha release of a new inventory system to QA. The system represented an estimate in excess of 3,000 man hours, so was far from trivial. The system consisted of a significant up front architecture and yes, some high level design. Not only that, but it was built on the new version of the .NET Framework (2.0) along with 2 completely new platforms (Microsoft BizTalk Server 2006 and Host Integration Server 2004).
Yet, even with a new version of the CLR- that brought with it a new IDE- and two new enterprise platforms, the project was a success. There was (almost) just the right amount of up-front architecture and design. For a multimillion dollar project of this magnitude, the need for a competent architecture team that had the experience and the discipline to identify design patterns and pre-define, wherever possible, the components and their relationship within the architecture was a key factor in the project’s success.
Now, I am not saying that we nailed the design the first time. That was never the intention. However, for the most part, the estimates, which represented vertical slice aspects of the solution were accurate (within 2 to 3 weeks, which compared to other schedules is extremely precise) and the system that was delivered was very close to what we designed. As a matter of fact, we were only about 300 hours short of our original estimates and were able to adust and regress as necessary to meet the required schedule deadlines. Despite successfuly leading the engineering aspect of the system, the project failed. Why? Because what we delivered was technically sound, but wasn't what the company wanted because, in the 3 years since the project's inception, the company changed. It was a classic "This is exactly what I asked for, but isn't what I need" scenario.
Agelists might say that I was plain ‘ol lucky to even have delivered the system in the first place. The extent to which I will agree is that I was definitely lucky to have an exceptional team to work with and a set of pre-defined patterns, or building blocks that we agreed on and were able to estimate primitively at first and refine over time, quite accurately. I never said that this was easy, as a matter of fact, the upfront estimation took about 2 weeks and was quite arduous (it costs to live2, right?), but by the time I had rolled into the system architect role (after a year and a half of false starts), I had a pretty good idea of the terrain and mapped it accordingly.
Despite the system no longer aligning with the needs of the business, I am very confident that if I honed these patterns and the widgets that accomplished the problem domain typical of many EAI projects, and I were to architect a similar system, or a number of similar systems over time, my ability to estimate in a timely and accurate manner would increase as would the precision. I would confidently be able to go into an average organization, without inheriting the political and business process baggage and deliver a design and architecture that would be positioned for a high probability of success. Thus, mechanizing software construction is doable, although not everyone can (or should) do it.
This is where the lesson lies: Agility is much more than mere software engineering practices. Most decent developers better be doing things like unit testing, TDD, refactoring, builds and continous integration and be well versed in design patterns. I mean, this is a requirement. That said, it doesn't matter what you deliver, if it isn't what the company needs anymore you have failed, period. But, who failed?
Most Business Requirements are Inadequate
If the skills of software engineers are disproportionate, then I would suggest that the problem is as bad, or perhaps worse among business analysts and product owners. It has been quite typical in organizations that I have worked in (which include major U.S. banks and former Fortune 500 credit card firms) for business analysts to have career pathed into this role due to mastery of the business domain. This trait, while important, does not make a business analyst. A business analyst should not only know his or her business process inside and out, but should be able to quantify and illustrate this process via lightweight requirements and intuitive models such as Use Cases and UML diagrams respectively (at a minimum), but I have met very few analysts that have these skills. And often, this gap transcends vertically all the way up the organizational ladder, painting the rather acrruare picture of "who's on first?".
Agile approaches to building working software harnesses the competencies of ordinary people to do extraordinary things. I am not saying, necessarily, that ordinary people are mediocre, or incapable of becoming great software engineers or analysts, but I do think that Agile methods take into consideration the unique contributions of the different members that make up a software engineering product team. There is nothing wrong with this, and there are many benefits to applying agile techniques, not only to construction, but to business process and requirements as well. Where problems begin to surface is when generalization is favored unilaterally over specialization. When this happens, a team may inevitably waste several iterations reinventing the proverbial wheel when a competent developer or architect could have been enlisted to apply proven design patterns to a common problem or set of problems. Effecive agile teams understand this.
The Point is Organizational Agility
Again, after delivery, the project never went live. It sat in QA atrophying day by day, week by week and month to month. Resources were re-allocated. Resignations ensued. So this suggests that engineering concerns such as software design and construction is just a small piece of the puzzle. However, I believe that when we start applying the term “agile” to all aspects of software engineering, we lose credibility and more importantly, discredit the body of knowledge that is the bedrock of modern approaches to design and construction. I am talking about the Fred Brooks and Steve McConnel schools of thought that despite not being officially "agile", can and have been effective and serve as a bedrock of what we have today. Let's embrace this heritage.
Architects, consultants, and independent experts, both within and outside of organizations can be successful by applying a combination of traditional and evolutionary techniques to delivering a solution to a customer. In this regard, what agile technique really addresses is the esoteric nature of software and strives to make it both simple and humane for ordinary people to digest, while empahsising quality and communication, but applying XP engineering techniques itself is not a panacea.
To understand the goals behind becoming more agile, it is important to apply the percepts of the Agile Manifesto to the organizational level. Individual interactions, customer collaboration and responding to change are the foundational tenets that really benefit a business organically, perhaps by several orders of magnitude over the engineering process itself. Agile approaches to managing an organization or project can be, and often are transformational. Bill Curtis from Borland Software (Borland is now in the process space) says that when IT shops go agile, the inefficiencies and opportunities for improvement within the organization are exposed because failures can no longer be blamed solely on IT. Real problems with personnel, business process and culture start to surface that are either dealt with or ignored. If the organization that once embraced mediocrity (whether they knew it or not) makes the hard decisions to confront and address these problems, it can rise above it and become truly extraordinary.
This is a great start to making software better, but the journey is far from over. I believe that in addition to the benefits gleaned in software engineering, the lessons learned from agile approaches to delivering a product (not just software products) could prove to be a valuable asset to business and economics theory, changing the curriculum for traditional MBA programs as we know them (as a side note, Toyota applied many of these techniques quite successfully in building the #1 car company in America). However, the other problem with software evolving as an engineering discipline is still at large. Perhaps a side-effect of the transparency ushered in by agile organizations will be a newfound incentive to educate, train and hold information workers accountable, regardless of the white collar nature of software creation. This is just one of many steps vital to maturing into an engineering discipline and if we can get there through agile collaboration, coupled by predective approaches to applying mature technqiues to finding that "sweet spot", the road ahead may well be paved in gold.