Something that's always slightly bemused me is that software development methodology
is something you never seem to hear discussed in organizations whose business is technology. Sure, product companies are certainly very interested in practices and tools to support good practices. (For example, product companies certainly care about testing practices.) But technical practices are kinda orthogonal to methodology debates. And I never hear about a company like Red Hat paying any attention at all to the latest fashions sweeping through the world of agile consultants
and project managers
. In fact, I'd be very interested to hear of a single example of a truly great software product that was developed (at least initially) according to a methodology.
I have a number of theories as to what the difference might be, but I'm not sure which, if any, of those theories are correct:
- arguments about methodology help keep especially boring projects interesting
- methodology is important in an environment where developers are viewed as interchangeable resources
- methodology is important in an environment where developers have heterogeneous ability/experience levels
- methodologies like
agile
are used by project teams to defend themselves against ignorant managers - good product companies just naturally do agile-ish stuff without really needing to think about it (or, if they don't, they die)
- methodology helps achieve predictability, but gets in the way of innovation
What I'm most interested in is why companies buy into this stuff without any real evidence that any of it is used by companies which are able to repeatedly build successful and innovative software. Or, does such evidence exist and I'm simply too disconnected from this world to know about it?
If this post sounds like snark, it's not. I'm actually genuinely interested...
Yes, I did just stick scare quotes around ;-)
The WebBeans RI is TDD! Mostly ;-)
Hehe, but TDD is not a :-)
I have also been wondering about these discussions. It is not only limited to methodology but you also have it regarding software metrics, effort estimation etc.
It took me some time to figure out what this was all about and I came to the conclussion that these discussions come up if there is a gap betwen the technical understanding of the managers and the developers. The manager does know that he somehow has to control the software development process in order to develop good software but he does not know what the code of good software will look like and how to write such code. So there is not much to for him.
What he does understand is methodology and metrics. So he can force the team to stand during his meetings, introduce a task board, write user stories and look happily at the line coverage report. It allows him to change (not necessarily improve) things without understanding what the developers actually do. Just things is not enough, they obviously have to have a well informed choice in which way they whish to change. This is where the discussion about methodology comes in. They are actually choosing the best way with the the software development.
This also explains why such discussions to not happen in technology companies: The gap is missing. In such companies the seniors usually do understand what the developers are doing and what ought to be done. If they see a deficiency in the development process they will address it directly instead of thinking about which method fits there department best.
Best Regards
Felix
* fixed teams (as a solution to our resource shifting hell) produce the software they can produce
* transparency in development progress (not doing big front-up plans that fail every second time)
* high involvment of product managers, so they know what can be done and what doesn't
To understand better why we have such problems: Resources are managed by product management in our company. They decide what fancy thing is to be developed and who they want to implement it. The problem ist: They never get finished with prototyping etc. As a consequence requirements, priorities and therefore resource assignment shifts every now and then.
To your point that agileness contradicts to technology innovation: That is not the case for us, because product mgmt. defines the products. We're there to support them and do the implementation of that fancy stuff.
What problem we still don't solve with agileness: Technological evolution. For example we still don't use a modern web framework or orm product but some 8-year-old self-mades.
But this is a different story with a different solution...
and this IMHO is a general feature of any process. there is a continuum of possible human behaviour in any situation (like a software project), from completely improvised and free-form (i.e., no process) to entirely directed (like robot-musicians in an orchestra).
agile practices are actually quite clever in providing some degree of direction at certain points in time (daily scrum meeting, sprint planning meeting, etc.) while allowing for almost free-form creativity in-between.
but you're certainly right that if maximum creativity is asked for, and the task is small enough (an important qualification) then a couple of intelligent, motivated, creative people with no strings attached whatsoever is likely the best route to success. it's how artists almost always work, for instance. and intelligent, motivated, creative people usually thrive in this environment, while the other 95% are a bit scared of it...
Gavin,
I always read your posts and I do agree with most of the things you say, but this time I disagree.
It may be wierd and unnecessary to have a set of projects methodologies when you have a excelent development team at your side.
But, in the real world, most of the developers are people who only knows how to write some lines of code, they don't really understand the meanning of developing that software, they only know how to do their stuff. And, if you don't put everyone in a room and explain exactly what they need to do to make the project fly, it just doesn't happen. To use scrum in a bad development team is not a good shoot. You will be the only one who knows what is expected from the team, and you will be overloaded of work.
So, it really depends on the quality of your team, I wish I could trust in my team to use it...
Regards, Mário from Brazil
I'm just a newbie project manager trying to introduce some development methodology in the company I work for. For a few months, I've studied a lot of things, like RUP, Scrum, XP and some others proprietary and set of processes. I've tried to identify which of them fits better to the team I'm leading and the needs of the company.
Our main problems: we can't estimate our work, we can't plan what should/must be done next week, the company's owners change the scope every day, we don't have testers, we don't have time to develop the tasks in the right way, and the team is not so good as expected by the big ones (they don't give us money to bring the very good guys).
My solution: methodology - there is a set of processes to be done. We need time to think about a problem, design it and develop. Then tests, tests and tests. So, we need a tester. If we know how to solve the problem, we can estimate how much hours/days we will need. Then, we can plan. There is a scope change process to... they can't change it how they change their underpants... Then, we can still doing our plan... Thinking about a problem, let us design it and write down what should be down. The dummy guys could be not as good as expected... they will just write the code to solve what is documented.
So... I found out this is the right thing to do.
In other company I worked on, the guys was better and the team was smaller. We work almost alone, so no methodology was needed.
This is my opinion...
It seems to me that methodology should not go in the way of innovation. Architecture (I mean real architecture, not our pitiful geek version of it) and construction makes heavy use of methodology and leave plenty of room to art (read innovation). I think it does not work in our field because:
Note that agile methodologies (reconsider everything every time and adjust) looks awfully like an acceptance of our failure as an industry to build an actual methodology. Try and explain Scrum to a bridge architect, he will stare at you with a blank face.
Experience and anecdote tells us that great chunks of valuable software can be implemented by really small teams - sometimes by individuals. The smaller the team the less the overhead of communication the greater the focus. Potentially astronomical productivity.
We all love idea the small business with big idea driven by big egos who cut code with miraculous results. There's often more madness than method, but who cares.
However, for one reason or another (energy, funding, acquisition, scale, boredom, success or failure) these situations tend to disappear over a few years. As the productivity drops off or the business grows, either the engineers or the managers feel the need to step in and make a change to the technology delivery strategy.
Very occasionally the change that comes about at these inflection points, is incremental, well managed and generates positive outcomes for the business and the engineers.
However, in my experience it seems more common for either the engineers or the managers go bonkers and screw the whole thing up for an extended period of time until one party manages to purge the other.
When the managers gain the upper hand, it's all about product marketing, waterfall type product delivery cycles and nothing to do with engineering; and when the engineers gain the upper hand it's all about the need for agile processes and screwing the phb's.
The curious thing is that often the post-purge organisations calm down and become a lot more stable and pragmatic. Perhaps the pop start talent has walked, but invariably room opens up for more modest and yet still productive types, who normally work reasonably well with a little process, some multi-displinary collaboration, a touch of project management and only a modicum of bitching.
In short, the software organisation evolves into something that resembles a more traditional engineering business.
I am reasonably confident that achieving this steady and potentially prosperous state depends more on effective and respectful talent management than any particular technical/development methodology.
Of course, after a while you get some more popstars into inject a little life into the place and the whole things starts again. But if you've made it that far, who cares :)
This is a topic near and dear to my heart. In looking back at my time in internal IT organizations building custom software to run those businesses (over 21 years of time), I found several things to be true.
Methodology exists because managers, both within IT and the business owner management aren't qualified to do their jobs.
Methodology exists because too many of the developers aren't qualified to do their jobs.
Methodology exists to be able to give status updates to managers that aren't qualified to do their jobs.
Methodology exists because .
What really works is a small group of qualified developers working directly with end users to build something truly useful and valuable. I have seen many successful examples of projects, even really large scale ones, but in every case it involved a small group of qualified developers working with end users directly. I think this is why open source development is so successful. There are no layers of management, business analysts, blah, blah, blah, in the way between qualified (meritocracy of open source) developers and end users (the community).
I recently had a discussion with the CTO of a very large company (everyone would know who they are if I were allowed to say), about this very topic, because he saw a need to replicate open source development within the four walls of his company. As we talked about the things I listed above, I saw him and everyone that worked for him shaking their heads in agreement.
In thinking about this, I just remembered a perfect example of what I stated above. In my previous company, the managers for development did an analysis of just where they were spending their time during the software development life-cycle. The idea was to look for opportunities to improve time to market for software projects. It turned out that 50% of the time was spend in analysis and design. About 25% in coding, and 25% in testing. So guess where they chose to optimize?
That' right folks, coding! What idiots! It's one of the reasons I had to quit that job (even though I didn't have another job lined up yet). If I think about it, I could give countless examples like the one above. The incompetency levels within most companies is just staggering!
So... what methodology (if any) was used for Seam and Hibernate? They are large, complex and innovate products. How large were the teams for those projects? Were the teams maybe too small to think about methodology?
If there was no officially sanctioned methodology, what elements of rigor/discipline (if any existed in your opinion) did you use to help with quality? Eg was work peer reviewed? Did you have any coding standards? How did you decide how test coverage would be achieved if test coverage was important to you? How did you decide who did and didn't work on the project?
Christian
I think there's also a difference between a technical team that are creating their own product, and the more corporate form of development in which developers are providing a service or delivering a solution to . This is even more pronounced when the developers are outsourced to say, IBM - they need methodology try and ensure the client ultimately gets what they wants.
Getting a team of people together that you know and trust - and working together on developing the is a completely different situation because the developers own the product... and don't need to answer to anyone like they do in the first instance.
The further the product owner / sponser is from the developers the more control you need around it...
Alistair
Not really any methodology as such. I guess we basically just think about the problem, and then write code to try and solve it ...
Generally not more than 4 developers at any one time - and like most open source projects there is a very strong culture of code ownership (sorry XP).
Not through any formal process.
Not really, no.
Mmmmm, I guess we just make sure we write tests for things if they break...
I've never paid much attention to coverage tools...
By looking at the patches they wrote and submitted, and hiring people who were so capable and motivated that they were already submitting good code.
hmm... doesn't Google use scrum?
Do you know any company that repeatedly built ? There are always up's and down's :)
Hehe, at least agile admits it! :) And it's not just that it's also trying to improve on failures (reflections in scrum)!
Btw, this is one of the most cited 'wrong comparisons' of software architecture and real architecture. You just can't do that kind of comparison (and yet you do :) ). Resources, factors and requirements involved are just plain different (dimensions if you will).
Anyway, here an interesting blog how you 'could' compare (read twist) bridge and agile analogy: http://agiletips.blogspot.com/2008/07/agile-bridge-analogy.html
Well, for open source projects it's kinda difficult to follow some methodologies especially if the team is distributed all over the world.
Anyway, as you said (and I'll turn it around a bit), predictability might be sacrificed in favour of innovation. If it's a concious choice it's one thing but if not may be there is a problem (ahead).
And let's not forget about
Perhaps Adobe have a better hit rate than most: longevity, innovation and quality. I wonder what methodology(ies) they use ...
This is the one that really gets under my skin - the assumption that talent is somehow a non-factor in our industry. It's really no different that sports. You would get laughed at if you suggested that a coach replace a key player with just any old sub. In sports, talent is so ingrained into the process - it's in your face. In IT, the perception is that it's just not there. One of the essential attributes, as it relates to methodologies, is personal discipline. The Boehm/Turner book does a great job of addressing this. When you have personal discipline you don't have to be managed, you take ownership and you get things done.
Scrum is not a heavy weight methodology, even though we try and make it into that (The Decline and Fall of Agile). Scrum is really just a set of lightweight project management concepts that mainly helps facilitate communications and prioritize features - no different that using Jira to create a packaged release of functionality by listening to what people are talking about. The essential success factor with Scrum is having a great set of talented self-organizing team members. When a sprint starts, Scrum dictates that the everyone who's not committed (chickens) gets out of the way and let's the committed team members (pigs) get to work without adding new things. In other words, let the talent kick in and let the team produce. The (ScrumMaster) is only really there to remove impediments. That my definition of a leader - somebody who's interested in helping others be successful. Unfortunately, popular culture is that the PM is there to boss people around. Boo.
As the aforementioned blog post clearly communicates, engineering practices cannot be neglected. Engineering practices are not really methodology. Gavin mentions several of them above: Frequent releases, lots of integration testing, etc... If you have no talent, these things get left out and you have a hollow process pushing a herd of developers toward failure.
People matter.
My own experience is: companies whose business is technology live by selling that technology to somebody else. They don't earn more out of having great products with outstanding quality - they earn more out of having a customer who won't be able to complain much, usually because that customer is locked inside a fixed-specification contract. If you sell technology, you don't get the most by adapting to changing requirements. Instead, you get the most by freezing the requirements as early as possible, even if that means having a product that matches the initial requirements more than the real needs of the customer. (This is not the case for RH - I work for them quite often, and I know they're committed to quality. But I found it to be the case in general).
OTOH, some companies develop technology for themselves. This is the case of the Company I'm currently consulting for: they're an Internet shop, so technology is key to their business, but they don't make money out of it, they make money out of selling fashion. The shift to better ways of doing software (call them the word if you like) improved their technology, and in turn it improved their business.
In the above I agree with all the things what you have said. Being a new person to visit this I had collect a lot of information which is valuable to me. I think more projects should be taken and improve it. Each a every person does not the use of the software on what he working he just do the coding.
john
Link Building