Jeff Atwood writes Coding Horror, one of my favorite technical blogs. In two posts he asks familiar questions (remember, he's from the XP world): Coding Horror: Why Does Software Spoil? and "Are Features the Enemy".
I think the answer is pretty straightforward. The way we pay for software induces perverse incentives. Here's the comment I wrote to his blog:
It's the business model.
If we rented software then there'd be a steady revenue stream for developers, and a lesser feature incentive.
Since we buy software (sort of) developers have perverse incentives. They have to "break" the prior version by not maintaining it, introducing new file formats, and they have to add features to hide the fact that the updates are driven by vandalism, not value.
Of course software rental completely screws customers unless the file/data formats are completely public and interoperability is assured. I'm sure customers will realize how important that is.
Thirty years from now.
Viruses and worms now help drive this model. Security issues make it much easier for companies that own both the OS and the software products to "break" older software, thereby forcing updates. Feature addition is simply a way to make customers accept the forced update. If Microsoft didn't add features, but forced Office updates by making old versions run less well, customers would scream.
These kinds of things don't really have to be planned. They're simply emergent. It's just the way natural selection operates on business.
Of course the only thing worse than the current business model is the combination of software leasing and proprietary data formats, including proprietary metadata models. (Ever try extracting and moving all your iPhoto data?)
Happily customers would never fall for that trick.
Oh, wait ...