Renee Hopkins has a compilation of notes from the Innovation Convergence conference.
[One of the presentations was on] Customer-Centric Innovation: Turning Consumer Pain Into Innovative New Products by Tom Kuczmarski and Scott Lutz.
Tom quoted a 2003 best practices study his company did: 85% of CEO respondents said conducting customer problem/need identification research prior to ideation is the most important driver of new product/service success in their organizations.
A main reason why research for new product development should focus on consumer needs and an understanding of consumers lives rather than product and service attributes is that the resulting ideas are more likely to be true breakthroughs.
This makes absolute sense to me. If you focus on needs, youll come up with new products that meet those needs. These products may or may not resemble current offerings, but at the very least they shouldnt be so far out in left field (a common problem with unfocused new product development efforts) that they dont still meet those needs, since that was the objective.
On the other hand, when you focus on researching what consumers do and dont like about an existing product, the best you can expect is incremental improvement suggestions.
One more point Tom made about starting with pain your new products are more likely to be profitable if they enable the solution to a problem on which consumers place a higher need intensity.