The cost of maintaining your optimized code adds up too. As does the risk that you introduced a bug. And this takes time away from newer features that customers might be willing to pay money for.

Occasionally performance is a feature, but a lot of times it's just an excuse for developers to have fun writing assembly.