While this is true to an extent, from experience this line of thinking has its limits and is very easy to misapply. On the one hand, yes you can tell people their ideas do not gel with the vision of the project, and sometimes that's the right call. And sometimes doing this a lot is best for the project.
On the other hand, even if a majority of the work is coming from one person, not only does your community learn your project, they also spend time contributing to it, fixing bugs, and helping other people. I feel it's only to a project's benefit to honor them and take difficult suggestions seriously, and get to the root of why those suggestions are coming up. Otherwise you risk pissing off your contributors, who I feel have the right to be annoyed at you and maybe post evangelion themed vent blog posts if you consistently shut down contributors' needs and fail to adapt to what your users actually want out of your software. And forking, while freeing and playing to the idea of freedom of choice, also splits your userbase and contributors and makes both parties worse off. It really depends on the project, but it pays to maintain buy-in and trust from people who care enough to meaningfully contribute to your project.