When people think of freedom they often consider the freedom 'to' do anything.
With money and time they could do anything.
But the other part is freedom 'from.' It is more important.
Freedom from highlights the freedom from requirements on your time or space or other responsibilities.
For example, many movie stars have plenty of time and even money, but often have to be mindful of their careers and public image. Additionally, they do not have freedom from paparrazi or the press. On the other hand, a relatively anonymous millionaire would have potentially time and money, as well as freedom 'from' public perception.
That said, sometimes it is worth it to have a public image, to be well known, and other things.
But better to be well known within a community, than to just be well known vaguely by many people.
This distinction might be why there is that Bill Murray quote exists (roughly stating that if you want to be rich and famous, maybe try being just rich first, it might be everything you want).
That makes sense to me.