If the auditorium is empty who's fault is that? If the audience chooses not to listen, they are exercising their freedom to choose. I see many people that claim to love freedom but seem to love only their own freedom, and not the freedom of others. Then they confuse the empty auditorium with censorship, but those people didn't stay home because they were afraid the government would imprison them! they just had better things to do!
If the government doesn't throw you into jail for publishing ideas they don't like, that's freedom of speech. If no one buys your book, or they say "this book is rubbish", you still have freedom of speech, your speech just didn't appeal sufficiently to an form an audience.
I wouldn't say that freedom of speech or freedom of listening is better than the other, but just that modern communication tools have explicitly enhanced freedom of speech more than they have freedom of listening, so there is some catching up to do on the listening side.