People are creatures that judge quite easy, mainly based on rumours that influence their emotions. Thus, they (mostly) say something without knowing the real facts.
So you might have heard something like: "Hey, don't go to [NAME PLACE]... No I haven't been there, but [EXPLANATION BASED ON RUMOUR]."
I personally trust my guts, which will tell if it is a good idea or not to travel to a certain place.