Public speaking is a skill not many poses and it's hard to develop, you need to learn to read the crowd and engage every single person no matter the size. I love public speaking but alot of people call me a freak. I'm also able to walk upto anyone and start a conversation. I'm even able to engage with someone on a personal level with in the first few seconds and few words that are spoken. Males me extremely effective at my job.

Your fears are common but as you practise more and learn you will become great at it.