The sight of empty rows of seats when you are giving a speech is a serious drain on your energy, and a constant distraction. It will also make your audience members think "why didn't everyone else turn up?". The problem is usually caused by enthusiastic organisers who want to ensure that everyone will have a seat, and therefore not only put out far too many chairs, but also provide much too large a room.
That's why you need to arrive early, and ask the organiser how many people are expected. Then count the chairs (yes, do it yourself). If there are obviously too many chairs, request that some are removed. Point out the disadvantages of a half-full room, and the fact that chairs can easily be added at the last minute. If you are able to, it may be possible to switch to a smaller room, or to change the layout from theatre-style to cabaret-style (circular tables with about ten chairs around each). In addition, a few people left standing shows how popular your talk is, which will reflect well on the organiser.
You might also wish to change the layout to remove a central aisle (one thing I always do if I can). Most importantly, you should take charge of the room, so that you and the audience feel comfortable, and have no distractions while you are speaking.
If you can't make any changes, try putting any handout material on the front rows only, and/or have people escorted to their seats at the front. If you end up with a tiny audience, ask them to re-arrange their chairs in a half-circle in front of you. Never, ever make an excuse like "I don't know why so few people turned up". Simply present your speech as though the audience is exactly what you expected.