Sometimes people start talking because they are hard of hearing and can't keep track of the speaker if there is no public address system or loop system to assist their hearing. They then start talking, and again because they are hearing impaired, they don't realise that they are talking too loudly for the rest to hear comfortably. If you could discuss the need to purchase a microphone or easy listening device with your guild members because of this problem (it is more appropriate at a meeting when your guest speaker is not there), it could solve the problem. The 'rude' people will be made aware of your thoughts without singling out anyone in particular and, if finances are tight, everyone will be aware that they don't want to have to buy extra devices because people can't keep quiet for 20 minutes or so.
Also, I suggest you always have the speaker as the first item so that everyone is fresh.
Best of luck.