The more polite members of your guilt will greatly appreciate your reminding the rest what you expect of them when a speaker is visiting. There is no excuse for rudeness. Just try a gentle reminder. If that doesn't work you may have to call them out. It is not fair to the others to let it go on. JUST MY OPINION!