Being on several boards, I totally agree that you need to have it in the by-laws. This stops most hurt feelings. If you don't have by-laws, make them. They can be as simple as one page stating what your group's mission is, when ad where you meet, how and when to elect officers, requirements for those officers and how to add or remove officers. At a later date you can amend and add more.