Oh please do not use glue on something that could/should be edible!
Instead of glue ... melt some sugar in an electric frying pan.
Then dip the edges of the graham crackers into the melted sugar.
Press edges together to "glue" the graham crackers ... these might be at right angles for corners of your houses, or it might be on the flat, if you are wanting your wall to be longer than the cookies are.
The sugar/glue will be brown. So to cover that up, you can ice overtop of the brown.
Then add on your candies as you wish.
I have made them this way for years. You can make small mini homes ... or giant castle like ones. Whatever the imagination wants. And sometimes, it's been a village of all sorts of different houses, churches, barns etc.
This way you have fully edible houses and everyone can enjoy it, when you are finished using it for decoration.