Hmm ... I have a small baggie of some of the leftover cuts that I decided not to use when I made a wall hanging using a glue stick. Most of the cuts have been glued before storing in the bag, but they were not washed before they were put in the bag. They've been in there a little over a year (15 months to be exact). I just checked and not even the light colors have yellowed.
BUT .... I agree with Adamae ... even if the glue doesn't yellow, it could promote some of the other problems listed above (bugs, deterioration of the fabric). I made a large wall hanging and used a LOT of the glue stick on it ... I went through several sticks!! I did however give the wall hanging a wash when I was done (a good long soak - no agitation - gentle soap). I dried mine, but you can choose to dry it flat.