Guilt is too damaging an emotion unless you have deliberately done something hurtful to a person. Enjoy a little time in the garden, but do what makes you happy! I walk in the morning, garden late in the day when it's cooled off, and quilt in between whenever I can.