When they are an irritation to use and/or store, rather than giving you pleasure.
When I get down to the remainder of a piece of fabric I cut it into 2.5" squares. Less than that is out of here.
I won't live with scrap bags and boxes, that would seriously drag me down mentally.