Well, there aren't any quilt police, but the odd block will not "fit" in the pattern correctly. You will have to fudge and distort it or the other blocks to get it sewn in. Better to either resew it or make another.
Think of it like this: If you had a puzzle with rigid pieces, would one that is a quarter inch bigger or smaller than all the rest fit? No. Just because this is fabric, and can be stretched/gathered/etc, it still won't look as good.