I think it will make a difference as to whether you use 3" or 5" blocks. Leaving 3" of fabric without stitching doesn't allow that much area to separate from the other layers when it's washed or used. 5" however is a much larger area for there to be a difference and you might experience some distortion of the fabric on those unattached layers from the squares that are attached together as it is washed and used.
The reason a quilt is tied or quilted is so the batting doesn't shred and fall apart because it is not a woven product. Flannel however is woven so it won't come apart like that. Technically you could make the whole quilt and not quilt it if you don't use batting.