I think you might be hooping the fabric and stabilizer too tight, it looks like you are using a tear away stabilizer and the stitches are tearing the design away from the stabilizer. You might try a cutaway stabilizer while you are adjusting your tensions. Your design does look like it is poorly digitized, a lot of machines have a built in design to use for tension adjustments. The design will look like a wide line, or maybe several wide lines. Maybe your machine doesn't have it. The book should have a section on tensions with photos on how it should look.