I've made thousands of button holes. First thing to do is quit working on the bag.. Instead make up a sample using the same fabric, same number of layers, interfacing, whatever you'll be sewing through.. Practice on that until you can get a button hole you like.
I mark my button holes on scotch transparent tape.. I lay the button on a piece of tape and make a short line on each side of the button.. then connect those two lines with a line through the middle.. That gives you the perfect size for your button hole (unless you are using a very thick button, then it needs to be a tad longer) If your bag is stretchy, like fleece or a knit you'll need to put a bit of stabilizer inside to help keep your fabric from stretching. I usually stabilize most any button hole anyway, it just gives them more strength. Next place your marked tape exactly where you want your button hole.
If you are making the button hole manually, then you'll need to set your machine for a very narrow zig-zag, and a very short stitch length.. zig-zag next to the center line from one end to the other. Lift the pressure foot, and move to the other side of the line, zig-zagging down the other side of the line, but leave a couple thread widths open down the center.. Stop when the lines are equal in length.. Next set your sttich length to "0" and the width to be as wide as both lines of stitching.. You'll be making a bar tack on the end and want the ends of the lines covered.. When you get the width wide enough take about 5 or 6 stithes back and forth.. end by turning your stitch width to 0 also and take a couple of stitches in one place to knot your threads. Go down to the other end of the button hole and bar-tack that end also.. Stitch right through the tape.. It will pull right off when you're finished and makes it very easy to see just exactly where your button hole is going to be and just how long each side needs to be...
I agree, if you can find an old Singer Buttonholer attachment, it is well worth the price you have to pay for one.. They are not expensive and make very very nice button holes.
If you're still having trouble I would suggest you try a new needle... make sure it is the correct size and type. Also make sure you are holding your threads at the back of the machine when you start stitching.. You can also loosen the top tension a tiny bit to insure that the top thread is pulled to the underside, makes the top look very smooth.
Sorry this is long, I hope it's helpful. If none of those things help, try taking your bag to someplace that does alterations, 1 buttonhole shouldn't be too much and may be cheap compared with the aggravation!!