You need to trim both sides - make sure you line up the seam with the diagonal on the ruler, otherwise you'll loose your points.
I use my easy angle ruler, and cut triangles from a strip that omits the top point. For a 4.5" HST unfinished, I start with a 4.5" strip of each color.
When I sew them together, I rarely need to trim.
Though if I'm doing something that ends up less than 2.5" unfinished, I prefer to cut a bit larger and trim down. Smaller pieces are harder to keep square when sewing.