At 4.5mm you're fighting two separate problems, and they need different fixes. First is the bunching, which usually means the holes aren't doing enough work for you. On leather that thick, I stopped trusting punch-style irons and switched to marking with a pricking iron, then opening each hole with an awl angled to my stitch. The awl lets the needle slide through instead of forcing the leather apart, and that alone cleaned up my tension massively. Thick veg-tan will distort around a punched hole if the tool isn't perfectly perpendicular, and once a few holes lean, the whole line wanders. Second, skive the seams. Not the visible edges, but where two or more layers stack up before you stitch them. Take the underside of the top layer down by half or so over the width of the seam. You lose nothing structurally because the linen thread is carrying the load, and the leather stops fighting you when you fold and clamp it. If your corners are still stubborn after skiving, dampen them lightly and let them sit under weight overnight before sewing. Trying to persuade stiff leather flat mid-seam is exactly where stitches go crooked. A couple of smaller things that helped me: shorten your stitches on the thick sections. Long stitches look fine on thin lining but they exaggerate any wobble on heavy panels. And wax your thread properly, really drag it across the block a few times, because unwaxed or lightly waxed linen slips and lets stitches loosen between pulls.
Also, get the work clamped in a stitching pony or lace it to the bench edge. Handholding thick panels while pulling tension is nearly impossible, and you won't notice you're tilting the needles until the seam starts drifting. One last thing, don't chase perfect tension by yanking harder. On leather that thick, consistent moderate pulls beat hard pulls every time. Hard yanking is what makes thread bunch at the back side. Same pull, same angle, every stitch.