You could try it mate, it won't do any harm, I have 2 layers of Dynamat (it may be 3 but I can't remember)
on my bootlid, plus another on the back of the number plate to help stop rattles. I'd suggest get your system
in, with Dynamat on the front doors, and then see what problem you have with the boot :)
As for the subwoofer box, is it ported or sealed ? It is always best to follow the manufacturers recommendations.
You mention the box is 1.25cu.ft, have you measured inside the box to confirm this is accurate ?
Have you also taken the displacement of the subwoofer when fitted into the equation ? By that I mean the area
inside the box that will be taken up by the subwoofer magnet/chassis/cone etc when the subwoofer is fitted.
You might be closer to the ideal volume when this is taken into account :)