I’m wondering if maybe we’re hitting some protection circuit in the amp. Since we push the amp pretty close to it’s limits, this could happen if either the speaker is in fact lower than 4 ohm, or if the amp is just a bit out of spec. If that’s the case, a reasonable solution might be to add a small resistor (1-2 ohms) in series with the speaker. (And possibly push the volume a bit higher to compensate.)
If this doesn’t help, I might be out of ideas for how to fix this.