Vibrations would be the biggest issue, you’ll most likely create a lot of false clashes and stabs unless you really insulate the board from the speaker.
The magnetic field from the speaker should not be an issue, but false clashes might be. A little bit of some soft material between the speaker/chassis/board might be a good idea. Lots of people have put the speaker next to the board and made it work, so I don’t think it should be that hard.
As for magnetic field interactions, IF it were causing issues you could put a large chunk of aluminum between board and battery. Aluminum tends to disrupt magnetic fields for some reason I can’t recall, and is often used to make “On/off” switches for magnets. It doesn’t stop the field, but it does get in its way and block it, if that makes sense. Im not totally confident that this would work to isolate the speaker from the board, but its the first thing I would try testing out if I were having problems.