Vibrations would be the biggest issue, you’ll most likely create a lot of false clashes and stabs unless you really insulate the board from the speaker.
The magnetic field from the speaker should not be an issue, but false clashes might be. A little bit of some soft material between the speaker/chassis/board might be a good idea. Lots of people have put the speaker next to the board and made it work, so I don’t think it should be that hard.