Is it possible for magnets to act as the contacts for a speaker?
Possible - yes.
Not sure if it’s a good idea though.
Depends on what you mean I guess…
magnet-to-magnet contacts tends to not work well because small bumps can cause brief disconnects. Magsafe and similar contacts uses pogo pins for contacts, and magnets to put pressure on the pogo pins.
Speakers are coils, that means that when you disconnect them while they are charged, they will try to increase the voltage to make current flow. This can cause sparks, which can kill electronics. (This is basically how the ignition coil in an old-fashioned car works.)