How can I keep people from connecting to my big jam box?

First, your JamBox speaker can only support one device at a time. So it can’t be accessed by someone else while your phone or computer is connected and sending music to it (or gain access to its microphone). Basically, Bluetooth 2.x is a one to one pairing.

Most 2.x to 4.x Bluetooth speakers have this limitation as they don’t support the higher Security Modes.

Some speaker makers have offered extensions with their speakers which can lock them to a device. Basically, they set an encryption key within the speakers ROM the phone or computer needs to handshake with after the establishment of the BT connection to enable the speaker to that connection otherwise the speaker refuses the connection. This is part of Understanding Bluetooth Security using Security Mode 2.

Sadly, Jawbone is out of business and there speakers won’t offer the higher protocols needed to protect you from an unconnected speaker if you disconnect it from your phone or computer without turning it off as well.