Headphone port loosened and wont work with any headset

I accidentally broke my headset jack while it was plugged in my controller and now anytime i plug a headset in said controller it has to be at a specific angle and only the left side will have audio. Is there a fix to it? The headset itself kinda works with other devices except the controller.