A1181 Core Duo MacBook randomly restarts and goes into kernel panic

I have an A1181 Core Duo 2006 Macbook that randomly restarts when a known good battery is installed and finally ends in kernel panic. When I take the battery out, it runs just fine. Without the battery installed, I let it run for a few hours, browse the internet, go into control panel, all the normal things with no problems. But when I add the battery, the restart circus begins and then the kernel panic. One thing I have done is re-apply some newer thermal grease to both the cpu and the graphics processor which seems to have helped stabilize things. But when a battery is added, the restarts resume. I have a newer battery which I've charged and tested. Pushing the button on the battery shows all green.

My question is, would replacing the battery connector fix this? Since the macbook works fine without a battery, I don't think its a logic board issue. I've taken the battery connector out, blew compressed air on it and on the board where it plugs in but that in itself did not fix it.