Is the yellow blob on the second photo the weird flux you're referring to?
What I'm seeing is that some three hole component appears to have been glued down (the yellow stuff) and has been removed. Whoever did the removal did scrape or tear the solder mask off when removing the glue; the streak of glue on the far side of the torn solder mask tells me the glue probably covered whatever that part was.
As far as the torn solder mask, that's just a big ground plane and shouldn't cause any problems unless some component comes in contact with it, which generally isn't likely. You can always order a small tube of solder mask, but frankly a piece of electrical or Kapton tape or even a coating of fingernail polish will do the job if it concerns you.
After a bit of poking around, that component is one of the three barrel plug sockets located on the top edge of the Lynx. You can see what it should look like in this picture.
After a bit more exploration, it appears that's the power jack, which would certainly explain why it won't power up, although it doesn't explain why it's not working on batteries. Just based on pictures, I'm fairly sure this is the part.
Power jack socket for Atari Lynx 2 - ZedLabz