Caulking applied to the seam between the door trim and the sheetrock can tear the paper on the sheetrock when prying the trim off. If there is such caulking, it's wise to slide a sharp utility knife along the seam to make a clean cut through the caulk down the full length of the trim before prying.

For best results, place a shim between the hammer and the wall to minimize damage.