After trying to use a flat head screw driver and getting about 1 out of 3 inserted without breaking the edges that the screw driver sits in, I looked around the garage and came up with this.
Take a bolt the same size and thread of the insert, just long enough for 2 nuts and a washer with about 1/4" at the bottom to grip the insert. I put the washer in between the two nuts and threaded them onto the bolt.
Then just screw the insert onto the protruding end of the bolt until it's flush to the bottom nut, get out the ratchet and ratchet it in until the washer in the middle is flush with the wood. That puts them at the perfect depth to grip both the spacer and the wood that the spacer is mounted too.
:edit: Alright, pictures added finally. Not to great with the zoom on my phone, but hopefully you can get the idea

Then you just back the bolt out of the insert!
About 1 in 6 or so times the insert comes out with the bolt, but then you can re insert with a screw driver easily without breaking the edges out. I'm guessing a little lube on the bottom nut might keep this from happening.
I took some picture I'll post shortly.