After tearing into the engine (removed hood, various piping, intake manifold) I saw what was causing the problem. The issue was in fact the actuator on the turbo. Specifically, the circlip that holds the actuator on to the vane lever had broken/fallen off, and the whole assembly was wedged together. I went to the auto parts store and bought a kit with a bunch of spare parts including assorted circlips - one of which fit.
The assembly seems to move quite easily and there is no rust or dirt, so I think there is no point in going any further. I tested the actuator with a vacuum pump and it seemed to be fine. I did notice some loss but I'm hoping that doesn't matter. I tried to make a better seal around the actuator vacuum port by using a larger piece of rubber hose and a hose clamp.
I am almost done putting the car back together, but I did notice that I seem to have lost two bolts out of the intake manifold. I called the dealership and, of course, they don't stock intake manifold bolts (which is pretty much confirmation of my suspicions that they don't stock anything for outside customers). So I ordered the bolts and they should be here Saturday.
About 1-2 hours of work and it should be back on the road.