Since many of the specters are suffering the effects of curses from the forest, even if they were only near the edges, it would probably actually help them. They're still pissed off unfortunately and would just be like haha, now I'm stronger!

On the bright side, the energy that a siphoning wizard would gain would make it easy to blast the specter away before it continued its attack.