If we're using the Governor's first attack on the prison as reason why Woodbury is easier to defend than the prison then I say that's weak evidence. The only reason they got bushwacked is because they didn't have anyone on lookout. I will also point out that Woodbury's defenses were just as easily breached by Rick's group and Woodbury had people on look out.
Also, I would argue that Rick's group wouldn't have withstood the Governor's 2nd attack if they were in Woodbury. The buildings wouldn't stand up to the grenade launcher and the 50 cal. At least in the prison they were able to fall back into the tunnels so the Governor's forces were not able to use the heavy weaponry against them.