You can actually run thru the underground lava tunnel into the fort and over to the castle, assuming you have run speed and some form of quick or instant invis, and can take a few hits. These mobs will not follow you into the field and they will lose aggro once you get far enough away.
It's still probably easier to jump the wall but this is a good backup plan!