The historic setting of Richard III’s childhood home Middleham Castle could be spoilt if plans to build 55 homes on nearby pastures are given the go-ahead, conservation groups have claimed.