The Olog-Hai trolls should be able to break walls and ground when fighting. Their hammers are heavy and Olog-Hai are strong, so they should be capable of doing this. That would make it impossible to hide anywhere from them and also add some realism to their working.
I personally think it would be better to add another troll unit. The Olog-Hai would be the main unit vs unit Mordor Hirable unit, and then a "Siege-Troll" or something could be added as a special unit that breaks blocks.