If no enemy character steps forward to meet the challenge, one of them must retire in ignominy. This character is nominated by the challenger (though they may not nominate a character that could not have accepted the challenge). The retiring character slinks off to the back ranks and is not allowed to attack that round – move the model into a rank where he's not in base contact with the enemy. Another model will step up and fight in his place, just as if he'd been slain. Furthermore, the model's Leadership cannot be used for any Leadership tests that take place that turn.
Once a challenge has been refused, the issuer can fight normally in that round of combat.