The Shulchan Aruch (O.Ch.180) writes:
" You may not remove the tablecloth and the bread until after
The Levush writes:
"It is proper to clean off the table so you will Bentch at a clean table."
The Kabbalist (Chesed L'Avraham- Azulai) say not to remove anything from the table until after Birkat Hamazon. Even the leftover bones and peels etc.should stay on the table so the ניצוצות הקדושה (holy sparks) that are attached to them, will be come out through the Birkat Hamazon.
The Kaf Hachayim (Palagi) makes a compromise between the Halacha (Levush) and the Kabbalist) You should collect the leftovers from all the plates and place it in one plate which shall remain on the table until after Bentching and the rest of the table should be cleaned.