Not every package that comes with xfce has a name that starts with xfce4, just most of them do. You may need to identify and remove other packages too.

I think that: sudo apt purge xfce4* sudo apt autoremove

should do it.

I’ll point out that the other answers here are also correct. It depends on how you want to clean it from your system.

“apt remove” will only remove the packages, not the config files
“apt purge” will remove the packages and config files
“apt autoremove” will clean up the orphaned dependencies
“xfce4” will only remove the DE
“xfce4*” will remove the DE and most of the other packages that come with xfce