There’s a Pareto effect when it comes to them, in that you can cover a large proportion of use cases with a small amount of work, but the more special cases consume proportionately more effort. For a MVP, you could restrict support to standard USB and SATA devices, and get a device you can run headless, tethered to the network through a USB Ethernet adapter. For desktop support, you’d need to add video display support, and support for the wired/wireless networking capabilities of common chipsets would be useful. And assuming that you’re aiming only for current hardware (i.e. Intel/AMD boards and ARM/RISC-V SOCs), there are a lot of legacy drivers in Linux that you don’t need to bring along, from floppy drives to the framebuffers of old UNIX workstations. (I mean, if a hobbyist wants to get the kernel running on their vintage Sun SPARCstation, they can do so, but it won’t be a mainstream feature. A new Linux-compatible kernel can leave a lot of legacy devices behind and still be useful.)
It’s possible though less than ideal. Drivers that connect to devices are part of the attack surface, and probably the part you’d least want implemented in C when the rest of the kernel is in Rust.