I was confused by what a FIT actually is. To clarify,

From the article:

A Flat Image Tree is the compiled Linux kernel paired with the associated DeviceTree content that is compressed and easily then distributed and executed by capable bootloaders.

From the ARM OSS Wiki:

During the Linux boot process, a “Device Tree Blob” (DTB) file is loaded into memory by U-Boot / UEFI, and a pointer to it is passed to the kernel. This DTB file describes the system’s hardware layout to the Linux kernel, allowing for platform-specific code to be moved out of the kernel sources and replaced with generic code that can parse the DTB and configure the system as required.