You rightly said, the fsm is used by the compiler for DTC available in the 7265x. The fsmfiles are the source code for the DTC plug-ins and the .hex are generated output in op-codes. You can open the xx_fsm.hex file in text format to see the opcodes.
These files are actually included as an array in the source code, which are then loaded in the DTC RAM for execution, when required.