I've heard mandarin fish are difficult to keep, mainly due to their diet I think? Just how difficult are they?

Their diet isn't a huge issue if you're okay with spending extra money on buying copepods (their favorite food). A lot of people will set up a refugium to breed copepods in. Or alternatively, you could just buy copepods on a monthly (possibly more frequent?) basis.

Be aware that the other fish in your tank will eat the copepods as well so that's why you would probably need a refugium to breed them in. Otherwise their population would be decimated too quickly.