So I took two quests from Nivarian but ended up getting sidetracked and completing the Abbey instead and failing them. The first one I'm not sure about, though I believe it also gave no rep malus, but the second one I am absolutely certain didn't reduce my rep on failing it.

Not all THQs lead to a reputation loss when failed. For example, Ilmaran bounty hunts are save in this regard, while Ilmaran relic hunts have a reputation penalty.

A bit of mixed bag....

Ah well then, they were both bounties. Any reason why it is like this?

I noticed some weird bounty behavior a while back regarding reputation and assumed that some bounties were either bugged or possibly rounded down reputation loss to 0 if your initial reputation was too low. Here's an older thread with similar conversation: viewtopic.php?f=4&t=14478 . In Lannegar, goblin bounties didn't reduce reputation but tome bounties did, even though they both provided a message on screen saying that reputation had changed.

Read above, apparently Ilmaran Witch Hunts dont give a loss of rep on a failure. And either way it wasnt me getting a message of rep loss but no actual rep loss. It was just no message at all.