So I was doing the quest where you need to go into the Grey Library and read the book to be transported to the Mountains; I went in, triggered the guards, killed one golem then left the building, went and rested, came back, the other golem attacked me, but the Conjurer wasn't aggro towards me. I went up to talk to him and there was just the normal dialogue about the necromancy tomes and asking why he was there, but he doesn't attack! All the golems are dead, so they don't reappear. Poor lonely conjurer!