OK, I think I found a fix!
I made a few changes to the source code, I've attached the files that have been updated, it should be placed in the src folder and you will need to compile primecoind yourself
Been testing it for the past few hours, seems OK now, it is SunnyKing's version that I've compiled and not HP version, but I wasn't going to mine on my server anyway
I'll test for a few more hours and see how it goes hmm...
EDIT: Spoke to soon again, same issue. Will make some more changes and keep testing, I'm not exactly a coder lol