Found out what is wrong, Chinese bots were creating insane amount of users/ pages on my multisite wordpress install, over 15,000 users and pages were created during the weekend. It took me a while to clean it all out, but after several hours of cleaning and implementing several spam defenses, everything is back to normal.
But I would like to say KOODOS to SME server, with the insane amount of traffic and with both CPU's running at 100% the whole weekend, SME server did not crash.
Also KOODOS to all SME developers for focusing on security. Besides hammering my wordpress install, all my ports were pinged over and over by dozens of Chinese IP's and they were not able to get in.
It did help by me not allowing clear passwords because I use public key to access my server, and I also have DenyHosts contrib. But SME held out great, thanks to all developers.