Dedicated server problems with wireless?

First off.. ports are forwarded correctly in my router, windows has proper firewall permissions set, I have a dedicated 1.1 server running with the affinity set to my 3rd and 4th cores (3.6ghz) and I am joined with the client affinity set to the 1st and 2nd cores

While hosting a dedicated server on my machine I am having clients randomly dropping while playing. When the disconnect happens for them, it will say "loading tile data #%" and then eventually "lost connection".

My PC is connected via wireless with a constant 75-100% signal strength, 100% line quality, and 16% noise level. Bandwidth tests show ~20Mb down/2Mb up (the speed of my cable internet)

Would this still not be adequate for hosting a small (<8 player) server? Do I need to run a 40ft ethernet cable to my machine to host a dedicated server? :-/