• Please make sure you are familiar with the forum rules. You can find them here: https://forums.tripwireinteractive.com/index.php?threads/forum-rules.2334636/

Server Suddenly high latency in all servers

Jump to latest Follow Reply
Sort by date Sort by votes
M

mallic

Active member
Nov 17, 2009
43
3
#1
Hey all,

I have heard this report from several members of multiple communities, but some time last week every person that connects to any of my servers and several other servers on the west coast has a base ping of around 100ms. I even have a base ping of 100 in my own server which is hosted in the same city as me once the population of a given server goes over about 15 people. This wasn't a problem during beta or any time up to a week ago. As far as I am aware of, there are no network related issues that would be causing this to happen as I am only pulling a data throughput of 12mbps peak down my box and my bandwidth limit is 20mbps burstable to 100. So there is not a bottleneck on my carriers. I have noticed in my servers that this happens regardless of how low I specify the ping kick measures or whether the game is set to client side or server side detection.

Is there something going on with tripwire/steam auth servers causing a massive amount interference lag? I have noticed that RS:2 servers added to the steam favorite list seem to not poll accurately every time. I have hit refresh on the steam favorite browser and have had it display information that is out of date by like two hours in regards to what players are on the server. Not sure what logs I could provide to help troubleshoot as I don't see anything about lag or the server having to catch up in the logs.
 
#2
Generally a high latency for your server is less about available bandwidth than it is about available CPU. If the server is suddenly struggling for some reason, then that will result in higher pings as the server takes longer to get around to responding to each request.

Nothing has changed that I am aware of that would cause a sudden CPU bottleneck, but the usual culprit for this sort of thing is bloated server logs. Now, in theory the bulk of the logging outputs should be suppressed on servers to stop this from happening, but perhaps your inis somehow escaped the setting overwrite. If you check the logs folder on the server, do you have a massive Launch.log file? If so, there are three things to do.

1) Check the server's ROEngine.ini to ensure that somewhere under [Core.System] you have these:

Suppress=ScriptLog
Suppress=ScriptWarning
Suppress=Warning

2) Make a copy of the log, zip it up, whatever you need to do, and put it somewhere we can get access to it for analysis
3) Restart the server

That will close off the log and start a fresh one, which will reset any performance issues if that is what's actually happening. If you _don't_ have a massive log file, then I'm less certain about the cause. You'd need to check the CPU performance of the server and make sure it's not getting overwhelmed. If it is, you can try reducing the server tick rate a little to ease the load.
 
Upvote 0
#3
The thing is though, it's not just my servers having issues. It's multiple servers across the board. Something is going ****y. Also I am hosting 4 64 slot servers and one 16 slot on a box with dual x5570s at 2.93ghz, I doubt it is a problem of not having the CPU power since the stable running cpu usage is about 30% at full load of all servers. RAM is at 11gb/32gb. I will try nuking the logs next restart cycle and see if that helps.
 
Upvote 0
#4
30% CPU (all cores combined) load sounds about right, but please also check single cores.

Generally speaking one full server (with active players, no idlers) on 4 core machine will mean that one of the cores is at 90-100% and 3 remaining cores at <20%, giving CPU usage of less than 50% but the server may still be struggling due to main thread core performance.
 
Upvote 0
#5
30% across 8 cores. I have a dual processor server. And no individual core is being held up in resource monitor. Also, yet again this problem started a week and a half ago with ZERO hardware or software changes. Literally just started happening across all servers in the middle of a match one day about a week and a half ago. Trying to troubleshoot this I have restarted all the servers, the box, ran speed tests, etc. I even tried shutting down all game servers but a single RO2 server and the ping for players is higher than it should be.

And I doubt it's my network as I host a teamspeak on the same box and the same person on that box has a 30-50ms ping higher in RO2 than TS. I understand that the net protocol is probably more intensive and actual data usage is higher in a game than a voice server, but this is an insane gap in performance. Unless there is a higher level issue on some carrier somewhere, I can not explain why all of a sudden across all servers I run for RO2 and NOTHING else regardless of hit detection mode (the only setting that I use that could theoretically affect cpu/network load) pings jumped a good 50 MS higher and have stayed that way for a week and a half.

Hell rising storm uses damned near no resources compared to things I have hosted in the past. I can host an arma3 server with 30 people in it on this box with less latency issues than currently. The CPU and RAM requirements for that dwarf RS2 by miles.
 
Last edited:
Upvote 0
#6
I don't think you have enough CPU power to handle even a single 64 slots server. I would say 45 slots max.

Recommended CPU for 64 slots server is Intel Xeon E3-1270 3.4 GHZ. Comparing Single thread performance here, see the result. I have a 1230v2 (which you can see in the comparison here with both previous CPU) and I had to eventually reduce slot count to 60 because when the four RS 2 servers I host were full at 64 slots, there was huge lags on all servers despite the fact that it should theoretically handles them (mine has better single thread performance but lower turbo boost ;)).

You may have dual CPU which in total is lot of CPU power, but the server is mostly single thread and need big single thread power. Big Giga Hertz.
 
Last edited:
Upvote 0
#7
I would buy that, but once again, server was handling about 210 people connected to it across all of the RS2 servers I host during the ENTIRETY of beta with 0 of these issues. This problem literally started out of nowhere after hosting servers throughout first beta wave the released server files all the way up to a week and a half ago with no issues. Also, if it were the case of not having the power, I would see at least a single core pegged to max during that gameplay. And I am not.

Also to address the Anti-matter dev, I tweaked logging settings and am now waiting for a full day of logs before I send them up. Will get back to you.
 
Last edited:
Upvote 0
#8
Maybe the numerous people trying to connect to your full servers are increasing CPU usage or making trouble to the servers (during BETA there was lot less people so maybe that's it).Try to add a password when server is full and having trouble to avoid people trying to connect, and see if after a few minutes it stops having problems. Then remove password and see if problems occurs again.
 
Upvote 0
#9
Good idea. Will try. Also have discovered that my server is having minor packet loss issues at carrier from my location after finally getting my modem replaced with one that doesn't seem to be dropping packets like it's hot. May have to have a big fight with provider >_<
 
Last edited:
Upvote 0
#11
Intel 82576 dual gigabit NIC. I have all management and non game server traffic on one port and all traffic for the game servers on the second. Peak data usage with all servers full is 13 Mbit/s. All settings are set to default and drivers have not changed in the entire time I have been hosting these. Only changes was locking down unneeded firewall ports in windows firewall. Also full server specs which I should have given in the first place:

Chassis: HP Proliant DL180 G6
Processor: Dual x5570s
RAM: 32 GB (can't remember speed off the top of my head atm but it's all circa 2009, so not the fastest)
NIC: Intel 82576 Dual Gigabit
Raid Card: HP SmartArray p410 with cache (can't remember cache amount atm)
OS: Windows Server 2012

HDD setup:
OS on a Raid 1 hardware partition with two 300 gb 15k SAS 6Gbps drives.
RS2 servers on a raid 5 partition with 4 300gb 15k SAS 6 Gbps drives.

I may just say **** it and pick up a poweredge r230 with a e3-1270v6 in it since I am too lazy to source a build to slap together an effective 4790k 1U/2U build and I am repeatedly hitting the single thread issue across multiple games. Shame source isn't still a popular engine lol. Could host tons of source based games without ever tripping on issues.
 
Last edited:
Upvote 0
#14
Rivimies;n2296760 said:
Very good NIC. Yeah, packets are lost somewhere else then.
Click to expand...

Yep packet loss was on Level3's (the tier 1 for the server) end. They apparently had routes all across the american southwest completely ****ed up. Good lord they are the worst tier 1 I have dealt with. I have to deal with them at work and they are constantly doing route and signal changes that mess things up.
 
Upvote 0
#15
At least you didn't buy Transit from Cogent (AS174). Their motto is "20% Packet loss is normal for us" ;)

However, it was $1Mbit/95th percentile back in the day; I suppose you got what you paid for, to speak. Luckily we can pickup HE and Tescali (Ti-Net) now for the same price in LA & SanJose. DeutcheTelekom/T-systems ain't half bad now either for pricing.

Anyone still buy transit from Savvis?
 
Upvote 0
You must log in or register to reply here.
Top Bottom
Back
Jump to latest Follow Reply
Jump to latest Follow Reply
Community News
Forums
Merch
Menu