ROLadder announces some of its features.

[HeyCarnut]

Agree with many of the criticisms of ELO for this as stated earlier. Is it set in stone? If not, perhaps a careful look at a TrueSkill like system is warranted.

Microsoft Research paper at:

[URL="http://dl.dropbox.com/u/6806907/MS_Trueskill_Ranking_NIPS2006_0688.pdf"][URL="http://dl.dropbox.com/u/6806907/MS_Trueskill_Ranking_NIPS2006_0688.pdf"][URL]http://dl.dropbox.com/u/6806907/MS_Trueskill_Ranking_NIPS2006_0688.pdf[/URL][/URL][/URL]

If you speak German, the paper that techniques like this follow (from Zermelo in the 1920's! Yes, that Zermelo...) :

[URL="http://dl.dropbox.com/u/6806907/Zermelo-MaxProbTournaments.xps"][URL="http://dl.dropbox.com/u/6806907/Zermelo-MaxProbTournaments.xps"][URL]http://dl.dropbox.com/u/6806907/Zermelo-MaxProbTournaments.xps[/URL][/URL][/URL]

Rob

Edit/PS: Also, a question: You state "...an in house developed system or a heavily modified version of ELO with many additions..." is to be used. Who will vet this mathematically, show its correctness, and verify the limits of how it itself can or cannot be 'gamed'? Thx.

 

[Snuffeldjuret]

If you want it to have a heavier emphasis on just the recent few matches, just change a few values and voila! There you got a system that is heavily dependent on the last few matches. It's tough to argue against a modified ELO system when both people argue for a faster converge to current skill.

To think that fullmjs results from 2004 has any affect on their todays score sounds redicolous to me, based on Zets saying that the system used today is already modified to make teams converge faster.

Let's not use the bargain tactics here, that does not work very well as if something is over the top it just sounds stupid instead of super cool exclusive .

Maybe you are discussing this becuase you don't want high ranked teams to be able to gain anythign at all when playing against lower ranked teams. If so, then that is not my impression and you should then try and make that point a bit clearer ^^.

Might as well suggest that every single player in a clan has their own rating, and the clan rating is the average one of the 5/8 most active players. Then if a very good player leaves a clan, it drops in rating right away, and if not the best 5 play, the rating loss for the team if it loses will only effect the clans rating if anyone from the 5 most active players where there. It may sound stupid at first, but I actually don't think it is that stupid. Games like DotA, HoN and I suppose LoL (I haven't played it though) are extremely familiar to team score based on 5 players individual score, and it seems to do rarther well there in my experience. It feels better than a strict clan rating anyway where all players are treated equal, it just needs a bit more "maintenence".

Only negative thing is that clans have to enter which players that played the match. A good thing though is that a player that changes clan can keep his personal rating, so that clan he joins does not get some easy wins to begin with to "steal" rating from other clans in an unfair manner. When he palys the team will be rated higher, but only when he becomes one of the more active ones the team itself will get a higher general rating. I hope you understand what I mean, because I don't write this very carefully .

 

[Zetsumei]

Edit/PS: Also, a question: You state "...an in house developed system or a heavily modified version of ELO with many additions..." is to be used. Who will vet this mathematically, show its correctness, and verify the limits of how it itself can or cannot be 'gamed'? Thx.

The great thing about point based systems is that you can make predictions based on a teams score what the match outcome will be. I have a lot of match data from Red Orchestra 1 which I will be using to tweak, test and compare systems. If the system I'm creating ends up less accurate than other existing systems then logically we wouldn't end up using that system.

This means you can run multiple rating systems, and based on real world performances you can see how accurate a rating system is and in what situation and under what circumstance what its variance in accurately predicting is. And if people would really demand it I could check whether improvements are significant (a < 0.05) using a repeated measures ANOVA.

Using a Kalman filter you could even use multiple rating systems together, letting it automatically select the best one for a certain situation (if there are different situations when different systems work better).

--------

I'm going look a bit in true skill perhaps for individual player rating within matches, for a bit of side fun (although you need to find an in game score that represents an individuals performance well, really hard to solidly proof it). As a good player might end up simply defending an underground passage to a cellar and never kill anyone due to that in a match while being a good player. So adding it into a point system does not necessarily make the point system more accurate or not (would have to see).

True skill is a very similar system to glicko2. And both are made for rating people at the end of a league or ladder (or individual players against each other after a round or 2 rounds), as they require multiple matches to calculate the variance, and average of the results in a so called rating period.

The current ROLadder system doesn't take in account that a scoring gets more inaccurate if a team doesn't play for a long time. Which is why the new system will take into account more information and for instance things such as map unbalance. You can say that currently in ROLadder the score is somewhat based on the last 5 matches, with a much heavier importance to the latest than the oldest.

Remember that the council (every clan got one vote) on ROLadder got a big say in what system people for instance like the most, even if it doesn't end up being the best. If for instance the players of roladder prefer a rung based system, then it will use a rung based system (I'd prefer a different system, but the choice is to the clans participating, and not to me).

But remember time is limited for me to create and adapt systems, and time is limited for the coders to implement features. But ideally I would want ROLadder to have every rating system someone can thinks of, with the ability to compare different systems, statistically. But that might be somewhat of a dream for me right now, definitely something nice for the future.

 

[213]

this isn't chess where the battle of wits is inherent. any sort of competition is artificial and contrary to the purpose of a video game, which is just for fun.

 

[Zetsumei]

this isn't chess where the battle of wits is inherent. any sort of competition is artificial and contrary to the purpose of a video game, which is just for fun.

Competition for a lot of people is fun. And even with competitions your goal doesn't have to be winning.

Competitive play allows you to play together with friends that you know and trust. And for those that want to, they can try and improve upon their own strength and possibly become part of the best of the best.

You can just run for fun, or you can run in a competition. In general competitions for games in the sense of electronic games , ball games, or board games are considered fun by a lot of people.

A lot of people will probably agree with my opinion, that the greatest game in terms of game play, is when everybody works together as a team, when facing a similarly skilled opponent. That is when a game will truly shine.

If I would describe clan play, I would call it having fun together with friends.

 

[KingLol]

Individual skill ratings is a step too far I reckon. It'll make the crap positions in matches even more hated while adding nothing except bragging rights for one or two players. It would fundamentally undermine the focus on teamplay, which is the whole point of having teams. If you were gonna have individual ratings you may as well just make MANLADDER: ROladder for 1v1s.

 

[Zetsumei]

Individual skill ratings is a step too far I reckon. It'll make the crap positions in matches even more hated while adding nothing except bragging rights for one or two players. It would fundamentally undermine the focus on teamplay, which is the whole point of having teams. If you were gonna have individual ratings you may as well just make MANLADDER: ROladder for 1v1s.

If there would be individual ratings built into a system, they wouldn't necessarily be made public. But as I said there its hard to find a score to determine how well someone absolutely performs, when people get send to for instance the basement in krasnyi to defend that spot.

But in for instance the fire fight (team death match) game type, it could offer something fun for individuals to participate in without joining a clan. There are loads of possibilities and methods to use systems correct or badly. I just think that keeping an eye open for possibilities is always something good.

 

[Snuffeldjuret]

I didn't mean individual skill that in any way is dependent on how many kills you get in-game or whatever. Only on end result. Which is the same for the whole team, hence it fits a team game. Remember that Dota/Hon are extremely focused on team-play, imho more than RO although I doubt you would believe me. To me Dota/HoN are the ultimate competitive team games partially as public gaming and clan gaming looks the same. Not only that, but the individual skill in Dota/HoN is much more about your understanding of teamplay, and not like in RO where it is actually hitting your target. Anyone who have spent 1000+ hours and on a high level in DotA/HoN will tell you same thing as I =).

Hence, individual rating is perfectly fine for a team game. As long as it is not based on anything but win/loss or team score. The rating you achive while playing 5on5 cannot be compared to 1on1 as to get a high rating in 5on5, you will need to have a great understanding of team-play.

This would improve things, I don't understand why you say it wont. It would brign much better consistency in which players a team plays with. Having accurate team ratings is worth nothing if the players a team uses differs a bit. I would guess that it would be a better system if the personal rating system would be used and have a worse system to actually calculate that rating. Just to mention one thing that would be neat with this system is when one team is formed by players that have already played with other clans, as they already have achived a rating that is close enough to give the new team a rating they should have.

EDIT:
As Zets said, it does not even have to be public. Although then it could be easier to abuse I guess. I don't see why it would be bad to have it public though. If you can find any bad reason that already isn't an issue, please let me know. Rememebr that it cannot be an already existing issue . Members of high skill clans can still brag about it, so it would not make it different if a player can brag about his own rating and not just the clans ^^.

 

[KingLol]

I just see incorporating individual player ratings in any capacity offering nothing. Whether publicly visible or not it just doesn't seem like a smart idea for 2 main reasons:

i.) It's extremely difficult to get a rating which is truly representative of player skill.

ii.) It takes away focus from the team performance. If you start attaching individuals to team performances it would lead to a stupid situation where a team could get a string of good results and then some players leave and "oh well all their good players left so that means they're less good so we'll move them down the ladder." which is laughably stupid!

I'm struggling to think of any real life sport that has individual ratings impacting team ratings, and the reason there aren't any is because it's a flawed idea on many levels.

edit:
Also team ratings ARE affected by which players you play with. If your team shows up with a kickass team and wins easily one day then loses to a bad team the next day because they're using different players then their drop in points will reflect their lack of strength in depth.

 

[Zetsumei]

text

Logically its hard if not impossible to realistically rate an individual player in a team effort (a clan leader that can't shoot **** can for instance through tactics be the difference of a win or a loss). And as you said playing with the wrong players is the flaw of the team.

But I do not dare to say whether it would actually increase or decrease the accuracy, since a clan generally got many members and you end up taking a weighted average over the active playing team, which could very well cancel out the inaccuracies of individuals.

Often individual players ratings are used in predictions that for instance predict a teams potential. Of course you're getting into vague statistics there but its not like it isn't used . And a score system that rates individual people could be used for a public server as well, or even for determining on who's the best with betting on the outcomes of matches.

Loads of crazy systems can be used for crazy things. I'm interested in generic research which is why I'm interested myself in seeing what things significantly affect the performance of a team. Researching things never hurts .

 

[Snuffeldjuret]

i.) Exactly the same issue for team skill. The error could be less if the team rating is based on the average of the players ratings. Thus an invalid argument.

ii.) You mean that a team that loses 2 of their top 5/8 players does not "get worse"? What if they lose their whole top 5/8, do they get worse then? Why not move them down the ladder a bit so they don't have to lose points in their X next matches. How can that be stupid? It is a brilliant concept, just tough to make functional. Invalid argument nr 2.

iii.) Comparing IRL and computer sports are not something to base statmenets on. So you don't need to struggle to find such a thing, it's pointless.

I'm sorry KingLol, but I wonder what kind of imagination you have of this concept. You should be a bit open minded, like not take for granted that I meant a rating system that is based on anything but team score for instance when I never said so. You try so hard to find flaws in this idea so that you find those that does not exist if the system itself is designed well.

 

[Zetsumei]

i.) Exactly the same issue for team skill. The error could be less if the team rating is based on the average of the players ratings. Thus an invalid argument.

Well remember that for team based ratings you use the information what the end result of a team was, aka did they win or not. For individual skill you need to find an individual attribute that accounts for team based skill.

Although you can create a statistic about how many times someone played in a winning match. For something like that to even find significant player differences the team would need to be varied enough from match to match to be able to conclude something about that on an individual basis (perhaps better suited for ranking gather or public matches in that sense).

I even wonder if there is a correlation between an individuals skill and a team outcome that overcomes the overall bias of skill of a team. If it does it could be a good method when teams are rather diverse like for instance if you bring in a ranking of gather matches, but when you always play with the same team it could likely happen that nearly every player would be ranked equally important. So whether it improves the system or not is tough to say, would need to try it out. A danger with that could be that punishments could somehow feel pretty random to people.

I think if including individuals in any way in a scoring system, then having them leaving should most likely affect the accuracy of someone's rating rather than drop the scoring by a certain amount (as in when you loose a match the other team does not necessarily go up by much more rather you as a team in that case go down more). As essentially its not the previous match results changed, but that you are less confident in how accurate a teams scoring is.

 

[Knighted]

There is nothing funner in a video game. Competition is the king.

 

[KingLol]

Rating the level of a team is much easier because there's only really 1 metric that counts: did your team win more rounds than the enemy? Individual skill is not so easy to rate since it's possible to be by far the best player in a game but have your team lose.

In response to your second point: yes I would absolutely expect a team to get worse if they lose 5 of their best 8 players. If they then go on to lose several matches against teams they would previously have beaten then their ranking will decrease accordingly and that reflects the weakening of the team. Automatically knocking them down X amount of places after the players leave based on the skill rating of those players is unfair and pure speculation.

3rd point: I was comparing it to real sports to highlight how strange it seems. I still think its a very strange concept.

I am open-minded to changes to the ladder, but only if they add value in some way and I'm really struggling to see what the benefit of this when faster converging team ratings would do the same job.

 

[Snuffeldjuret]

Well remember that for team based ratings you use the information what the end result of a team was, aka did they win or not. For individual skill you need to find an individual attribute that accounts for team based skill.

No.
No. How many times do I have to say no? The individual rating is ONLY based on what the end result for the teams was.

The individual rating is NOT there to decide which player is better than the other, or to get a perfect representation of a players skill in rating. It is there ONLY to make a better rating for the team.

If a team wins 10 matches in a row with the same players then ofcourse all players will have same rating even if the players themselves are not equally good, but is that a problem? No, it's not as the focus is on the team rating. And in such a situation the team would have same rating, no matter if there were individual rating or not.

Some might see it as unfair to knock them down, other might see it as a more mathematically accurate model of the speculation of the team skill, which a rating system is all about. So why complain about that?

We are talking about such trival stuff here that I am going insane :/. Try to look for opportunities, not for problems. That is how you find opportunities. Do not start looking for problems before you have found several opportunities.

The concept itself is not that strange, as gather systems in team based games uses exactly a system like this. Most famous might be DotA as it is played almost exclusively in "gather form". The idea we are discussing here is just adapting that concept into a much smaller player base of a single team. And there is where you look for problems later, once you accept the benefits of a system like this.

 

[Zetsumei]

Read the bits below the bit you quoted as well, as I explicitly talked about ranking based on team end result there . And I even mentioned gathers specifically as a situation where it could work.

 

[KingLol]

The point is that having individual players rated based on how well the team does is made completely redundant by the team rating....which is based on how well the team does. I honestly can't see how adding those individual ratings would benefit anything.

 

[Snuffeldjuret]

I know Zets, I did read it. I should say the same to you as you didn't read my words carefully enough . I bet mixing the phrases "individual skill" and "individual rating" isn't really helping. I never meant it as skill, but as rating. The Team's skill-rating is based on the individual players rating, not their skill.

The text after the one I quoted just repeated what I said. Except from the gather stuff. Including gather related things in the whole concept should amplify the flaws of an individual rating that is the base of the teams rating. I feel it would be wierd if the clans rating was not based on clan matches, by the clan members. It is already a little wierd with the idea of players changing clans and clan rating being affected by it. If the clan rating is not affected by a player quitting, then there is one less reason to use such a system.

KingLol, the question is: Shall a rating system show an approx skill of the team, or shall the rating system be a value of old results. I started talking about this as the discussion about Rung ladder touched some similar areas. I think it shall display an approx skill of the team, you think it shall display the past results. I think it should be able to converge as fast as possible, while you don't mind if it takes a few matches. I think that those few matches is a crack that grows bigger and bigger, you think not. We disagree of the very foundation of what the rating is there for, so it is completely useless for us two to discuss it further.

To point out one thing that would make this good. Imagine a clan that has 12 players. 8 of them are qually good, and the other 4 are all a bit less skilled than the other. Now, as those 8 play quite a lot, this team reaches the top of the ladder. One day it happens to be so that only 5 of these can play, so 3 of the less skilled players play the match. The end result was as expected 5-5, and the team lost a lot of points to a team further down in the ladder. The team that won didn't deserve that many points, as they didn't face an opponent that was well represented by the teams skill rating.

Small inaccurate things like this everywhere makes the whole ladder filled with inaccuraties, and that is something that Zets is trying to remove.

 

[KingLol]

In your example, the team that won DID deserve to win the points because they beat the higher ranked team and the higher ranked team suffers for not having the strength in depth required to win.

Using player skill ratings to change something like that would be SO easily exploitable.

With your same example: say its the last day of the "ladder season", the top team realises if they lose then they will drop into second place so what they do instead is have all their less good players play in the match. With a system like the one you seem to be describing this can then change the points that the top team stands to lose by enough so that they cant possibly drop into second place.

As for basing it on old results, then I see no problem with that since it's what all rankings in everything are based on. Your individual rankings based on team performance would also be subject to this. Rating a team based on how you think they'll be in their next match is simply reduced to an educated guess.

 

[Zetsumei]

text

When I said gathers and public I meant it for a separate say gather ladder, not using information from one ladder for another .

You need to remember that if the selected group you end up playing in a competitive match changes the outcome of a match. That a lot of people will aim of keeping a good new player secret till there is a big match. As then you get that the system will predict the new player is worse allowing the team to gain more points if they win.

Remember if a teams strength isn't consistent that for a rating system that takes into account variance that doesn't have to matter a lot as it will then take into account that a team isn't really consistent. You can make calculations taking into account variance .