Kramnick's Current Study Of Cheating In On-Line Chess

It's a massive problem people lose and say He/She cheated.

@kalafiorczyk said in #4:
> Kramnik's analysis analyzed by a professional data scientist:
>
> dorianquelle.github.io/blog/Cheating-In-Titled-Tuesday/
>
> As the old saying goes: lies, damned lies, and statistics.
I don't understand this analysis... How does it shows what the "professional" says it shows?
How can you show there is no prevalence of anything when your analysis define the prevalence as the normal?

đi hậu xuống ăn tốt

@pokraise said in #23:
> I don't understand this analysis... How does it shows what the "professional" says it shows?
> How can you show there is no prevalence of anything when your analysis define the prevalence as the normal?

Here are two papers explaining the statistical approach used by Kramnik and his mathematician friends:

web.stanford.edu/class/hrp259/2007/regression/storke.pdf
www.researchgate.net/publication/227763292_Storks_Deliver_Babies_p_0008

@kalafiorczyk said in #4:
> Kramnik's analysis analyzed by a professional data scientist:
>
> dorianquelle.github.io/blog/Cheating-In-Titled-Tuesday/
>
> As the old saying goes: lies, damned lies, and statistics.

I have looked through report, although there is a number of interesting points there are aspects that make the analysis weak.

(1) Report does not state the source of the 'ratings'. If chess.com's blitz rating is being used this would make the report very dubious as clearly there would be a circularity.

(2) Defining the 'best move' based only on 1 sec of engine analysis is weak, imo. As a ~1500 player using a fairly powerful modern PC I would analyse my own games at a minimum of 10 seconds per move - and these games would be of the throwaway type, rated OTB games I use 60seconds with even long times of 1+ hour at key conjunctions. The report makes no attempt or mention that to indicate whether 1 second is sufficient. Does the analysis change if 2 seconds etc are used? who knows. At the very least some sort of qualification check should have been performed, possibly using a larger time interval with a correspondently smaller dataset.

(3) The report's key metric is 'average diversion of a over game' and ignores just how powerful selective usage of engine can be on the result of a game. As many elite chess players has pointed out, just knowing an engine evaluation of the game would provide a significant advantage, far less than recommended move in a given position.

The principle basis of Kramnik's analysis is a comparison between on-line blitz play and OTB ratings, with the reasonable assumption that the level of cheating on the latter is likely to be extremely small compared to on-line. This is a very solid way of detecting cheating, as the main purpose of cheating in this context is money based on results. Caruana in a recent c2squared podcast has stated he would not expect there to be any difference in the quality of play between elite player whether playing OTB or Titled Tuesday. - i.e. there is no such thing as a 'blitz specialist' at these level - he put the difference below 25 rating point at worst. If this was true then clearly even 1SD difference over sufficient game would be suspicious. From my understanding of the mental differences between OTB/Blitz I would actually expect to see a bias towards the higher OTB rated player compared to strict ELO probabilities - in short events like Titled Tuesday should have an inherent bias towards higher rated OTB players.

@kalafiorczyk said in #25:
> Here are two papers explaining the statistical approach used by Kramnik and his mathematician friends:
>
> web.stanford.edu/class/hrp259/2007/regression/storke.pdf
> www.researchgate.net/publication/227763292_Storks_Deliver_Babies_p_0008
Does Kramnik claim to use this approach or is it your (second hand I guess) interpretation of his approach?

@pokraise said in #27:
> Does Kramnik claim to use this approach or is it your (second hand I guess) interpretation of his approach?
NO!
Those papers deliberately show the HUGE error of presenting a mere correlation as showing a cause and effect relationship. BEWARE! They try to make their point using humour!

The relationships between stork numbers etc. and human birth rates are all coincidence and NOT a real cause and effect relationship.
Lots of things correlate to shoe size but you need to take great care before formulating any cause and effect relationships. Shoe size is of course related to age.

I would say that Kramnik and his academic stats expert would definitely try very hard to avoid the correlation trap. The Kramnik approach seems adhoc, subjective and for sure other experts would focus strongly on the assumptions inherent in the proposed method. Why not use something based on the existing scientific literature?

I am not qualified to assess the Kramnik approach. But I do wonder about some aspects of the approach.

So, just to bring things back into perspective (for but a moment, alas) ...

There's a lot at stake here ... of course there will be strong arguments on each side ...

Now, consider who takes which side and make up your own mind ...

And I'll pipe up again here with this comment ...

Russian mathematicians have a formidable resume of achievements ... maybe they will identify the critical 'parameters' Kramnick mentions several times, although never going into any details besides cpl and expected rating performance ...