Stockfish testing by Stefan Pohl

Latest Website-News (2014/10/18): The result of Stockfish 141012 is online (a nice step forward (+9 Elo to Stockfish 140928 and +30 Elo to Stockfish 5)). And the new opening-database SALC is online and is in use for all of my testwork from now. Download it [You must be registered and logged in to see this link.] or in the Downloads & Links-section and check the Readme-file for further information. A big thanx to Hauke Lutz for his work on the SALC-sets!!!
 
The Endless RoundRobin-tournament was restarted with the new SALC-openings last week and from now with Komodo 8 and Equinox 3.2 as participants.

---

Stockfish testing
 
Playing conditions:
 
Hardware: i7-2630QM 2.0GHz Notebook, Windows 7 Home Premium 64bit, 4GB RAM
Fritzmark: singlecore: 3.97 / 1905 (all engines running on one core, only), average meganodes/s displayed by LittleBlitzerGUI: Houdini: 2.0 mn/s, Stockfish: 1.7 mn/s
Hash: 128MB per engine
GUI: LittleBlitzerGUI (draw at 120moves, resign at 450cp (for 4 moves))
Tablebases: None
Openings: 10moves_SALC_500.epd (download the file at the "Download & Links"-section)
Ponder, Large Memory Pages & learning: Off
Thinking time: 70''+700ms per game/engine (average game-duration: 3.5 minutes)(standardized to the hardware-speed and the thinking time of the excellent [You must be registered and logged in to see this link.]). One 5000 games-testrun takes 100 hours (running on only 3 of 4 cores)
 
Each Stockfish-version plays 1000 games against Houdini 4, Komodo 8, Gull 3, Fire 3, Rybka 4.1
 
Latest update: 2014/10/18 (Stockfish 141012)
Current testrun: none
 
Download the individual statistics [You must be registered and logged in to see this link.]
     Program                   Elo    +    -   Games   Score   Av.Op.  Draws
   1 Stockfish 141012 x64    : 3203    6    6  5000    62.7 %   3109   37.7 %
   2 Stockfish 140928 x64    : 3194    6    6  5000    61.5 %   3109   39.3 %
Below you find the old ORDO-calculation from Stockfish 5 to Stockfish 140928 (old opening-set and with Komodo 7a instead of Komodo 8). Take a look at the draw-rate in both lists and how much the new SALC-set lowered it !!!
     Program                      Elo    +    -   Games   Score   Av.Op.  Draws
   1 Stockfish 140928 x64       : 3194    7    7  5000    64.0 %   3091   47.9 %
   2 Stockfish 140809 x64       : 3191    7    7  5000    63.6 %   3091   47.2 %
   3 Stockfish 140723 x64       : 3190    7    7  5000    63.4 %   3091   46.1 %
   4 Stockfish 140611 x64s      : 3190    7    7  5000    63.4 %   3091   47.4 %
   5 Stockfish 140727 x64       : 3189    7    7  5000    63.3 %   3091   45.2 %
   6 Stockfish 140703 x64       : 3188    7    7  5000    63.1 %   3091   46.3 %
   7 Stockfish 140628 x64       : 3188    7    7  5000    63.1 %   3091   46.3 %
   8 Stockfish 140714 x64       : 3184    7    7  5000    62.6 %   3091   47.4 %
   9 Stockfish 140606 x64s      : 3183    7    7  5000    62.5 %   3091   47.9 %
  10 Stockfish 140623 x64s      : 3182    7    7  5000    62.4 %   3091   47.9 %
  11 Stockfish 5 140531 x64s    : 3173    7    7  5000    61.2 %   3091   48.2 %
Below you find a diagram of the progress of Stockfish in my tests. Red dots: Elo of Stockfish in the (no longer existing) LS-ratinglist based on at least 10000 games (with 45''+500ms thinking time and 64 MB Hash). Blue dots: Elo of Stockfish in my current Stockfish-testruns (adjusted to the LS-Elo-ratings (with the Stockfish DD-rating)), based on 5000 games (with 70''+700ms thinking time and 128 MB Hash).
Use the button in the lower right corner of the diagram to zoom it (can cause a distortion...)
(LS-Elo of Houdini 4 was 3184 Elo.)

Latest Website-News (2014/11/14): Testrun of Stockfish 141109 finished. +10 Elo to Stockfish 141102 - nice step forward. Now +42 Elo to Stockfish 5.
Endless RoundRobin-tournament updated. From now, Stockfish 141112 will replace Stockfish 140928 in the Endless RoundRobin.
Testrun of Elektro 1.1c finished ("Experiments"-section).

---

Stockfish testing
 
Playing conditions:
 
Hardware: i7-2630QM 2.0GHz Notebook, Windows 7 Home Premium 64bit, 4GB RAM
Fritzmark: singlecore: 3.97 / 1905 (all engines running on one core, only), average meganodes/s displayed by LittleBlitzerGUI: Houdini: 2.0 mn/s, Stockfish: 1.7 mn/s
Hash: 128MB per engine
GUI: LittleBlitzerGUI (draw at 120moves, resign at 450cp (for 4 moves))
Tablebases: None
Openings: 10moves_SALC_500.epd (download the file at the "Download & Links"-section)
Ponder, Large Memory Pages & learning: Off
Thinking time: 70''+700ms per game/engine (average game-duration: 3.5 minutes)(standardized to the hardware-speed and the thinking time of the excellent [You must be registered and logged in to see this link.]). One 5000 games-testrun takes 96 hours (=4 days) (running on only 3 of 4 cores). The version-numbers of the Stockfish-development engines are the release-date, written backwards (year,month,day))(example: 141028 = October, 28, 2014), downloaded at [You must be registered and logged in to see this link.]. I always use the latest version of one day, if more than one version per day is released. And I use the version "for modern computers".
 
Each Stockfish-version plays 1000 games against Houdini 4, Komodo 8, Gull 3, Fire 3, Rybka 4.1.
 
Latest update: 2014/11/14 (Stockfish 141109)
Current testrun: none
 
Download the individual statistics [You must be registered and logged in to see this link.]
 
     Program                   Elo    +    -   Games   Score   Av.Op.  Draws
   1 Stockfish 141109 x64    : 3215    7    7  5000    64.3 %   3110   37.4 %
   2 Stockfish 141102 x64    : 3205    7    7  5000    63.0 %   3110   39.4 %
   3 Stockfish 141012 x64    : 3203    6    6  5000    62.7 %   3110   37.7 %
   4 Stockfish 141024 x64    : 3202    7    7  5000    62.6 %   3110   38.1 %
   5 Stockfish 140928 x64    : 3194    6    6  5000    61.5 %   3110   39.3 %
 
Below you find the old ORDO-calculation from Stockfish 5 to Stockfish 140928 (old opening-set and with Komodo 7a instead of Komodo 8). Take a look at the draw-rate in both lists and how much the new SALC-set lowered it !!!
     Program                      Elo    +    -   Games   Score   Av.Op.  Draws
   1 Stockfish 140928 x64       : 3194    7    7  5000    64.0 %   3091   47.9 %
   2 Stockfish 140809 x64       : 3191    7    7  5000    63.6 %   3091   47.2 %
   3 Stockfish 140723 x64       : 3190    7    7  5000    63.4 %   3091   46.1 %
   4 Stockfish 140611 x64s      : 3190    7    7  5000    63.4 %   3091   47.4 %
   5 Stockfish 140727 x64       : 3189    7    7  5000    63.3 %   3091   45.2 %
   6 Stockfish 140703 x64       : 3188    7    7  5000    63.1 %   3091   46.3 %
   7 Stockfish 140628 x64       : 3188    7    7  5000    63.1 %   3091   46.3 %
   8 Stockfish 140714 x64       : 3184    7    7  5000    62.6 %   3091   47.4 %
   9 Stockfish 140606 x64s      : 3183    7    7  5000    62.5 %   3091   47.9 %
  10 Stockfish 140623 x64s      : 3182    7    7  5000    62.4 %   3091   47.9 %
  11 Stockfish 5 140531 x64s    : 3173    7    7  5000    61.2 %   3091   48.2 %
Below you find a diagram of the progress of Stockfish in my tests. Red dots: Elo of Stockfish in the (no longer existing) LS-ratinglist based on at least 10000 games (with 45''+500ms thinking time and 64 MB Hash). Blue dots: Elo of Stockfish in my current Stockfish-testruns (adjusted to the LS-Elo-ratings (with the Stockfish DD-rating)), based on 5000 games (with 70''+700ms thinking time and 128 MB Hash).
 
Use the button in the lower right corner of the diagram to zoom it (can cause a distortion...). If the zoom is activated, you can save the diagram (as a JPG-picture (in originial size)) on your PC with a right mouseclick and then choose "save image"...
(LS-Elo of Houdini 4 was 3184 Elo.)
Big Elo improvement. Now 42 elo more than Stockfish 5 

Somebody did a huge amount of work!

VitruviusH wrote:

Somebody did a huge amount of work!

yes though we prefer to go by Toms tests

Latest Website-News (2014/12/04): Because the score of Stockfish 141112 is still incredible in my Endless RoundRobin tournament, I decided to do a testrun of Stockfish 141112 for my Stockfish-testing, although I already tested Stockfish 141117, which is newer. But as you can see, Stockfish 141117 is a bad regression and Stockfish 141112 is much stronger (+10 Elo). So we have a new best version: Stockfish 141112, which is +53 Elo stronger than Stockfish 5 and +10 Elo stronger than Stockfish 141117. Next test: Stockfish 141130 (result not before Tuesday).

Endless RoundRobin-tournament updated.

Testrun of Firenzina 2.4.3 for my just-for-fun Ippolit-derivative-testing finished ("Experiments"-section).

Stockfish testing



Playing conditions:



Hardware: i7-2630QM 2.0GHz Notebook, Windows 7 Home Premium 64bit, 4GB RAM

Fritzmark: singlecore: 3.97 / 1905 (all engines running on one core, only), average meganodes/s displayed by LittleBlitzerGUI: Houdini: 2.0 mn/s, Stockfish: 1.7 mn/s

Hash: 128MB per engine

GUI: LittleBlitzerGUI (draw at 120moves, resign at 450cp (for 4 moves))

Tablebases: None

Openings: 10moves_SALC_500.epd (download the file at the "Download & Links"-section)

Ponder, Large Memory Pages & learning: Off

Thinking time: 70''+700ms per game/engine (average game-duration: 3.5 minutes)(standardized to the hardware-speed and the thinking time of the excellent FGRL Bullet-ratinglist). One 5000 games-testrun takes 96 hours (=4 days) (running on only 3 of 4 cores). The version-numbers of the Stockfish-development engines are the release-date, written backwards (year,month,day))(example: 141028 = October, 28, 2014), downloaded at [You must be registered and logged in to see this link.] I always use the latest version of one day, if more than one version per day is released. And I use the version "for modern computers".



Each Stockfish-version plays 1000 games against Houdini 4, Komodo 8, Gull 3, Fire 3, Rybka 4.1.



Latest update: 2014/12/04 (Stockfish 141112)

Current testrun: Stockfish 141130



Download the individual statistics here



Program Elo + - Games Score Av.Op. Draws

1 Stockfish 141112 x64 : 3226 7 7 5000 65.7 % 3110 37.6 %
2 Stockfish 141117 x64 : 3216 7 7 5000 64.3 % 3110 38.3 %
3 Stockfish 141109 x64 : 3215 7 7 5000 64.3 % 3110 37.4 %
4 Stockfish 141102 x64 : 3205 7 7 5000 63.0 % 3110 39.4 %
5 Stockfish 141012 x64 : 3203 7 7 5000 62.7 % 3110 37.7 %
6 Stockfish 141024 x64 : 3202 7 7 5000 62.6 % 3110 38.1 %
7 Stockfish 140928 x64 : 3194 7 7 5000 61.5 % 3110 39.3 %



Below you find the old ORDO-calculation from Stockfish 5 to Stockfish 140928 (old opening-set and with Komodo 7a instead of Komodo 8). Take a look at the draw-rate in both lists and how much the new SALC-set lowered it !!!

Program Elo + - Games Score Av.Op. Draws

1 Stockfish 140928 x64 : 3194 7 7 5000 64.0 % 3091 47.9 %
2 Stockfish 140809 x64 : 3191 7 7 5000 63.6 % 3091 47.2 %
3 Stockfish 140723 x64 : 3190 7 7 5000 63.4 % 3091 46.1 %
4 Stockfish 140611 x64s : 3190 7 7 5000 63.4 % 3091 47.4 %
5 Stockfish 140727 x64 : 3189 7 7 5000 63.3 % 3091 45.2 %
6 Stockfish 140703 x64 : 3188 7 7 5000 63.1 % 3091 46.3 %
7 Stockfish 140628 x64 : 3188 7 7 5000 63.1 % 3091 46.3 %
8 Stockfish 140714 x64 : 3184 7 7 5000 62.6 % 3091 47.4 %
9 Stockfish 140606 x64s : 3183 7 7 5000 62.5 % 3091 47.9 %
10 Stockfish 140623 x64s : 3182 7 7 5000 62.4 % 3091 47.9 %
11 Stockfish 5 140531 x64s : 3173 7 7 5000 61.2 % 3091 48.2 %

Nice work!