After more testing of the Chessnut Evo here are my findings and opinions:

I still like the machine as a tool for learning and practising chess. As stated in my previous post the integration with Chess.com for example is very convenient. I like that the board positions are automatically recognised from the web site and I can try them out over a real board and my moves register in Chess.com. Of course, from such web sites I can play their chess bots over the real board and this works well.

I must admit to having not been familiar enough with the Maia project to know that it makes its moves instantly on purpose. There is no form of node searching such as Monte Carlo. I understand this is attempting to simulate humans playing by intuition. However, on testing Maia on the Evo I discovered that Maia 1600 (supposedly 1600 ELO) actually plays like a complete beginner blundering pieces all over the place! I then tried Maia 1900 against mobile shredder set to 1798 ELO(I was trying for 1800 but the slider control in Shredder does not give precise control) and the Maia was crushed blundering a piece to a pawn fork! I include the game below.

However, this is not a fault of the Evo’s implementation. I downloaded Lc0 on my PC and used the same weight files as the Evo. I had to set the Lc0’s(v0.24) UCI setting ‘Slowmover’ to a value of 0 disabling any node search and therefore it moves instantly as per the Evo. This played in exactly the same way, no surprises. I pitted Lc0 Maia 1900 against Evo Maia 1900 for two games and the result was a game each, but certainly not at the level of play of a human rated 1900 ELO! I have e-mailed Chessnut about this and must say they are excellent at responding and have stated they are going to improve the Maia weight files with their own training.

Here is the game between Evo Maia 1900 v Shredder Mobile set to 1798 ELO:


White’s move 12.Be3?? is shocking. Losing either a Night or Bishop! I seriously doubt a player rated 1900 would make this mistake.

So at the moment playing against Maia may be fun but is probably not going to be challenging enough for many. I think that in many positions we need to deduct several hundred ELO to get a more accurate level of play. Don’t bother pitting it against another dedicated machine.

I am also disappointed with the SF16 implementation. All that power available for analysis but restricted to moving instantly. It really should be possible for the user to set thinking times for the engine. I have queried this with Chessnut and they intend to increase the thinking time for better analysis and have said they will consider my suggestions for adding playing levels and infinite analysis. We will have to see. That being said, SF16 moving instantly is going to give many of us a hard time! Also, as mentioned in a previous post installing DroidFish provides us with everything needed.

The Maia project is interesting but instant moves without any MCTS currently lets it down. The Evo has the Lc0 engine of course, but as this also moves instantly must also have zero search. This means that Evo’s feature of creating your own personalities by training with 200 games is limited. At best we may get for example a Magnus Carlson personality but moving instantly throughout a game and therefore I seriously doubt playing at his strength. Maybe if Magnus had zero time per move for every move in a game it may be a more accurate simulation but this is hardly a realistic form of play! I’ve not confirmed this yet by training because at the moment it’s too cumbersome to put 200 games into the Evo.

Games played on the Evo using the built in bots are not stored locally but in the cloud somewhere. This is not great and I have also queried this with Chessnut because if and when their servers are shutdown or people may not have internet for some reason, they won’t have access to their games! Chessnut are developing a web site where user’s can log in and gain access to their games. At the moment only a single game per PGN is possible although this is going to be improved. This means to transfer a PGN with 200 games for training purposes won’t work, I would have to transfer one game at a time! I will wait until the website can handle PGN files with multiple games before testing the training feature. It may be easier via one of the chess web sites, but I’m wanting to pick games from my own databases.

Am I disappointed in my purchase? No, I still like the concept and don’t treat the Evo as a normal dedicated machine. I think it has much potential but still offers a great way of learning the game. For beginners in chess the Maia engine will be fun, but for more serious play then there is SF built in (virtually no thinking time) and of course playing online through chess.com for example, and of course installing Android chess apps such as DroidFish with the option of playing against Komodo 9 or 10 or some other free engines from the App store (via APKPure).

Ray

I'm conducting further tests to try to establish the actual ELO rating of Maia 1900 on the Evo. For this I'm using the Shredder for Android app which calculates the ELO of the opponent over several games and based on the result of each game adjusts its own ELO for the next game. In this way it targets the ELO of the opponent plus or minus a margin of error, but I believe gives a very good idea of the player's rating. So far the Maia 1900 bot has an ELO of 1465 but I think this will drop more as I get it to play more games. I'm not bothering to save these games, just playing as many as I have time for.

I have noticed that sometimes Maia appears to play reasonable moves but often it tends to be just a matter of time before it starts blundering pieces. I would think this is because it just does not have enough information for all positions in a particular game. Maybe if the neural networks were trained with a number of games a magnitude higher it may fair better, but I wish they would have implemented some node searching. Just looking 4 or 5 ply ahead would make quite a difference I think.

Regards
Ray

Ich kann die Aussagen von Ray alle so unterstreichen. Bin mal gespannt, ob die nächsten Wochen/Monate nun an der Software gebastelt wird, nachdem die Produktion nun so langsam zur Routine werden sollte. Baustellen bzw. Verbesserungsbedarf gibt es ja durchaus in recht großer Menge, wenn ich mal so auf die anderen Threads verweisen darf.

Viele Grüße
Bernd