Bots analysis

blipadouzi · Post by **blipadouzi** » Thu Mar 02, 2006 10:07 am

I was curious as to which bots were the strongest and which were the weakest...so I performed a lengthly and scrutinous test or all the bots and here are my results.

The figures are 95% accurate...the 5% margin of error is to account for luck and fate which is dependant on the role of the dice. The following list is listed with the strongest bots on top working down in order to the weakest.

Reaper
BotofDoom
Killbot
Evil Pixie
Bort
Brainiac
Angry
Pixie
Cluster
Stinky
Yakool
Quo
Boscoe
Shaft
Trotsky
Nefarious

Notes:

-Chimera was excluded as it is nothing more than a random generator.

-Communist was excluded as it is my understanding that Trotsky is a glorified version of Communist.

-Nefarious is by far the worst bot of all...one of my tests was to pin each bot in a one-on-one elimination competition. Each round the bots had to play 7 consecutive games...most wins of the 7 was the winner.
Nefarious played 5 rounds, and of the 35 games played, it did not win one single game.

BlackPanther · Post by **BlackPanther** » Thu Mar 02, 2006 10:12 am

You must be SO bored to have done that!

SnyperEye · Post by **SnyperEye** » Thu Mar 02, 2006 10:37 am

Looks like Preachermans baby has gone soft

jOnNiE · Post by **jOnNiE** » Thu Mar 02, 2006 11:57 am

nef is the worst? I would have never guessed that

blipadouzi · Post by **blipadouzi** » Thu Mar 02, 2006 12:33 pm

BlackPanther wrote:You must be SO bored to have done that!

Well, I am at work after all...

But in all truth, I enjoy doing those kind of things.

AquaRegia · Post by **AquaRegia** » Thu Mar 02, 2006 1:02 pm

Nef is worst at WHAT, exactly?

Different bots were written to excel on different card settings, continent bonuses, and maps; I doubt ANY were intended to begin a game with only two players.

Can you give a bit more detail about your "experimental design"?

SunTzu · Post by **SunTzu** » Thu Mar 02, 2006 1:11 pm

What were all of the parameters? (map, card setting, etc.)

When I've tested my maps Nef usually comes in 3rd, after BoD & Reaper.

michelle · Post by **michelle** » Thu Mar 02, 2006 1:29 pm

blipadouzi wrote: -Nefarious is by far the worst bot of all...one of my tests was to pin each bot in a one-on-one elimination competition.

all of the bot were made with love by either dustin on a member of this community this bot here was made over a long period of time by a member who put a lot of work into her, and I think its rude to say what you did with out putting the effort in a making one of your own

GregM · Post by **GregM** » Thu Mar 02, 2006 3:23 pm

Interesting stuff blipadouzi, but how did you calculate it? And how did Angry get a spot in the top half of your rankings?

blipadouzi · Post by **blipadouzi** » Thu Mar 02, 2006 3:34 pm

I did misword the part about Nefarious...he is "by far the worst", but specifically in this scenario.

I didn't mean to imply overall. And I am sorry for not clarifying that.

I do plan on running another tournament with different parameters.

To answer many of your requests, here are the parameters I used for this tournament...

1) I took an existing map (Armageddon) and modified it slightly to make it more evenly distributed to a 2 player game...as it is originally best suited for 6 players. I also made the map into one continent.

2) I then create a predefined starting scenario giving both sides exactly half of the map. With 0 armies assigned to any locations.

3) Then I proceeded to do eliminations...start with 16 teams, pair them off...the losers get eliminated the 8 remaining get paired off and play and so on until a clear winner is defined.

4) Of the losers, another elimination tournament was done to determine the 3rd, 4th, 5th, 6th and so on places.

So as Nefarious is concerned, I think the map was a little too restrictive for his abilities.

If anything, this helped to identify his weakness...

Who knows, if I would have divided the map into 6 or 7 continents, he may have faired better.

Here are a few other contests I want to run...

1) Same map with bots selecting their starting positions

2) Either the same map or another one with several continents to allow certain bots to take advantage of continental battles.

3) 3 or more player battle...(another theory I have for some of the bots not fairing so well is the fact that they were only playing against one opponent)

blipadouzi · Post by **blipadouzi** » Thu Mar 02, 2006 3:35 pm

GregM wrote:Interesting stuff blipadouzi, but how did you calculate it? And how did Angry get a spot in the top half of your rankings?

I have to be honest...Angry totally surprised me...but it seemed that battle after battle he would win.

Then once he fought against Killbot and Reaper...that was the end of his winning streak.

But like I said above...it may have just been the scenario.

In one of my future tournaments he may do worse...or who knows...better.

We also have to understand that bots, unlike humans, have set parameters that almost never change. A situation presents itself, and they react. Where as a human can randomly change their thought patterns.

GregM · Post by **GregM** » Thu Mar 02, 2006 3:53 pm

Allow me to say that that seems a very strange way to test performance. The map has only one continent, and it's not at all representative of commonly played maps. Testing by playing bots one-on-one is unrealistic too.

It seems to me that the way to get a good picture of bots' abilities would be to play them in the environment they were designed for and are used in, i.e. something normal like
-Classic map
-10% conts
-4,6,8,10,15,20... cards
-6 players per game
-random start

You could let the computer do the work of testing it, by setting all the bots to random, clearing the agent-based rankings, and letting the bots fight overnight on autoplay. Or better yet, let it go for a couple days. After a while, just sort them by win percentage.

AquaRegia · Post by **AquaRegia** » Thu Mar 02, 2006 4:07 pm

Greg was very diplomatic. I'll be more blunt:

blip, your "test" is useless for assessing any reasonable definition of game play. No games even remotely like that are ever played.

You are quite full of strong opinions for someone who joined the forums three days ago and who, as far as I can tell, has never played an online game of Lux.

By the way, welcome.

One Big Wave · Post by **One Big Wave** » Thu Mar 02, 2006 4:35 pm

AquaRegia wrote:I am a big scary mod bully....Fear me

blipadouzi · Post by **blipadouzi** » Thu Mar 02, 2006 5:02 pm

GregM wrote:Allow me to say that that seems a very strange way to test performance. The map has only one continent, and it's not at all representative of commonly played maps. Testing by playing bots one-on-one is unrealistic too.

It seems to me that the way to get a good picture of bots' abilities would be to play them in the environment they were designed for and are used in, i.e. something normal like
-Classic map
-10% conts
-4,6,8,10,15,20... cards
-6 players per game
-random start

You could let the computer do the work of testing it, by setting all the bots to random, clearing the agent-based rankings, and letting the bots fight overnight on autoplay. Or better yet, let it go for a couple days. After a while, just sort them by win percentage.

I totally agree with you, but for a first run at it, I wanted to find a way to compare them to each other while being able to eliminate or rather, rate the players. I only saw this possible in a one-on-one environment. But as I said, I do intend on making a battle-royal type tournament down the road.

AquaRegia wrote:Greg was very diplomatic. I'll be more blunt:

blip, your "test" is useless for assessing any reasonable definition of game play. No games even remotely like that are ever played.

You are quite full of strong opinions for someone who joined the forums three days ago and who, as far as I can tell, has never played an online game of Lux.

By the way, welcome.

Thanks for the welcome...as for my opinion, in this thread, I have none...I'm just stating the numbers as they came out. And even if it was my opinion and even if it was a strong one...I don't see how my being in the forums for only 3 days has any relevance to that.

I've been playing Risk in various shapes and forms for well over 25 years...and I've played "Lux" in paper format before Risk made it to computer in any form. (explanation)...I was captain of a Risk club in High School and one day when we were bored we got some 6' x 10' paper from the art department and drew a map of the world on it...it came out to having something like 1000+ provinces/states/countries, which in Risk terms were deemed as countries, and as a result we broke down some countries into continents...so we have about 25 continents.

The game went on for 3 months...8 players.

Anyway, now I know better...next time, I'll keep the results to myself.

Post by **dustin** » Thu Mar 02, 2006 5:11 pm

I think only having 1 continent screws with many of the bots.

I can also see why Angry would do OK. When pitted against a bot that attacks slower he will on average gain more land (because he attacks lots without reason)and have more income from that land to attack more, etc. Normally this gets balanced out by the other bots gaining income from the continents they have craftily captured.

paranoiarodeo · Post by **paranoiarodeo** » Thu Mar 02, 2006 5:30 pm

∞ paranoiarodeo laughs politely ∞

Fun little experiment, but I suspect the Nobel committee won't be calling anytime soon ...

Bertrand · Post by **Bertrand** » Thu Mar 02, 2006 5:39 pm

blipadouzi wrote:\Anyway, now I know better...next time, I'll keep the results to myself.

Your tests are interesting because they are wildly different than what bot authors planned for. Don't be put off by the negative comments: any discussion is better than no discussion at all. Debate is fun in all it's forms.

Reaper is not designed at all for 2 player games. When 2 players are left, he simply tries to take over the world. By the time 2 players are left, most games are already won. The strategy in a 6 player games is infinitely more complex: to kill or not, take a continent or not, pop a border or not, cash a card or not, farm or not... All these decisions are dependent on the game parameters (cards, cont % increase, map...).

The moral: the setup you use to compare bots (or humans for that matter) is very important.

blipadouzi · Post by **blipadouzi** » Thu Mar 02, 2006 5:46 pm

Bertrand wrote:
blipadouzi wrote:\Anyway, now I know better...next time, I'll keep the results to myself.
Your tests are interesting because they are wildly different than what bot authors planned for. Don't be put off by the negative comments: any discussion is better than no discussion at all. Debate is fun in all it's forms.

You know, you're right...thanks for the kick in the pants.

We all need those once in a while.

blipadouzi · Post by **blipadouzi** » Thu Mar 02, 2006 7:02 pm

I've thought it over, and I think I know how I can fairly and equitably assess the bots.

I'll run another tournament, picking 6 bots at random. Pit them against each other in ALL the maps. Then repeat this with another 6, then another 6 and so on until they haveall passed.

While they play, I will keep track of how many battles each of them won. Then I will take the top 6 and do it allover again, then the next 6 best and so on.

I will repeat this process 3 or 4 times until a clear and definite winner is determined.

The only thing I want to do and have no figured out is a scoring system. I'd like to give them more than just a 1 for winning the game...butsay a 200 for winning it in 3 minutes or a 100 for taking 5 minutes. Obviously since it took longer, the bot had a harder time to win. But that's my problem and is not really important. Any suggestions would be welcome.

For now, I won't plan on posting the results...but if anyone asks for them then I will.

Bertrand · Post by **Bertrand** » Thu Mar 02, 2006 7:21 pm

blipadouzi wrote:For now, I won't plan on posting the results...but if anyone asks for them then I will.

I'm asking... But if you want to test the bots on *all* the maps, it will take you... let me think... a few years!

Seriously, you should select just a few maps, classic and some highly rated ones like "american civil war" for example.

It's even more important to vary the game parameters. A 5-5-5 game is a very different beast than a 5-6-8-10-15 game: in 555 you have to worry about continents, and in the highcard games killing players is the only way to go. Most bots excell in only one style of play.

I thought that only bot authors were interested in bot wars... Have fun, and don't forget to play with them also.

blipadouzi · Post by **blipadouzi** » Thu Mar 02, 2006 9:20 pm

Bertrand wrote:I thought that only bot authors were interested in bot wars... Have fun, and don't forget to play with them also.

No worries there...I've had tons of hours of fun playing against them.

Stanski · Post by **Stanski** » Fri Mar 03, 2006 3:37 am

I will agree that nef does not perform as well as many of the other bots out there. From my experiments (this is random bots and maps, along with different settings) it looks like this based on win percentage and average place factored in together:
1. Botofdoom
2. Reaper
3. killbot
4. evilpixie
5. trotsky
6. boscoe
7. quo
8. brainiac
8 (close to a tie). bort
9. nefarious
10. yakool
11. shaft
12. communist
13 on down-angry, stinky, cluster, and pixie are all pretty equally bad

fenwick · Post by **fenwick** » Fri Mar 03, 2006 9:52 am

Blip, I love your Obssesion over this. I wish you would throw all this energy into creating an AI or a Map. Your out of the box thinking could introduce an interesting personality to the already rich Bot community.

blipadouzi · Post by **blipadouzi** » Fri Mar 03, 2006 11:52 am

I've created a map which was added to the plugin manager this morning..."Babylon 5", but it still needs some fine tuning.

As for AI creation, I downloaded the AI creator and decided I had to invest more time into learning it than just an hour or so...but I have posted some ideas for bots.

And thanks.

mercer · Post by **mercer** » Fri Mar 03, 2006 9:11 pm

I agree with fenwick, your obsession is entertaining and welcome - I also encourage you to apply this passion to creating your own robot, you clearly have the mind for it and I believe that there is plenty of room for some more robots.

Given the testing you have done so far, you should be able to create a robot which places top in the tests you have outlined so far - at which point most likely you will find that the problem immediately presents itself, as soon as you try and play this robot you create in a real game and find it doesn't play as well as other robots already in place....

...at which point I would hope that this insight would motivate you to further tweak your robot until you have a monster that can excel across the board at all maps and take top rankings!

mercer

Mike · Post by **Mike** » Fri Mar 03, 2006 10:41 pm

How in the hell did angry do well? I find fault in your methods, and think there must have been other forces corrupting the test. For one, angry is the worst bot (close second for worst next to stinky), second, Nefarious is far from the worst.

For my money, Boscoe is consistantly the best.

blipadouzi · Post by **blipadouzi** » Fri Mar 03, 2006 10:52 pm

Mike wrote:How in the hell did angry do well? I find fault in your methods, and think there must have been other forces corrupting the test. For one, angry is the worst bot (close second for worst next to stinky), second, Nefarious is far from the worst.

For my money, Boscoe is consistantly the best.

Like previously mentioned...the circumstances of a simgle continent and only 2 player battle allowed Angry to gain an advantage as the othr more powerful bots were out of their element.

But also as previously mentioned, as per suggestions made here, I will be starting over with a better and more refined testing method.

Alexander_the_Great · Post by **Alexander_the_Great** » Sat Mar 04, 2006 12:06 am

OMG, did I just spend the last 15 minutes reading all the posts within this thead ... I must have

In any case blipadouzi, you have been posting a lot of feedback/thoughts/comments/etc to the Map Theme and AI Creation area for a new guy ... which is really cool

I didn't even register in the forums for a few weeks after I started.

I dig your obsession and analytical approach to the game/bots/etc. I agree with a lot of what has been said by other players and as far as your tests go - at least they offer some insight into them ... or at least a different approach. I also agree that it would be great to see you put together an AI and from the looks of your interest I'd imagine he'd be pretty good.

Thanks for the info and I'm looking forward to more conclusive information on the bots as well as a new AI

- AtG

Dang · Post by **Dang** » Wed Aug 09, 2006 11:06 am

I ran a similar experiment to blips using waterworld and found that the only bot that can beat angry is Bot of Doom. and thats only if BoD has the first turn. In fact, on waterworld angry matched up against any 5 bots other than BoD wins every time precisely b/c of what someone else said about the angry bot going willy nilly and the others trying to establish fronts. Angry is only any good on large maps with no choke points and few if any continents. In addition, the rollover on the sides of waterworld help angry. On solar system dominance, angry constantly attacks 1 corner and never makes headway.

Bots analysis

Bots analysis

Re: Bots analysis

whew

Angry

Who is online