I have a question regarding Genetic Programming. I am going to work on a genetic algorithm for a game called Battleships.
My question is: How would I decide upon a "decision" model for the AI to evolve? And how does that work?
I have read multiple papers and multiple answers that just speak about using different models, but could not find something specific, which, unfortunately, I apparently need to wrap my head around the problem.
I want it to evolve over multiple iterations and "learn" what works best, but not sure how to save these "decisions" (I know to a file, but "encoded" how?) in a good way, so it will learn to take a stance to previous actions and base off info from the current board state.
I have been contemplating a "Tree structure" for the AI to base decisions on, but I don't actually know how to get started.
If someone could either point me in the right direction (a link? Some pseudo-code? Something like that), that'd be very much appreciated, I tried to google as much as possible, watch multiple youtube videos about the subject, but I think I just need that little nudge in the right direction.
I may also just not know what exactly to search for, and this is why I come up blank with results on what and how I implement this.
ANSWER PART I: The basis for a genetic algorithm is a having a group of actors, some of which reproduce. The fittest are chosen for reproduction and the offspring are copies of the parents that are slightly mutated. It's a pretty simple concept, but to program it you have to have actions that can be randomly chosen and dynamically modified. For the battleship simulation I created a class called a Shooter because it 'shoots' at a position. The assumption here is that the first position has been hit, and the shooter is now trying to sink the battleship.
public class Shooter implements Comparable<Shooter> {
private static final int NUM_SHOTS = 100;
private List<Position> shots;
private int score;
// Make a new set of random shots.
public Shooter newShots() {
shots = new ArrayList<Position>(NUM_SHOTS);
for (int i = 0; i < NUM_SHOTS; ++i) {
shots.add(newShot());
}
return this;
}
// Test this shooter against a ship
public void testShooter(Ship ship) {
score = shots.size();
int hits = 0;
for (Position shot : shots) {
if (ship.madeHit(shot)) {
if (++hits >= ship.getSize())
return;
} else {
score = score - 1;
}
}
}
// get the score of the testShotr operation
public int getScore() {
return score;
}
// compare this shooter to other shooters.
@Override
public int compareTo(Shooter o) {
return score - o.score;
}
// getter
public List<Position> getShots() {
return shots;
}
// reproduce this shooter
public Shooter reproduce() {
Shooter offspring = new Shooter();
offspring.mutate(shots);
return offspring;
}
// mutate this shooter's offspring
private void mutate(List<Position> pShots) {
// copy parent's shots (okay for shallow)
shots = new ArrayList<Position>(pShots);
// 10% new mutations, in random locations
for (int i = 0; i < NUM_SHOTS / 10; i++) {
int loc = (int) (Math.random() * 100);
shots.set(loc, newShot());
}
}
// make a new random move
private Position newShot() {
return new Position(((int) (Math.random() * 6)) - 3, ((int) (Math.random() * 6)) - 3);
}
}
The idea here is that a Shooter has up to 100 shots, randomly chosen between +-3 in the X and +- 3 in the Y. Yea, 100 shots is overkill, but hey, whatever. Pass a Ship to this Shooter.testShooter and it will score itself, 100 being the best score, 0 being the worst.
This Shooter actor has reproduce and mutate methods that will return an offspring that has 10% of its shots randomly mutated. The general idea is that the best Shooters have 'learned' to shoot their shots in a cross pattern ('+') as quickly as possible, since a ship is oriented in one of four ways (North, South, East, West).
The program that runs the simulation, ShooterSimulation, is pretty simple:
public class ShooterSimulation {
private int NUM_GENERATIONS = 1000;
private int NUM_SHOOTERS = 20;
private int NUM_SHOOTERS_NEXT_GENERATION = NUM_SHOOTERS / 10;
List<Shooter> shooters = new ArrayList<Shooter>(NUM_SHOOTERS);
Ship ship;
public static void main(String... args) {
new ShooterSimulation().run();
}
// do the work
private void run() {
firstGeneration();
ship = new Ship();
for (int gen = 0; gen < NUM_GENERATIONS; ++gen) {
ship.newOrientation();
testShooters();
Collections.sort(shooters);
printAverageScore(gen, shooters);
nextGeneration();
}
}
// make the first generation
private void firstGeneration() {
for (int i = 0; i < NUM_SHOOTERS; ++i) {
shooters.add(new Shooter().newShots());
}
}
// test all the shooters
private void testShooters() {
for (int mIdx = 0; mIdx < NUM_SHOOTERS; ++mIdx) {
shooters.get(mIdx).testShooter(ship);
}
}
// print the average score of all the shooters
private void printAverageScore(int gen, List<Shooter> shooters) {
int total = 0;
for (int i = 0, j = shooters.size(); i < j; ++i) {
total = total + shooters.get(i).getScore();
}
System.out.println(gen + " " + total / shooters.size());
}
// throw away the a tenth of old generation
// replace with offspring of the best fit
private void nextGeneration() {
for (int l = 0; l < NUM_SHOOTERS_NEXT_GENERATION; ++l) {
shooters.set(l, shooters.get(NUM_SHOOTERS - l - 1).reproduce());
}
}
}
The code reads as pseudo-code from the run method: make a firstGeneration then iterate for a number of generations. For each generation, set a newOrientation for the ship, then do testShooters, and sort the results of the test with Collections.sort. printAverageScore of the test, then build the nextGeneration. With the list of average scores you can, cough cough, do an 'analysis'.
A graph of the results looks like this:
As you can see it starts out with pretty low average scores, but learns pretty quickly. However, the orientation of the ship keeps changing, causing some noise in addition to the random component. Every now and again a mutation messes up the group a bit, but less and less as the group improves overall.
Challenges, and the reason for many papers to be sure, is to make more things mutable, especially in a constructive way. For example, the number of shots could be mutable. Or, replacing the list of shots with a tree that branches depending on whether the last shot was a hit or miss might improve things, but it's difficult to say. That's where the 'decision' logic considerations come in. Is it better to have a list of random shots or a tree that decides which branch to take depending on the prior shot? Higher level challenges include predicting what changes will make the group learn faster and be less susceptible to bad mutations.
Finally, consider that there could be multiple groups, one group a battleship hunter and one group a submarine hunter for example. Each group, though made of the same code, could 'evolve' different internal 'genetics' that allow them to specialize for their task.
Anyway, as always, start somewhere simple and learn as you go until you get good enough to go back to reading the papers.
PS> Need this too:
public class Position {
int x;
int y;
Position(int x, int y ) {this.x=x; this.y=y;}
@Override
public boolean equals(Object m) {
return (((Position)m).x==x && ((Position)m).y==y);
}
}
UDATE: Added Ship class, fixed a few bugs:
public class Ship {
List<Position> positions;
// test if a hit was made
public boolean madeHit(Position shot) {
for (Position p: positions) {
if ( p.equals(shot)) return true;
}
return false;
}
// make a new orientation
public int newOrientation() {
positions = new ArrayList<Position>(3);
// make a random ship direction.
int shipInX=0, oShipInX=0 , shipInY=0, oShipInY=0;
int orient = (int) (Math.random() * 4);
if( orient == 0 ) {
oShipInX = 1;
shipInX = (int)(Math.random()*3)-3;
}
else if ( orient == 1 ) {
oShipInX = -1;
shipInX = (int)(Math.random()*3);
}
else if ( orient == 2 ) {
oShipInY = 1;
shipInY = (int)(Math.random()*3)-3;
}
else if ( orient == 3 ) {
oShipInY = -1;
shipInY = (int)(Math.random()*3);
}
// make the positions of the ship
for (int i = 0; i < 3; ++i) {
positions.add(new Position(shipInX, shipInY));
if (orient == 2 || orient == 3)
shipInY = shipInY + oShipInY;
else
shipInX = shipInX + oShipInX;
}
return orient;
}
public int getSize() {
return positions.size();
}
}
I would suggest you another approach. This approach is based on the likelihood where a ship can be. I will show you an example on a smaller version of the game (the same idea is for all other versions). In my example it is 3x3 area and has only one 1x2 ship.
Now you take an empty area, and put the ship in all possible positions (storing the number of times the part of the ship was in the element of the matrix). If you will do this for a ship 1x2, you will get the following
1 2 1
1 2 1
1 2 1
Ship can be in another direction 2x1 which will give you the following matrix:
1 1 1
2 2 2
1 1 1
Summing up you will get the matrix of probabilities:
2 3 2
3 4 3
2 3 2
This means that the most probable location is the middle one (where we have 4). Here is where you should shoot.
Now lets assume you hit the part of the ship. If you will recalculate the likelihood matrix, you will get:
0 1 0
1 W 1
0 1 0
which tells you 4 different possible positions for a next shoot.
If for example you would miss on the previous step, you will get the following matrix:
2 2 2
2 M 2
2 2 2
This is the basic idea. The way how you try to reposition the ships is based on the rules how the ships can be located and also what information you got after each move. It can be missed/got or missed/wounded/killed.
ANSWER PART III: As you can see, the Genetic Algorithm is generally not the hard part. Again, it's a simple piece of code that is really meant to exercise another piece of code, the actor. Here, the actor is implemented in a Shooter class. These actor's are often modelled in the fashion of Turning Machines, in the sense that the actor has a defined set of outputs for a set of inputs. The GA helps you to determine the optimal configuration of the state table. In the prior answers to this question, the Shooter implemented a probability matrix like what was described by @SalvadorDali in his answer.
Testing the prior Shooter thoroughly, we find that the best it can do is something like:
BEST Ave=5, Min=3, Max=9
Best=Shooter:5:[(1,0), (0,0), (2,0), (-1,0), (-2,0), (0,2), (0,1), (0,-1), (0,-2), (0,1)]
This shows it takes 5 shots average, 3 at a minimum, and 9 at a maximum to sink a 3X3 battleship. The locations of the 9 shots are shown a X/Y coordinate pairs. The question "Can this be done better?" depends on human ingenuity. A Genetic Algorithm can't write new actors for us. I wondered if a decision tree could do better than a probability matrix, so I implemented one to try it out:
public class Branch {
private static final int MAX_DEPTH = 10;
private static final int MUTATE_PERCENT = 20;
private Branch hit;
private Branch miss;
private Position shot;
public Branch() {
shot = new Position(
(int)((Math.random()*6.0)-3),
(int)((Math.random()*6.0)-3)
);
}
public Branch(Position shot, Branch hit, Branch miss) {
this.shot = new Position(shot.x, shot.y);
this.hit = null; this.miss = null;
if ( hit != null ) this.hit = hit.clone();
if ( miss != null ) this.miss = miss.clone();
}
public Branch clone() {
return new Branch(shot, hit, miss);
}
public void buildTree(Counter c) {
if ( c.incI1() > MAX_DEPTH ) {
hit = null;
miss = null;
c.decI1();
return;
} else {
hit = new Branch();
hit.buildTree(c);
miss = new Branch();
miss.buildTree(c);
}
c.decI1();
}
public void shoot(Ship ship, Counter c) {
c.incI1();
if ( ship.madeHit(shot)) {
if ( c.incI2() == ship.getSize() ) return;
if ( hit != null ) hit.shoot(ship, c);
}
else {
if ( miss != null ) miss.shoot(ship, c);
}
}
public void mutate() {
if ( (int)(Math.random() * 100.0) < MUTATE_PERCENT) {
shot.x = (int)((Math.random()*6.0)-3);
shot.y = (int)((Math.random()*6.0)-3);
}
if ( hit != null ) hit.mutate();
if ( miss != null ) miss.mutate();
}
@Override
public String toString() {
StringBuilder sb = new StringBuilder();
sb.append(shot.toString());
if ( hit != null ) sb.append("h:"+hit.toString());
if ( miss != null ) sb.append("m:"+miss.toString());
return sb.toString();
}
}
The Branch class is a node in a decision tree (ok, maybe poorly named). At every shot, the next branch chosen depends on whether the shot was awarded a hit or not.
The shooter is modified somewhat to use the new decisionTree.
public class Shooter implements Comparable<Shooter> {
private Branch decisionTree;
private int aveScore;
// Make a new random decision tree.
public Shooter newShots() {
decisionTree = new Branch();
Counter c = new Counter();
decisionTree.buildTree(c);
return this;
}
// Test this shooter against a ship
public int testShooter(Ship ship) {
Counter c = new Counter();
decisionTree.shoot(ship, c);
return c.i1;
}
// compare this shooter to other shooters, reverse order
@Override
public int compareTo(Shooter o) {
return o.aveScore - aveScore;
}
// mutate this shooter's offspring
public void mutate(Branch pDecisionTree) {
decisionTree = pDecisionTree.clone();
decisionTree.mutate();
}
// min, max, setters, getters
public int getAveScore() {
return aveScore;
}
public void setAveScore(int aveScore) {
this.aveScore = aveScore;
}
public Branch getDecisionTree() {
return decisionTree;
}
@Override
public String toString() {
StringBuilder ret = new StringBuilder("Shooter:"+aveScore+": [");
ret.append(decisionTree.toString());
return ret.append(']').toString();
}
}
The attentive reader will notice that while the methods themselves have changed, which methods a Shooter needs to implement is not different from the prior Shooters. This means the main GA simulation has not changed except for one line related to mutations, and that probably could be worked on:
Shooter child = shooters.get(l);
child.mutate( shooters.get(NUM_SHOOTERS - l - 1).getDecisionTree());
A graph of a typical simulation run now looks like this:
As you can see, the final best average score evolved using a Decision Tree is one shot less than the best average score evolved for a Probability Matrix. Also notice that this group of Shooters has taken around 800 generations to train to their optimum, about twice as long than the simpler probability matrix Shooters. The best decision tree Shooter gives this result:
BEST Ave=4, Min=3, Max=6
Best=Shooter:4: [(0,-1)h:(0,1)h:(0,0) ... ]
Here, not only does the average take one shot less, but the maximum number of shots is 1/3 lower than a probability matrix Shooter.
At this point it takes some really smart guys to determine whether this actor has achieved the theoretical optimum for the problem domain, i.e., is this the best you can do trying to sink a 3X3 ship? Consider that the answer to that question would become more complex in the real battleship game, which has several different size ships. How would you build an actor that incorporates the knowledge of which of the boats have already been sunk into actions that are randomly chosen and dynamically modified? Here is where understanding Turing Machines, also known as CPUs, becomes important.
PS> You will need this class also:
public class Counter {
int i1;
int i2;
public Counter() {i1=0;i2=0;}
public int incI1() { return ++i1; }
public int incI2() { return ++i2; }
public int decI1() { return --i1; }
public int decI2() { return --i2; }
}
ANSWER PART II: A Genetic Algorithm is not a end unto itself, it is a means to accomplish an end. In the case of this example of battleship, the end is to make the best Shooter. I added the a line to the prior version of the program to output the best shooter's shot pattern, and noticed something wrong:
Best shooter = Shooter:100:[(0,0), (0,0), (0,0), (0,-1), (0,-3), (0,-3), (0,-3), (0,0), (-2,-1) ...]
The first three shots in this pattern are at coordinates (0,0), which in this application are guaranteed hits, even though they are hitting the same spot. Hitting the same spot more than once is against the rules in battleship, so this "best" shooter is the best because it has learned to cheat!
So, clearly the program needs to be improved. To do that, I changed the Ship class to return false if a position has already been hit.
public class Ship {
// private class to keep track of hits
private class Hit extends Position {
boolean hit = false;
Hit(int x, int y) {super(x, y);}
}
List<Hit> positions;
// need to reset the hits for each shooter test.
public void resetHits() {
for (Hit p: positions) {
p.hit = false;
}
}
// test if a hit was made, false if shot in spot already hit
public boolean madeHit(Position shot) {
for (Hit p: positions) {
if ( p.equals(shot)) {
if ( p.hit == false) {
p.hit = true;
return true;
}
return false;
}
}
return false;
}
// make a new orientation
public int newOrientation() {
positions = new ArrayList<Hit>(3);
int shipInX=0, oShipInX=0 , shipInY=0, oShipInY=0;
// make a random ship orientation.
int orient = (int) (Math.random() * 4.0);
if( orient == 0 ) {
oShipInX = 1;
shipInX = 0-(int)(Math.random()*3.0);
}
else if ( orient == 1 ) {
oShipInX = -1;
shipInX = (int)(Math.random()*3.0);
}
else if ( orient == 2 ) {
oShipInY = 1;
shipInY = 0-(int)(Math.random()*3.0);
}
else if ( orient == 3 ) {
oShipInY = -1;
shipInY = (int)(Math.random()*3.0);
}
// make the positions of the ship
for (int i = 0; i < 3; ++i) {
positions.add(new Hit(shipInX, shipInY));
if (orient == 2 || orient == 3)
shipInY = shipInY + oShipInY;
else
shipInX = shipInX + oShipInX;
}
return orient;
}
public int getSize() {
return positions.size();
}
}
After I did this, my shooters stopped "cheating", but that got me to thinking about the scoring in general. What the prior version of the application was doing was scoring based on how many shots missed, and hence a shooter could get a perfect score if none of the shots missed. However, that is unrealistic, what I really want is shooters that shoot the least shots. I changed the shooter to keep track of the average of shots taken:
public class Shooter implements Comparable<Shooter> {
private static final int NUM_SHOTS = 40;
private List<Position> shots;
private int aveScore;
// Make a new set of random shots.
public Shooter newShots() {
shots = new ArrayList<Position>(NUM_SHOTS);
for (int i = 0; i < NUM_SHOTS; ++i) {
shots.add(newShot());
}
return this;
}
// Test this shooter against a ship
public int testShooter(Ship ship) {
int score = 1;
int hits = 0;
for (Position shot : shots) {
if (ship.madeHit(shot)) {
if (++hits >= ship.getSize())
return score;
}
score++;
}
return score-1;
}
// compare this shooter to other shooters, reverse order
@Override
public int compareTo(Shooter o) {
return o.aveScore - aveScore;
}
... the rest is the same, or getters and setters.
}
I also realized that I had to test each shooter more than once in order to be able to get an average number of shots fired against battleships. For that, I subjected each shooter individually to a test multiple times.
// test all the shooters
private void testShooters() {
for (int i = 0, j = shooters.size(); i<j; ++i) {
Shooter current = shooters.get(i);
int totalScores = 0;
for (int play=0; play<NUM_PLAYS; ++play) {
ship.newOrientation();
ship.resetHits();
totalScores = totalScores + current.testShooter(ship);
}
current.setAveScore(totalScores/NUM_PLAYS);
}
}
Now, when I run the simulation, I get the average of the averages an output. The graph generally looks something like this:
Again, the shooters learn pretty quickly, but it takes a while for random changes to bring the averages down. Now my best Shooter makes a little more sense:
Best=Shooter:6:[(1,0), (0,0), (0,-1), (2,0), (-2,0), (0,1), (-1,0), (0,-2), ...
So, a Genetic Algorithm is helping me to set the configuration of my Shooter, but as another answer here pointed out, good results can be achieved just by thinking about it. Consider that if I have a neural network with 10 possible settings with 100 possible values in each setting, that's 10^100 possible settings and the theory for how those settings should be set may a little more difficult than battleship shooter theory. In this case, a Genetic Algorithm can help determine optimal settings and test current theory.
来源：https://stackoverflow.com/questions/35820854/ai-how-to-model-genetic-programming-for-battleships