In this project, our aim is to design and program a pokerbot which is able to participate in a Kuhn Poker game with others. By classifying the received cards, it is supposed to make suitable actions based on our own designed strategy.
Kuhn Poker is an extremely simplified poker game. As a simple model zero-sum two-player imperfect-information game, it is amenable to a complete game-theoretic analysis. More details of its rules will be introduced in our agent and strategy part.
To achieve the goal, our task is divided into two main parts. One part is to build and train a model to classify the cards given by the server. The other is to develop an agent with a suitable strategy to make actions after receiving these cards. Details are introduced as follow.
- Initial Condition Setup
Set the parameters noise level = 0.7 and rotation = 180.
Generate 5000 training data and 1000 testing data with noise level randomly between 0 to 0.7 and rotation degree randomly between 0 to 180.
For examples:
-
Feature Extraction Process Except change the value from analog (0 to 255) to digital (0 and 1), also apply noise cancellation in the extraction process.
Step 1: Apply a light Gaussian Blur
Step 2: Apply a loose single-value threshold to filter some light noise out
Step 3: Apply a 3x3 averaging kernel to get a stronger blur effect
Step 4: Apply a hysteresis threshold to nicely keep the shape of words -
Result Comparison
Before / After
A_9: 
A_12: 
J_14: 
J_42: 
K_39: 
K_75: 
Q_49: 
Q_56: 
-
Destination Classify four different cards
J Q K Aby using 32x32 pixel images. -
Model Architecture Based on the concept of VGG structure, use TensorFlow package to create a small five-layer CNN model.
Due to the easy classification task, only use three convolution layers to extract the feature and two fully connected layers to fulfill the goal.
Take ReLU as the activation layer after each convolution layer and fully connected layer, except the last layer.
Also, apply 2x2 max pooling layer after each convolution layer. At the end, apply Softmax as the activation function at the last layer to cooperate with cross-entropy loss function due to the one-hot coding labels.
-
Traning Result With 5000 training data and 1000 testing generated by 0-180 degree rotation and 0-0.7 noise level, our testing accurcy is 0.973.
It is robust enough to deal with high noisy inputs and still perform very well.
- Check https://en.wikipedia.org/wiki/Kuhn_poker.
- Note that there is a difference between our project and the game introduced in Wikipedia. In this project, the cards can be 3 or 4, which is set by the backend.
- The confidence of BET or CALL is different for different card in hand, last move and outcome from last round.
- For example, the confidence of BET while having "J" in hand is small. And the confidence of BET is large for "A" or "K" in hand.
- When last move is BET, the confidence of BET should be smaller than before for the purpose of being cautious. Vice versa. This is achieved by multiplying or dividing a number bigger than 1. It is called decay.
- When outcome from last round is negative and last move is BET or CALL, then the confidence of BET is smaller to be more cautious. Vice versa. This is achieved by updating the decay.
- While making actions, a random number flag in (0,1) is created. If flag > confidence, then we BET or CALL. Otherwise, we CHECK or FOLD.
- Decay will be passed to next round. Confidence remains same all the time.
- If we have J or A (or K for 3 cards game), decay will not be considered. That is, for J and A, we give a fixed confidence to bet.
| Possibility to bet (confidence) | J | Q | K | A |
|---|---|---|---|---|
| First Action(Player1) | 0.05 | 0.3 | 0.55 | 0.9 |
| Second Action(Player2) | 0.05 | 0.3*decay or 0.3/decay | 0.55*decay or 0.55/decay | 0.9 |
| Third Action(Player1) | 0.05 | 0.3*decay or 0.3/decay | 0.55*decay or 0.55/decay | 0.9 |
Before running our codes, some pakages are needed to be installed. These pakages has been listed in requirement-linux.txt and requirement-windows.txt. More specific steps to install them have been provided in README.md.
To play local game, we need to have a local KuhnPoker server installed and running in the background. With a token given by the local server, we can connect to it and then start our game automatically.
We can also choose to play the game with others on the public server. We have to specify a --global flag for the script and then wait for others to join. More details about how to realize it have also been provided in README.md.
In this part, most of the important functions in our python files will be introduced.
File data_sets is used to deal with data before training, including generating training and testing data.
Function extract_features converts an image to features that serve as input to the image classifier.
Parameters:
- img: Image
Image to convert to features.- file_name: str, default = None
Passing the file name to save_without_generate() to save the feature image for easy observation.
Returns:
- featres: list/matrix/structure of int, int between zero and one
Extracted features in a format that can be used in the image classifier.
Function load_data_set prepares features for the images in data_dir and divide in a training and validation set.
Parameters:
- data_dir: str
Directory of images to load.- n_validation: int, default = 0
Number of images that are assigned to the validation set.
Returns:
- featres: list/matrix/structure of int, int between zero and one
Extracted features in a format that can be used in the image classifier.
Function generate_data_set generates n_samples noisy images by using generate_noisy_image(), and store them in data_dir.
Parameters:
- n_samples: int
Number of train/test examples to generate.- data_dir: str in [TRAINING_IMAGE_DIR, TEST_IMAGE_DIR]
Directory for storing images.
Function generate_noisy_image generates a noisy image with a given noise corruption. This implementation mirrors how the server generates the images. However the exact server settings for noise_level and ROTATE_MAX_ANGLE are unknown.
Parameters:
- rank: str in ['J', 'Q', 'K', 'A']
Original card rank.- noise_level: int between zero and one
Probability with which a given pixel is randomized.
Returns:
- noisy_img: Image
A noisy image representation of the card rank.
Fuction save_without_generate is used to save image sperately with corresponding name.
Parameters:
- img: Image
Specific feature image for storing.- file_name: string
Specific name correspond to the feature image.- data_dir_feature: string
Target directory.
File model is used to build and train our own model for classifying card images given by the server in the games.
Function build_model adds convolutional layers and fully connected layers to define the model.
Returns:
- model: model class
Return the untrained model.
Function train_model fits the model on the training data set.
Parameters:
- model: model class
Model structure to fit, as defined by build_model.- n_validation: int
Number of training examples used for cross-validation.- write_to_file bool
Write model to file; can later be loaded through load_model.
Returns:
- model: model class
Return the trained model.
Function load_model is used to load a model from file using load_model in keras.
Returns:
- model: model class
Return the previously trained model.
Function evaluate_model evaluates our model on the test set.
Parameters:
- model: model class
Model structure to fit, as defined by build_model.- data_dir: int
Number of training examples used for cross-validation.
Returns:
- score: float
Return a measure of model performance.
The function identify uses the trained model to classify a single card image.
Parameters:
- image: Image
Image to classify.- model: model class
Trained model.
File agent is used to make actions based on our designed strategy in the game. Some important functions used in agent are introduced in the following.
Function __init__ is use to load our trained model, determine the game type and initialize other parameters in our agent.
Parameters:
- game_type: str, {'3', '4'}
The parameter game_type determines how many cards we have when playing the game. With different numbers of cards, our __init__ function will choose different confidence designed for the game.
Function make_action is used to choose a new action depending on the current state of the game. This method implements our PokerBot strategy designed above. By using the state and round arguments, we can decide our next best move.
Parameters:
- state: ClientGameState
The parameter ClientGameState tracks the state object of the current game. A game consists of multiple rounds from deal to showdown.- round: ClientGameRoundState
The parameter ClientGameRoundState tracks the state object of the current round, from deal to showdown.
Returns:
- Actions : str, {'BET', 'CALL', 'CHECK', 'FOLD'} (and in round.get_available_actions()) A string representation of the next action an agent wants to do next, should be from a list of available actions.
Function on_image is called every time when the card image changes. Use this method for image recongition procedure.
Parameters:
- image: Image
The parameter image tracks the current Image object.
Function on_error will be called in case of error either from server backend or from client itself. It is easy to use this function for error handling, including logging and raising error.
Parameters:
- error: str
The parameter error records a string representation of the current error.
Function on_game_start will be called once at the beginning of the game when server confirms both players have connected. It will check whether .\log folder exists or not. If not then, it will make one. Meanwhile, it can initialize the conf and decay array.
Function on_new_round_request is called every time before a new round is started. A new round is started automatically.
Parameters:
- state: ClientGameState
State object of the current game.
Funtion on_round_end is called every time a round has ended. A round ends automatically. It will log last round's result, if any, update current decay and log it.
Parameters:
- state: ClientGameState
State object of the current game.- round: ClientGameRoundState
State object of the current round.
Function on_game_end is called once after the game has ended. A game ends automatically and it will print the result of the game.
Parameters:
- state: ClientGameState
State object of the current game.
Result: str, {'WIN', 'DEFEAT'} End result of the game.
Function __init_conf_decay is called at game start to initialize conf and decay array according to the game type ['3', '4']. The array is in the order of ['J', 'Q', 'K', 'A'(if game type='4')]. Tune the numbers for Q and K to raise possibility to win.
Function __update_decay is called to update decay when every round ends. New values of decay depend on the last game's result and the last move.
Parameters:
- outcome: str
The parameter outcome records the result of the last game.- last_move: str
The parameter last_move records the last move.
Software project management is a sub-discipline of project management in which software projects are planned, implemented, monitored and controlled..1
In this part, the planning, implementing, monitoring and controlling are introduced. The team is gathered by Detian Guo based on personal relationship and team member's availability.
At the beginning of a project, the team identified the scope of the project, and split it to a set of tasks must be completed. Based on a naïve work load estimation, the team set up a project schedule as below.
The team is split to two sub teams. One focused on how to find and realize the strategy to win a Kuhn Poker game. The other one focused on find the best model for poker card identification. Plan was monitored and adjusted according to project status.
The team scheduled a weekly meeting to check how did the plan went in the last week. Difficulties by achieving the best strategy and/or the best model were also discussed. The plan was adjusted according to real workload rather than estimated workload.
The quality, time plan and risk of a project are the three primary elements to be managed for a project. In this assignment, our keen resource to manage is the available time of team members, which plays an important role on the three pillars mentioned above. This is achieved by the weekly meeting.
- [1][WikiPedia](https://en.wikipedia.org/wiki/Software_project_management)