Facebook researchers have developed a general AI framework called Recursive Belief-based Learning (ReBeL) that they say achieves better-than-human performance in heads-up, no-limit Texas holdem poker while using less domain knowledge than any prior poker AI. They assert that ReBeL is a step toward developing universal techniques for multi-agent interactions in other words, general algorithms that can be deployed in large-scale, multi-agent settings. Potential applications run the gamut from auctions, negotiations, and cybersecurity to self-driving cars and trucks.
Combining reinforcement learning with search at AI model training and test time has led to a number of advances. Reinforcement learning is where agents learn to achieve goals by maximizing rewards, while search is the process of navigating from a start to a goal state. For example, DeepMinds AlphaZero employed reinforcement learning and search to achieve state-of-the-art performance in the board games chess, shogi, and Go. But the combinatorial approach suffers a performance penalty when applied to imperfect-information games like poker (or even rock-paper-scissors), because it makes a number of assumptions that dont hold in these scenarios. The value of any given action depends on the probability that its chosen, and more generally, on the entire play strategy.
The Facebook researchers propose that ReBeL offers a fix. ReBeL builds on work in which the notion of game state is expanded to include the agents belief about what state they might be in, based on common knowledge and the policies of other agents. ReBeL trains two AI models a value network and a policy network for the states through self-play reinforcement learning. It uses both models for search during self-play. The result is a simple, flexible algorithm the researchers claim is capable of defeating top human players at large-scale, two-player imperfect-information games.
At a high level, ReBeL operates on public belief states rather than world states (i.e., the state of a game). Public belief states (PBSs) generalize the notion of state value to imperfect-information games like poker; a PBS is a common-knowledge probability distribution over a finite sequence of possible actions and states, also called a history. (Probability distributions are specialized functions that give the probabilities of occurrence of different possible outcomes.) In perfect-information games, PBSs can be distilled down to histories, which in two-player zero-sum games effectively distill to world states. A PBS in poker is the array of decisions a player could make and their outcomes given a particular hand, a pot, and chips.
Above: Poker chips.
Image Credit: Flickr: Sean Oliver
ReBeL generates a subgame at the start of each game thats identical to the original game, except its rooted at an initial PBS. The algorithm wins it by running iterations of an equilibrium-finding algorithm and using the trained value network to approximate values on every iteration. Through reinforcement learning, the values are discovered and added as training examples for the value network, and the policies in the subgame are optionally added as examples for the policy network. The process then repeats, with the PBS becoming the new subgame root until accuracy reaches a certain threshold.
In experiments, the researchers benchmarked ReBeL on games of heads-up no-limit Texas holdem poker, Liars Dice, and turn endgame holdem, which is a variant of no-limit holdem in which both players check or call for the first two of four betting rounds. The team used up to 128 PCs with eight graphics cards each to generate simulated game data, and they randomized the bet and stack sizes (from 5,000 to 25,000 chips) during training. ReBeL was trained on the full game and had $20,000 to bet against its opponent in endgame holdem.
The researchers report that against Dong Kim, whos ranked as one of the best heads-up poker players in the world, ReBeL played faster than two seconds per hand across 7,500 hands and never needed more than five seconds for a decision. In aggregate, they said it scored 165 (with a standard deviation of 69) thousandths of a big blind (forced bet) per game against humans it played compared with Facebooks previous poker-playing system, Libratus, which maxed out at 147 thousandths.
For fear of enabling cheating, the Facebook team decided against releasing the ReBeL codebase for poker. Instead, they open-sourced their implementation for Liars Dice, which they say is also easier to understand and can be more easily adjusted. We believe it makes the game more suitable as a domain for research, they wrote in the a preprint paper. While AI algorithms already exist that can achieve superhuman performance in poker, these algorithms generally assume that participants have a certain number of chips or use certain bet sizes. Retraining the algorithms to account for arbitrary chip stacks or unanticipated bet sizes requires more computation than is feasible in real time. However, ReBeL can compute a policy for arbitrary stack sizes and arbitrary bet sizes in seconds.
See more here:
Facebook develops AI algorithm that learns to play poker on the fly - VentureBeat
- Podcast: Lexi Gavin-Mather talks about her life in poker and finding love on the circuit - PokerStars - April 24th, 2024 [April 24th, 2024]
- Crypto Whales Bet 300K USDT On A Crazy Hand Of Poker At CoinPoker - Cryptonews - April 24th, 2024 [April 24th, 2024]
- Two implicated in poker box theft | News - Jamaica Star Online - April 24th, 2024 [April 24th, 2024]
- Taxi driver and tattoo artist held after cops find poker boxes in car | Loop Jamaica - Loop News Jamaica - April 24th, 2024 [April 24th, 2024]
- Two charged after theft of poker boxes from St Elizabeth bar - Jamaica Observer - April 24th, 2024 [April 24th, 2024]
- Introducing Lukas "RobinPoker" Robinson's Poker on a Boat: The WPT Voyage Experience - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- Support The Quad City Areas VFWs During This Poker Run - 97X - April 24th, 2024 [April 24th, 2024]
- Poker Player Ryan Garcia Upsets Heavily Favored Devin Haney in What Should Have Been a Title Fight - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- Dustin Melanson Crushes Final Table to Become WSOPC Playground Main Event Champion - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- Above the Felt's Poker Room Grand Opening in Wisconsin - World Casino Directory - April 24th, 2024 [April 24th, 2024]
- Poker in the Ears Episode 305: Lexy Gavin-Mather Tells Her Poker Story - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- Poker Hall of Famer Crandell Addington Passes Away at 85 - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- The best NBA poker players of all time - Dunkest - April 24th, 2024 [April 24th, 2024]
- Best Offshore Poker Sites for the US Players in 2024 - ReadWrite - April 24th, 2024 [April 24th, 2024]
- "Groundbreaking Ruling" Sees Poker Sites Ordered to Refund Losses to Players - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- Josh Reichard Leads Final 16 in WPT SHRPS; Lonis, Linde & Tice in Contention - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- BetMGM And Z4 Nevada Online Poker Rooms Expected In 2025 - PlayUSA - April 24th, 2024 [April 24th, 2024]
- BR Boat Club hosts 35th annual poker run to benefit Dreams Come True - Unfiltered with Kiran - April 24th, 2024 [April 24th, 2024]
- News: Poker player forfeits chip lead for birth of his son - pokerstrategy.com - April 24th, 2024 [April 24th, 2024]
- The 6 Best Tether Poker Sites in 2024 USDT Poker Sites Reviewed - Cryptonews - April 24th, 2024 [April 24th, 2024]
- Lorenz Schollhorn Wins the Biggest-Ever WPT Prime Gold Coast Championship - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- World Series of Poker Broadcast Schedule Released - CardsChat.com - April 24th, 2024 [April 24th, 2024]
- Tarpon man used housemate's shotgun to ambush him after night of poker, police say - Tampa Bay Times - April 24th, 2024 [April 24th, 2024]
- Bubble Bursts on Day 2 of WPT SHRPS as 101 Players Advance; Josh Reichard Leads - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- "Dirty Diaper" Comes Back to Haunt Rigby as Opponents Play it Back at WPT SHRPS - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- "One of My Biggest Dreams": Reigning POY Bin Weng Reflects on WPT SHRPS Victory - PokerNews.com - April 24th, 2024 [April 24th, 2024]
- 'Like poker night for women': Younger generations are clicking with mahjong - The Jewish News of Northern California - April 24th, 2024 [April 24th, 2024]
- Robin Weigert ('We Were the Lucky Ones') reveals that tennis, poker and water parks helped the cast bond for the ... - WiscNews - April 24th, 2024 [April 24th, 2024]
- Millions of Dollars on the Table in This Week's High Stakes Poker Episode - PokerNews.com - April 2nd, 2024 [April 2nd, 2024]
- Ben Ludlow Riding Wave of Momentum onto WPT Voyage - World Poker Tour - April 2nd, 2024 [April 2nd, 2024]
- News: WSOP Mystery Millions to award the bracelet at random - pokerstrategy.com - April 2nd, 2024 [April 2nd, 2024]
- ACR Poker Guaranteeing Thousands Of Online Seats With Venom Fever - Poker News - CardPlayer.com - April 2nd, 2024 [April 2nd, 2024]
- Blind River Bar Reopening In Time For Tickfaw 200 Poker Run - SpeedontheWater - April 2nd, 2024 [April 2nd, 2024]
- Mystery Bounties, Ontario Online Poker and the Sale: Q&A with Partypoker's Jaime Staples - Pokerfuse - April 2nd, 2024 [April 2nd, 2024]
- News: Second half of 2024 WPT schedule is out - pokerstrategy.com - April 2nd, 2024 [April 2nd, 2024]
- Over $1500 Added Value in our PokerNews Home Games on PokerStars in April - PokerNews.com - April 2nd, 2024 [April 2nd, 2024]
- PokerNews Freerolls Every Weekend on WPT Global - PokerNews.com - April 2nd, 2024 [April 2nd, 2024]
- NFL Draft is poker season, and Michigan's J.J. McCarthy might simply be its latest QB bluff - Yahoo Sports - April 2nd, 2024 [April 2nd, 2024]
- Local goes all-in with poker club - Sandusky Register - April 2nd, 2024 [April 2nd, 2024]
- The Commerce Casino & Hotel and the World Series of Poker (WSOP) Forge Historic Partnership - PR Newswire - April 2nd, 2024 [April 2nd, 2024]
- World Series of Poker at Turning Stone awards record $6.2 million in winnings (See top players) - syracuse.com - April 2nd, 2024 [April 2nd, 2024]
- Jakub Michalak Puts on a Grand Performance in the Merit Poker Carmen Series High Roller - PokerNews.com - April 2nd, 2024 [April 2nd, 2024]
- Snake Helps Thief With Casino Robbery - Poker News - CardPlayer.com - April 2nd, 2024 [April 2nd, 2024]
- Life Outside Poker: Farah Galfond on Days of Our Lives, Balancing Poker & Family - PokerNews.com - April 2nd, 2024 [April 2nd, 2024]
- Commerce Casino Announced As New Home of The 2024 World Series of Poker Tournament of Champions - Poker ... - CardPlayer.com - April 2nd, 2024 [April 2nd, 2024]
- Attention, Ontario Poker Players: You Can Win a WSOP 2024 Main Event Seat for Just $20 - Pokerfuse - April 2nd, 2024 [April 2nd, 2024]
- Everygame Poker Launches Fast Fortune - Their Own Spin & Go Game! - Business Wire - April 2nd, 2024 [April 2nd, 2024]
- PokerStars UKIPT and ESPT Team Up for Spectacular Summer Stop in Spain - PokerNews.com - March 27th, 2024 [March 27th, 2024]
- How poker chips sparked Marquette to its 87-69 comeback win over Western Kentucky - Marquette Wire - March 27th, 2024 [March 27th, 2024]
- Brett Murray Embraces His Role as The God of Thunder - World Poker Tour - March 27th, 2024 [March 27th, 2024]
- $100000 Paid Out to Whistleblowers: How WPT Global Rewards "Integrity in Poker" - Pokerfuse - March 27th, 2024 [March 27th, 2024]
- Video Poker Takes Your Money In 10 Lines Of BASIC - Hackaday - March 27th, 2024 [March 27th, 2024]
- Plug Pulled 500 Hours Into Poker Bankroll Challenge: "We Just Take the L" - PokerNews.com - March 27th, 2024 [March 27th, 2024]
- Everygame Poker Offer Free Spins on Popular Slot Releases - World Casino Directory - March 27th, 2024 [March 27th, 2024]
- Everygame Poker's Easter Free Spins feature New Rockstar World Tour and No Deposit Bonus for Chinese Slot - WebWire - March 27th, 2024 [March 27th, 2024]
- Balatro's dev doesn't really care for Poker, almost named the game Joker Poker - GoNintendo - March 27th, 2024 [March 27th, 2024]
- Junglee Poker's Dhina Dhin Dha Tournament Series Wraps up with a Bang More Action-Packed Events Anticipated Ahead - The Week - March 27th, 2024 [March 27th, 2024]
- Fausto Tantillo Goes Runner-Runner to Snag the Chip Lead on Day 2 of the Merit Poker Carmen Series Main Event - PokerNews.com - March 27th, 2024 [March 27th, 2024]
- World Poker Tour turns to AI dubbing - Broadcast - March 27th, 2024 [March 27th, 2024]
- Fouad Rabahie Becomes the Maestro of ESPT Madrid Main Event - PokerNews.com - March 27th, 2024 [March 27th, 2024]
- John Malkovich Reflects on Iconic Rounders Role as Teddy KGB in GQ Interview - PokerNews.com - March 27th, 2024 [March 27th, 2024]
- This Week in Poker Podcasts for March 25, 2024 - Pokerfuse - March 27th, 2024 [March 27th, 2024]
- Rumored Ryan Garcia in Millions of Debt from Poker - HighstakesDB - March 12th, 2024 [March 12th, 2024]
- 'Pioneer Woman' Ree Drummond credits church and poker for family reunion - Yahoo Entertainment - March 12th, 2024 [March 12th, 2024]
- FRIESEN: Poker game pays off, as Jets join shootout in the 'Wild West' - Winnipeg Sun - March 12th, 2024 [March 12th, 2024]
- Sugen Singh Takes The Trophy in the 2024 Sunbet Poker Tour Time Square Main Event - PokerNews.com - March 12th, 2024 [March 12th, 2024]
- High Stakes Legend Tom Dwan Signs with ACR Poker As ACR Pro - PR Newswire - March 12th, 2024 [March 12th, 2024]
- DeSantis gave the Seminole Tribe exclusive rights to run online sports bettings in Florida. Now the states racetracks and poker rooms are fighting... - March 12th, 2024 [March 12th, 2024]
- Theres another run at opening Dallas poker houses - The Dallas Morning News - March 12th, 2024 [March 12th, 2024]
- Amidst Accusations, Tom Dwan Named Ambassador of Unregulated Poker Site - PokerNews.com - March 12th, 2024 [March 12th, 2024]
- Poker Returns to Potawatomi in Milwaukee on May 3 With Brand-New Room - PokerNews.com - March 12th, 2024 [March 12th, 2024]
- PokerStars Players in India Moved to Junglee Poker Platform - PokerNews.com - March 12th, 2024 [March 12th, 2024]
- Triton Poker: Holz Cracks Aces for Fourth Title; Rokita Comes Out on Top in Rollercoaster Heads-Up - PokerNews.com - March 12th, 2024 [March 12th, 2024]
- Poker roguelike sensation Balatro sells more than 500000 copies in just 10 days - PC Gamer - March 12th, 2024 [March 12th, 2024]
- News: Tomas Geleziunas goes back-to-back in the Dublin Poker Festival - pokerstrategy.com - March 12th, 2024 [March 12th, 2024]
- Top 10 Richest Poker Players Ever - Net Worth Rankings 2024 - Techopedia - March 12th, 2024 [March 12th, 2024]
- Your Questions About the Pennsylvania Online Poker Bill, Answered - Pokerfuse - March 12th, 2024 [March 12th, 2024]
- Virtue Poker (VPP) Do the Risks Outweigh the Rewards Monday? - InvestorsObserver - March 12th, 2024 [March 12th, 2024]
- Indie Poker-Like Game 'Balatro' Has Sold More Than 500,000 Copies in Two Weeks - Bloomberg - March 12th, 2024 [March 12th, 2024]
- This Week in Poker Podcasts for March 11, 2024 - Pokerfuse - March 12th, 2024 [March 12th, 2024]