For my understanding, what ranking are you creating that is different than your points formula?
I had this question as well. If your points system is solid, the existing rankings should be pretty accurate as far as evaluating raw performance — at least, for those players with a large sample size over many games.
First let me say that I do think LLMs (AIs) can do many amazing things, and this type of task may be one of them—as it is data-driven and not terribly subjective. It does not require much if any reflective judgement, mainly just data crunching and pattern recognition. I have found bots like this to be very good at straightforward tasks, for example finding where something went wrong in a website’s code.
If it also can absorb from poker literature what kind of variance to expect in tournaments of the size you are hosting, that would probably be a big plus. It might be worth prompting it to take a look at any available articles or studies of tournament variance and results. (Even extraordinarily talented pros can bust without cashing in a tourney. I remember hearing a podcast a few years ago where a well-known successful pro said he once played 30 tournaments in a row without seeing any profit.)
That said, how much you glean from the results is likely to depend a lot on what types of questions and follow-up questions you ask to make sure it is looking deeply enough into the data, and not generating off-point or simplistic conclusions.
............................
FWIW, about a decade ago I used to handle all the record-keeping for a friend’s two-table tournament which generally had 15-18 players every week. I amassed about three years of data in a spreadsheet on a player pool of about three dozen players, about half of whom attended between half and 90% of games. The rest were one-off guests, or more occasional participants. (We had an end-of-year special event for a bigger prize, where starting stacks were based on performance during the year, which was the purpose of the record-keeping.)
This was long before A.I., but I recall being able to glean a fair amount of insight into the player pool just by looking at and sorting the spreadsheet in various ways. In addition to our points system, I had fields for stuff like number of 1st/2nd/3rd/4th/5th place cashes, number of total cashes, earnings/losses, rebuy frequency, etc., both in total and per appearance. I came up with formulas to estimate stuff like how high people generally placed, normalizing the numbers based on number of players (since something like a 7th place finish in a field of 15 is not quite the same as in a field of 18).This was also interesting in terms of who had the least cashes/earnings per appearance.
Somewhat more intriguing were anomalies like the player who had a lot of top 3 finishes, but who also had a lot of early bust-outs. This of course was a guy who played a high-variance, gambly game. Then there were others who had above average overall rankings, but relatively poor earnings. These were nittier players who tended to go deeper into the game than many of the weaker or more wild players, but who rarely finished 1st or even in the top 3 places, because they did not play aggressively enough in the middle and later stages of the tournaments.
I would think an LLM might be able to suss out more complex insights, such as who appears to be overperforming vs. underperforming; or, whose win rate has the highest long-term quality as opposed to just the biggest numbers. Among those who have fewer data points than your most frequent regs, the LLM might be able to offer insight into who is likely to climb up the standings as they play more often. And so forth.
Long message, I know... Basic point is that if you nudge Chatty to look harder and make finer distinctions beyond just “who is the best player,” you might really get some good insights out of it. But I would remember to double-check and be skeptical of all answers, because the bot does not really know what poker tournaments are and is only going to be as smart as your prompts.
