Nobody Knows If Your Game Will Pop Off

push to talk #27 // on how to tell whether you're cooking or cooked

Game devs will look at this chart and just be like “hell yeah.”

Recently I've started telling people that you can predict whether or not a game is gonna bang based solely on the YouTube comments it gets.

I'm only half kidding. YouTube is the only social media site that almost everybody uses. 71% of teens in the US hit the ‘Tube at least daily,¹ and its 2.5 billion monthly active users² come from just about every country with widespread internet access.

YT’s demographics are much more representative of the general population than, say, any given subreddit, where the comments section is often representative of a weirdly specific, niche, and probably psychopathic subculture.

So when you see a ton (like, thousands) of YouTube commenters really hyping your game up, there's a decent chance you're making something with broad appeal. And if you find yourself getting cooked in the YT comments section³ you can probably expect the same negative reaction on other platforms. 

But, look, I’ll admit it: the "read the YT comments" method of video game fortunetelling has its limits. It doesn't tell you how many copies you're likely to sell, or what price point would make sense, or what regions or platforms you should prioritize. 

To do all that, you need to bring in the professionals.

How the Games Industry (And Maybe Everybody) Forecasts Sales

It is a truth known far and wide that if you want to forecast sales for your game, there is only one tried-and-true method: looking at how similar games performed and then plugging made-up numbers into a custom excel spreadsheet.

Some have called this beautiful, bold method "building a forecasting model:"

The wielders of the spreadsheets all say that their methods are unimpeachable, and only the strongest pressures can lead their glorious models to spit out inaccurate results. As Epic Games PM Troy Ferrio put it: 

Jordan Checkman of Riot Games confirms this method is advanced and proprietary and don't you dare call it a spreadsheet: 

Owlchemy Labs CEOwl⁴ Andrew Eiche says it's what Jerome Powell would do: 

And Odyssey Interactive tech person Paul Chamberlain says if you have any misgivings about the spreadsheet method, you probably just aren't spreadsheeting hard enough:

Alright, enough goofing around. 

If it's not obvious—three things are happening here:

1. Everyone in games actually does use a custom spreadsheet formula and rough sales data from competitors to forecast their own sales.

2. The people who do this forecasting work for major games studios have a sense of humor about the fact that these formulas are kinda BS, but also…

3. There’s not really a better way to do it. Seriously. What other option is there?

Okay—but what actually goes into these spreadsheets? It's not totally made-up numbers, right?

And, okay, fine. Not exactly.

What Actually Goes In the Spreadsheets

The traditional way to approach a sales projection for a game is to start by thinking about potential market size—in dollars spent by players—for your chosen genre. That’s your TAM (Total Addressable Market).

One way is to benchmark against competitors. In its simplest form, you might look at the worst-performing competitors in your genre, and mark them down as the "lower bound." Then peep the best-performing games, and mark that as "upper bound." Add together, divide by 2, and boom, there's your middle-of-the-road outcome.

But what if that midcase doesn't look so good? No problem. That’s when you start making a case for how you'll outperform the market leader in that genre.  

If they got 3 million sales, we can expect 4 million sales, because we'll be on one additional platform and we beat the current market leader across several key areas like graphical fidelity, IP appeal, additional game systems, and on and on. 

This is basically how a lot of companies in the games industry make projections. It's how pitch decks for publishers and VC firms get made. 

And, look, of course I'm oversimplifying. There are great analysts out there who build out large, complicated models capable of intaking massive amounts of data and spitting out probabilistically-distributed revenue projections based on a dozen factors other than just "what the competitors did." You can pull in industry data from market research firms and other places. But there's a lot of number-fudging, guesswork, and hand-waving even in those models.

In a lot of cases, the purpose of the projection isn’t even to predict sales, but to simply say “here’s how many sales we would need to get in order to justify what we’re spending.” Almost any projection is made because you need to convince someone to give you money.

As a friend put it: “It’s not that we put bullshit numbers into the spreadsheet. The spreadsheet generates the bullshit numbers we use elsewhere.”

But—all that aside—the spreadsheet approach to sales projections can be surprisingly accurate depending on how good your data is and how close your game is to the comps you're referencing. 

For example, if you're making a sequel to an existing game—especially if it's a game in a series that comes out regularly—you have a really solid baseline of sales data to work with. This is part of the appeal of Activision's approach to CoD, or the annual sports game model used by EA. By releasing Madden and FIFA every year, EA is able to put together very strong projections for how their game will sell from one year to the next. 

If, on the other hand, there was a six year gap between your previous game’s release and its sequel, a lot can happen in between to change the market conditions and thus the predictability of your release. Releasing annually generates more data points and signal to work with. If there's a downward trend in the market, you'll spot the direction the curve is moving in year after year. 

Even better than this from a business predictability perspective is live service. Instead of putting out a new game once a year, you can put out a new patch and a few optional cosmetic skins every two weeks. Boom. Now a single year's worth of sales generates unbelievably predictable results, because you have so many more recurring data points to work with. This is how you get a legit sales forecast.

Is Projecting Sales the Same As Predicting Hits?

But let's back up for a second. We started out by asking how to tell whether your game is gonna pop off, and ended up talking about ways to project sales using spreadsheets. Those aren't necessarily the same thing.

If you work at a games studio that's big enough to have a spreadsheet wizard on staff, I have an experiment for you to try. Go up to them and strike up a conversation about some hot new game that's been announced recently—probably one that’s at least had a gameplay trailer out—and ask the Excel Expert whether they think that game is gonna be a hit. 

To answer your question, 99 times out of 100, they are not going to pull up their fancy spreadsheet and start filling in numbers.

Instead, they'll start talking about the pedigree of the team behind the game, or some gut-feel read about the appeal of the game's art style, or even referencing some memes they've seen online about it. They’re unlikely to mention sales projections.

You'll hear "they're cooking—that game looks so good" or "they're cooked, there's no shot they're stealing players away from [X competitor]."

If you want a hit game, some will even tell you, you gotta “just make a good game.” But wait. How could good game fit into a spreadsheet model?

My friend Omeed Rameshni refuses to pay for Photoshop, so instead he called me at midnight last night after reading an early draft of this article to verbally describe the above meme.

Projecting sales is a science. Predicting hits is an art. 

So let's zoom out, and ask a bigger question: How can you predict whether any given thing—whether a video game or something else—will be a hit? 

Two Ways to Predict a Hit

I want to talk about two ideas people have proposed for predicting “hits” outside of a gaming context:

1. An academic’s theory of career success for scientists, inventors, and artists

2. An investor’s theory about the qualities that are most likely to make startup founders succeed

Idea #1: Make A Lot of Things (The Equal-Odds Rule)

The first “hit prediction” technique worth discussing is that proposed by UC-Davis emeritus professor Dean Simonton: The "equal-odds rule.”⁵

Simonton proposed the equal-odds rule in an academic paper in 1997.⁶ In it, he argues that creativity is like genetic evolution: specifically the twin forces of blind variation and selective retention that underlie the process. Lots of people try a bunch of creative stuff, and some of it sticks, and the creator (and others) build on the stuff that sticks.

Crucially, Simonton doesn’t believe that any one creative work has any better chance of being a hit than another. And the way he arrives at this conclusion is by demonstrating (pretty convincingly, with a lot of data to back it up) that, generally, the creatives who have the most hits are also those who produced the most works.

Quoting Simonton’s ‘97 paper:

"[The equal-odds rule] says that the relationship between the number of hits and the total number of works produced in a given time period is positive, linear, stochastic, and stable."

Put in plain English: do more stuff, make more hits. The only way to reliably generate hits is to make lots of things, many of which will not be hits.

For a game developer, the takeaway from the equal-odds rule is as straightforward as it is depressing: there is no way to predict the success or failure of any given creative work. You just gotta keep trying stuff.

On one level, this idea is appealing to me. Two of the first game developers I ever interviewed for Push to Talk were Lethal Company creator Zeekerss and Strange Scaffold’s Xalavier Nelson Jr. Both of these creatives produced over 10 games before publishing their best-known hits,⁷ and I made the case in my articles about them that their relentless approach to game development was the main factor in their success. Even if you reject the idea of each work having equal odds of becoming a hit, it’s undeniable that the experience gained from each shipped game informs and levels up future work.

There’s also a ton of cases—particularly with fiction writers—where an author who churned out dozens upon dozens of novels wound up producing a few bangers alongside a pretty good number of duds. I’m thinking about Asimov, Philip K. Dick, and Faulkner here.⁸

But on another level, there’s something profoundly unsatisfying about the idea of the equal-odds rule. Some artists are just better, aren’t they? Harper Lee was a one-and-done who wrote one of the most successful American novels ever. Jane Austen dropped six books—three of which are still read widely today—then died at age 41. Fyodor Dostoevsky wrote 16 novels, and easily 6 of those were among the greatest ever written. My guy was batting a .375 on GOATed novels. You’re telling me he had “equal odds” on each of those? I don’t really buy it.

But then I remember—the equal-odds rule isn’t about the quality of any given work. It’s about hits, meaning works that becomes extremely popular and successful. By that standard, could it be true?

I’m still not totally convinced. Some part of me has to believe that some people simply have more of what it takes to produce a hit. Maybe intelligence? Or good-old-fashioned raw talent?

Y Combinator co-founder Paul Graham has a different idea.

Idea #2: Determination

Paul Graham is an outrageously prolific investor who is perhaps most famous for posting long, sometimes pretty incredible essays on his personal website (which has inexplicably had the same visual design for 20 years).

One essay published way back in 2009 presented Graham’s theory about the personal quality that causes some startup founders to be more likely to produce a hit company than others:

Like all investors, we spend a lot of time trying to learn how to predict which startups will succeed. We probably spend more time thinking about it than most, because we invest the earliest. Prediction is usually all we have to rely on.

We learned quickly that the most important predictor of success is determination. At first we thought it might be intelligence. Everyone likes to believe that's what makes startups succeed. It makes a better story that a company won because its founders were so smart. The PR people and reporters who spread such stories probably believe them themselves. But while it certainly helps to be smart, it's not the deciding factor. There are plenty of people as smart as Bill Gates who achieve nothing.

In most domains, talent is overrated compared to determination—partly because it makes a better story, partly because it gives onlookers an excuse for being lazy, and partly because after a while determination starts to look like talent.

—Paul Graham, The Anatomy of Determination

Later in the essay, Graham goes on to try to dissect the different “components” of determination—and ultimately lands on a definition of determination as a combination of willfulness, discipline, and ambition.

But, wait a minute. Isn’t that sort of the same thing as saying “stick with it for a really long time and produce lots of work.” My god! We just looped back around to the equal-odds rule.

Or did we?

In an essay called What We Look For In Founders, published just a year later, Graham listed five qualities that he thought successful founders tended to embody. The first of these was, again, determination, which isn’t surprising, but I was kind of amazed that Graham again left out talent and intelligence from the list. (The remaining four qualities that Graham says he looks for in founders are flexibility, imagination, naughtiness,⁹ and “friendship” aka a strong co-founder.)

It’s a short essay, so you can just read it in full to get the gist, but the most remarkable thing about the essay is what later happened to the young founders Graham mentions as exemplars of what to look for in founders in the piece. All of them went on to become outrageously successful: you’ve got the WePay guys (sold to JPMorgan Chase for $400 million in 2017), the AirBnB guys (current valuation: $96.95 billion), the founders of Justin.TV (which became Twitch, sold to Amazon for $970 million in 2014), and two guys who later founded AI companies. I’ll let you Google them.

This suggests that Graham himself possesses some talent for spotting “hit” startup founders. That raises a natural question: what is that quality called? I mean: if determination is the quality most predictive of success for a startup founder, what do you call Graham’s uncanny ability to spot startup founders who are likely to create companies worth hundreds of millions (or even billions) of dollars?

It could be the equal-odds rule. Y Combinator has invested in over 5,000 companies,¹⁰ so that jives with Simonton’s theory about “hits” scaling with raw output, including lots of flops. Maybe Graham would call this quality “determination.” Or maybe it’s one of the other four qualities he listed. But neither of these things explains the fact that in this particular blog entry from 2010 Graham specifically called out like eight dudes as having the qualities of successful startup founders, all of whom only later went on to become giga-millionaires. In this one blog post at least, Graham batted a thousand. A 100% hit rate. How do you explain that?

Is this a skill that can be learned and applied to other things? Or are some people just naturally, freakishly good at certain types of predictions?¹¹

It’s possible, maybe, that some people possess the ability to predict success—to know for sure when a thing is gonna pop off.¹²

As for the rest of us, the best we can hope for is a really good spreadsheet.

Unless… there might be a third way.

The Third Way: Doing Stuff to Generate Signal

I’ll leave you with one more idea—really more of a shadow of a concept—which takes for granted the insight from Simonton and Graham that just doing more things gives you more chances to win.

Experience is one of the most obvious upsides of just doing things. Doing a thing a lot will (probably) make you better at that thing.

But there’s another upside, particularly for doing things in public: Anything you do in public generates signal. That is, you put stuff out in the world, and people react to it in ways that unmistakably reveal new information about the market itself and the appeal of your product for that market.

This is an idea I’ve been grappling with for a while. It’s why I’m only half-kidding when I say that you can tell whether a game is gonna pop off based on the YouTube comments. I first took at shot at talking about it in my post about games marketing on easy mode and hard mode:

Sometimes, for whatever reason, a game possesses qualities that make it bang when marketed. The trailers go viral. Content creators play it without being paid. Players come begging to join playtests. That's marketing easy mode. 

Early reactions from players are, in fact, market signal, and you should do whatever you can to get good market signal before you put your game out. Playtests, surveys, trailers—all of these things give you opportunities to learn how players are reacting. And if you’re getting anything less than an enthusiastic response, the thing you’re making is probably not going to pop off.

It’s so tempting to dismiss these signals as noise. You get 70% of playtesters leaving lukewarm reviews, and you tell yourself “Oh, they’re just not the target audience.”

Or, worse, you misread tepid cues as strong positive signal. A big content creator who came to your private playtest tells you “this game rocks!” and you take them at face value—even though they’re just being polite, and they’re not gonna play your game on your stream. Not unless it grows their audience.

The hardest thing about reading these signals is how resistant they are to being weighted, quantified, digitized, or squashed down until they’re small enough to fit into a spreadsheet formula. If you host a private playtest for your game that’s scheduled to run from 5pm to 7pm, and almost everyone is still playing and refusing to leave at 2am, what does that mean?

Signal doesn’t just come from outside. Sometimes the call comes from inside the house.

I’ve heard game developers from at least three major studios describe the moment they first realized their game was really good. And each time, the story went something like this:

We’d been forcing the team to stick around for team-wide playtests at 4pm every day, and all of a sudden we found that we didn’t have to ask people to show up or stick around anymore. People kept playing even after the playtest ended. Soon, we were playing our own game more than we were playing anybody else’s, simply because we were having so much fun.

Nobody knows if your game will pop off. But you know—if you’re being honest with yourself—whether or not you’re hitting that bar.

---

That’s it for this week. I’m gonna go make important investment decisions by reading YouTube comments.

See you next Friday.

1. Source on teen YT usage: https://www.emarketer.com/content/teens-use-youtube-on-daily-basis-more-than-tiktok

2. Global YT usage data: https://backlinko.com/youtube-users

3. No disrespect intended toward these devs—the backlash here isn’t just about the game itself imho. Would love to chat from someone from that dev team if you’re reading.

4. That’s his real job title btw: https://www.linkedin.com/in/andreweiche/

5. I found out about Simonton’s work via this excellent Substack post.

6. Read that paper here: https://www.infocenters.co.il/lesley/multimedia/14629.pdf

7. In the case of Strange Scaffold, it looks like the upcoming I Am Your Beast—which was one of the most most-played demos of the most recent Steam Next Fest, is set to become a monster seller.

8. I know I’m gonna get yelled at from three different fanbases for saying that about these guys, but it’s true.

9. It’s insane that the following was written in 2010, given what we know now. From the essay:

Naughtiness

Though the most successful founders are usually good people, they tend to have a piratical gleam in their eye. They're not Goody Two-Shoes type good. Morally, they care about getting the big questions right, but not about observing proprieties. That's why I'd use the word naughty rather than evil. They delight in breaking rules, but not rules that matter. This quality may be redundant though; it may be implied by imagination.

Sam Altman of Loopt is one of the most successful alumni, so we asked him what question we could put on the Y Combinator application that would help us discover more people like him. He said to ask about a time when they'd hacked something to their advantage—hacked in the sense of beating the system, not breaking into computers. It has become one of the questions we pay most attention to when judging applications.

—Paul Graham, What We Look For In Founders

10. Source on that figure is YC’s official site: https://www.ycombinator.com/companies

11. See the bizarre phenomenon of superforecasters, which appears to be the scientific community’s term for real-life human oracles: https://en.wikipedia.org/wiki/Superforecaster

12. If you work on video games and you have this power please contact me immediately lol