Anyone who's worked in game development for a while knows that launching games is challenging. Creating online games is even more difficult, as it requires building Internet services like matchmaking, account systems, chat services and many more. Guild Wars, an online Role-Playing Game I helped develop at ArenaNet, requires ~30 backend services, and its sequel, Guild Wars 2, has ~80! By the time I left ArenaNet Guild Wars 1 included 6.5 million lines of code, which is about one-fourth of the number of lines in the Linux operating system kernel!

In addition to the programming involved, it’s necessary to incorporate game content that includes tens of thousands of design, art, animation, sound, and dialogue assets, which means games are astoundingly large endeavors. Consequently, it should be no surprise that game teams fail during development on a regular basis -- somewhere around 50% of products fail before they even launch!

Post-launch, players have high expectations for new games because they have so many alternatives to choose from, with over 9000 games released on Steam in 2018, and over 750 games released per day -- per day! -- on mobile as far back as 2015. Not to diminish the amazing contributions of previous game developers, but a twelve-year old kid can build a game similar to Asteroids in an afternoon using the Scratch programming language, so the bar for making commercially viable games is waaaaay up there.

If our goal is to make a top game, then not only do we need to overcome the development risks to get to launch, and make something gamers love too!

So how can we create the complex technology required for a major online game and marry it to the great game content players want, while avoiding the risks associated with technical complexity and product-market fit? And how can we do it reliably and repeatedly while making games that are not clones or sequels, because big game publishers have locked up that market except for projects with the largest budgets.

We want our development methodology to maximize our ability to efficiently create games while minimizing costs until we know players will love the result. We need to avoid common patterns of failure that sabotage projects. Briefly:

• Technical debt generated by haste
• Backwards progress caused by milestone-driven development
• Poor team morale caused by crunching
• Launch-date unpredictability caused by bugs
• Post-launch development sluggishness caused by the live-operations burden

Finally, we need to ensure that the method we utilize recognizes that game-development is an inherently artistic process that cannot be refined solely to metrics & measurements.

The approach used in Alpha-Driven Development (“ADD”) is to invite a small number of gamers to play early in development in order to gain their feedback while there is still time to radically alter the core game based on their feedback.

We allow early players time to uncover deep gameplay truths and reach the elder game in order to ensure that their experiences are enjoyable and rewarding, and that they desire to continue playing, forever.

Core tenets of Alpha-Driven Development

• Enable rapid developer iteration at the start of the project, allowing developers to deploy a reliable game to alpha testers many times per day.
• Begin testing early in development with a small group of external players, known as alpha testers, who are representative of the target audience for the game.
• During alpha-testing, prioritize validating the game design over finding bugs
• Grant alpha-testers access to the full experience; do not lock them away in a beautiful corner.
• It should be immediately obvious --everywhere -- that the game is in early development so as to focus attention on gameplay over polish.
• Developers should iteratively build features, frequently deploying partial systems to testers to gain feedback rather than building entire systems at once.
• To avoid tester attrition the development team must endeavor to build reliable systems, and fix (non-cosmetic) bugs before building new features.
• Programmers own the code they develop, and must fix their bugs before building new systems. This leads to new systems being built by programmers who write the most reliable code that requires the least maintenance.
• Crunching is not a reliable way of delivering high-quality products; tired developers make mistakes. Developers should check in features when they’re reliable, not race to add them before play-sessions, which can lead to bugs that destroy the play session.

Technology development priorities

Rapid development iteration

Developer iteration speed is a measure of how fast a developer can build and test on their local computer. It should also include how fast a developer can build, test, and deploy changes to alpha-testers to gain feedback.

Developer iteration speed is the single most important quality to optimize for because it has the greatest impact on team productivity. Since the largest cost in game creation is developer salaries, we want to optimize development activities so as to maximize the value of our developers.

It is essential to begin this process at the inception of development. Optimizing build-speed later in the project lifecycle is less useful as there is less time for benefits to accrue.

What does “good” look like? Guild Wars averaged 20 builds per day deployed to alpha-testers for four years (~17,000 builds). Most of those were “code-builds”, which took 5 minutes from build-inception to player-in-game, including sync, compile, link, deploy, client and server patching, and server activation. Code-builds included code and design changes, but excluded art and level data; “art-builds” took 25 minutes each and were run only a few times per day -- we could have done better there.

Early external alpha-testing to discover core gameplay

Game developers often hate to show their unfinished work to outsiders because they frequently get criticized due to the unpolished nature of the work. End-users and game-publisher staff tend to over-index on how a game looks compared to how it plays, which is unfortunate because great artwork and polish are expensive endeavors that can be deferred until later in development when the core gameplay has been discovered and refined, and we know we have a product players love.

When beginning alpha-testing it’s important to overcome the bias against early testing by starting with a friendly audience and setting the right expectations. Begin alpha-testing by inviting 10-20 friends who best represent the target audience for the game. That is, not just friends, but friends who might like our game.

Set player and developer expectations at the beginning of the project. In all of the external communications, continually reinforce the mantra that this effort is not about finding bugs (though that is a side benefit), but about aiming to discover the best, most fun gameplay experiences. As such, the game is not going to look pretty. And it manifestly should not look pretty. Everywhere a player looks should reinforce the “alpha-ness” of the game, including gray-box artwork, placeholder text, and alpha banners.

If an alpha-tester takes a screenshot and it escapes onto the Internet, it should be clear to viewers that the game is early in development, not a game with low production quality.

Secondly, alpha-testers should have full access to all the game systems, features and levels that developers do so they can get the full experience that’s been created. There should be no gating systems to wall off alpha-testers into the most beautiful corner, as they can only provide feedback on what they see. Don’t wall them off!

Grant alpha-testers the ability to play whenever they want, but encourage them to play at times when the development team can join them, e.g. Wednesday & Friday from 3-5 PM.

Iterative feature development

Developers should iteratively build features, frequently deploying partial systems to testers to gain feedback rather than building entire systems at once. Just as it is less expensive to design on paper than to design in Photoshop or Visual Studio, so it is less expensive to course-correct a partially complete system than a fully-baked one.

For example: when building a friends list in the game, the developer should deploy in phases:

1. A napkin-sketch brainstormed iteratively with several developers.
2. A document reviewed upon by the development team.
3. A forum post shared with alpha-testers for comment.
4. A white box on the screen that shows the size and location.
5. A static bitmap that shows the (unprettified) elements it will contain.
6. A non-interactive UI that shows fake state changes for fake friends.
7. An interactive UI that allows for state changes for fake friends.
8. An interactive UI that actually talks to the friends server.
9. Repeat as necessary to add more advanced features.

Alpha test reliability is important

With two weekly play-sessions over a two-year development cycle, there are only ~400 total sessions, and so we want to avoid losing those sessions because “the build is broken”. Also, alpha-testers who become frustrated because the game doesn’t work will leave, which makes it harder to gauge a game’s potential, increasing our desire for reliability.

Consequently, developers must endeavor to build reliable systems, and fix (non-cosmetic) bugs before building new features so as to avoid wasted play-sessions. Developers should fix breaking or critically-annoying bugs before writing new code so as to maintain a high level of operational reliability.

If there are twenty alpha-test builds per day, and two playtests per week, it reduces the pressure on developers to race to deploy features, which is the most common source of failed playtest sessions. If there’s always another build-train leaving to the station soon it’s less critical to get untested code-changes onto the current train. We desire developers to build partial, but reliable, changes and commit them to master when they will not break. They will still break from time-to-time so an exception-reporting system (e.g. Rollbar.com) is absolutely essential.

Measure alpha-test results

Listen to alpha testers to reduce developer bias. Listen to the alpha testers to gain qualitative feedback, but be sure to guide efforts by player analytics and behavior, not forum posts from the loudest ones. Finally, listen to feedback without being controlled by the feedback -- sometimes people think they want things that turn out not to be so much fun. Many designers and players thought they wanted an auction house in Diablo 3, but it turns out to reduce the value of items earned playing the game.

Strive to find the fun, while avoiding content-creation expenses. With alpha-testing we're able to inexpensively develop an engaging core loop of the game, leaving us time and budget to pivot when necessary, which helps us avoid making mistakes due to cash-constrained decision-making. This process allows us to discover the emergent gameplay that underlies our game design and accommodate it during development. We can discover the elder-game and ensure that it's deep enough to maintain long-term engagement. Our staffing growth and spending can then be based on readiness, not milestones or preconceived assumptions built when the project was first started.

Beta is Launch

Beta-test should be an opportunity to share the game with players to evangelize gameplay; it’s not an opportunity to find bugs or perform load-testing. At the commencement of beta our game should already be highly reliable so that our development team is in position to respond to the needs of the community instead of being unresponsive because the team is correcting defects. As the team has already been operating live in front of players for the duration of the alpha test, high reliability and low bug-count should already be a solved problem. And we've trained to be able to release at a high cadence once we launch, because the launch-release cadence is "only" several times a week instead of twenty times per day.

—

Want to know more about how Alpha-Driven Development works with real players? We wrote a blog post about that.