11 Commandments for Online Multiplayer and F2P

1. "Online" is no longer a feature, it's your foundation.
Historically the AAA development process has evolved from a point in time where online multiplayer was actually an optional bullet point on the back of the box. Back then a successful game could ship with or without online multiplayer. It is in this environment that the current FPP process was built, and why the focus to demonstrate a vertical slice of gameplay is more or less faked. With this mindset, it is acceptable for "online" to be eluded to and then added later during production. These days--due to resources limitations, dependencies, and risk--the online development is more often than not outsourced as a feature to another studio.

One of the earliest and most extreme examples of this was the development of Ghost Recon Advanced Warfighter (GRAW). In 2003 Tiwak was working on the brand new engine (Yeti) that the GRAW single player campaign was being built upon. However, due to the various aforementioned reasons, a decision was made to ensure that GRAW would ship within the expected holiday window. Following in the footsteps of Splinter Cell: Pandora Tomorrow, where Shanghai was successfully outsourced to create the multiplayer, it was decided that the GRAW multiplayer would be produced by Red Storm Entertainment (RSE). However, unlike the Unreal-based Splinter Cell, the Yeti engine was a developing technology, and for all parties involved it made sense to reduce risk even further by leveraging the seasoned GR2 engine for its well-tested lighthouse network library and successful multiplayer. This decision in turn required RSE to leverage a studio worth of resources to literally recreate their own version of GRAW in a different engine. The benefit of this arrangement doubled the team’s size and yielded a high quality hit game within a timely manner--as well as reduced each studio’s operational risk by splitting the development cost. Unsurprisingly, this practice not only continues today, but has been institutionalize with more than a decade of Ubi games treating online as a feature with great success.

There are, of course, downsides to this process especially pertaining to product vision and communication challenges. GRAW like any other project probably had its own unique share of challenges due to co-development, but it also had a few seemingly minor consequences that are of significantly more importance to acknowledge today. The first is that Yeti was never built with network replication in mind--which unknowingly created significant challenges, limitations, and risk to shipping Ghost Recon Online on the same engine years later. The second is that the GRAW cover system, one of its biggest breakthrough features at the time and a competitively unique selling point, was nowhere to be found in the multiplayer. It is a logical symptom of the split development process, but also probably the best "worst case" example of how treating online as a feature can literally yield two entirely different games.

Today, demands on the next generation of online games are changing. The online component can no longer be treated as a separate feature from the core when single and multiplayer modes are now touted as a seamless and cohesive experience.  Outsourcing to another studio becomes notably problematic as considerations for the technical and design differences need to be closely monitored and coordinated. Next generation projects that revolve around the mandate to create this kind of seamless online/offline gameplay while still following traditional black box style development are more likely to render the developers ineffective—in the worst case resulting in re-scoping years of work, cutting dependencies, and even rebooting entire projects.

It's of importance to recognize that upcoming projects are being driven as the result of forward-looking next-generation global initiatives—requiring them to be designed around buzzword features such as fully online, systemic gameplay, data driven analytics, micro transactions, and companion gaming to name a few.  As the industry evolves, institutionalized processes focusing on core story and gameplay that lead to online being segregated as another feature to develop need to evolve with it.

2. Think long term, because it's a self-fulfilling prophecy.
Short term thinking leads to long term operational losses, and F2P/online projects are long term in their very nature.  Projects of this kind need to be built smart to reduce operational overhead, adapt to changing markets, and have the flexibility to support unforeseen monetary and strategic channels. As such, it should be entrenched that F2P online projects have an indefinite life span to prevent short term thinking. Mandating a concept like this from above helps change policies and procedures at all levels from HQ to QA. 

3. A live service environment is your platform, so treat it that way. 
Anyone would recognize the silliness of developing a game on PC and then gold mastering to GameBoy. On day one you should strive to be operating in a live service environment.

4. Ignoring technical debt bites you in the ass forever.
Remember your project has an indefinite lifespan and the earlier you address the debt the more you save. If you believe in Commandment #2, there is always time to do it right regardless of who might say otherwise.

5. Your monthly development cost is indicative of your monthly operating cost.
If it takes a lot people to make your game it's likely to require even more to operate it. Remember Commandment #3. Be careful of when and where you scale development as you are actually scaling your operating costs in the process.

6. Numbers are only as meaningful as the questions you asked.
These days where data can be tracked and analyzed on just about anything, the real danger is the possibility for it to lead to a blind unquestionable trust in numbers and their KPI's once established. New questions need to be asked in order to constantly test and probe various hypothesis. A KPI cannot explain the "why".

7. If you don't know how much it costs to make your virtual item then you are doing it wrong.
Your content pipeline is a real world manufacturing plant from beginning to end. If it takes 1000 man hours at an average cost of $20/hour to make a virtual item, then you need to sell $20k worth before you start making a profit. Free to play games literally make their bread and butter off this process and there is always room to reduce its overhead. Doing so allows the bread and butter to more efficiently support other operations that don't have a clear ROI.

8. Elite military soldiers don't buy mystery boxes full of epic loot.  
You are not making a store, you are designing a user experience. Support the IP and fantasy of the brand from beginning to end.

9. Players don't stop playing your game, they divorce it.
Bad experiences are more memorable than the good ones. Many small bad experiences can usurp the great ones and cause unidentifiable retention issues. It costs less to acquire new users than it does to get them back, so don't lose them. Note that happy players are your best acquisition tool and prioritize keeping players happy to benefit long term conversion.

10. If first time conversion yields buyers remorse, assume they will not convert again.
If people feel good about buying they will buy again. Don't mechanically force people to buy something they don't want.

11. At least people don't die from our mistakes.
We should make mistakes so we can learn and never punish people for making them. Besides, tomorrows patch will have the fix.

Troubleshooting Poor Framerate

The process and steps to solve performance issues are highly context dependent and requires deep and broad knowledge of the engine tech and tools. Having knowledge of the project at all levels of development helps to isolate and solve all sorts of weird performance-causing issues that would have been hard to connect otherwise.

A typical process tends to have these steps:

• Theorize – Review possible causes based on contextual knowledge of the project as a whole. 
• Establish Repro Case – Check if it is reproducible on another machine and by another human. If necessary create isolated test cases based on possible theories as needed.
• Narrow Down the Possibilities – Validate the nature of the frame rate issue by using the debug tools available (ie. an FPS monitor, draw call data, etc.) to do a sweep of the level and identify if the issues is related to tech, art or design changes. If its art or design related, check file history.  Assess whether new tools needed to identify the causes.
• Compare and Contrast – What is different with this map/build/script/code?  What has change?  Establish whether this problem has always existed, and roll back to an alternate build.
• Isolate Probable Causes – Turn off various features and subsystems to isolate and identify the cause of the problem (ie. vegetation, AI, scripts, physics, sound etc.).
• Resolution – Once the cause is established and understood, communicate the nature of these issues as well as any new constraints that need to be defined to prevent future problems to relevant members of the team. As much as possible try to make sure the solution addresses the cause and not the symptoms. Short term fixes always bite you in the ass later. 

Usually each problem gets isolated and diagnosed in a pretty unique manner based on the engine, project and tools that might or might initially not be available, and of course the current stage of project development. Here are a few brief examples of a variety of issues and the various solutions used to resolve them:

• Tech / engine / tools pipeline related level performance issues:
  
  In a previous project that contained a destruction system, as you progressed and destroyed buildings through any of the levels, the performance deteriorated even if you were looking at nothing. The initial theory pointed to the physics system where it was speculated that objects were not properly coming to rest.
  
  I ended up creating several test cases using the engine debugger as well as external physics debuggers to help isolate the cause. One test case was simply a single building before and after a destruction performance check. I was able to deduce the primary slowdown was caused by a performance optimization which collapsed all of the destruction debris into a single draw call. Once the physics objects came to rest they were being “hidden” from the scene rather than completely cleared. This also had the side effect of drawing an entire building even if only a single piece of debris was in the camera frustum. Imagine debris scattering all over a level then hidden resulting in every object in the game being drawn all the time.
  
  In addition, there were issues of physics objects not coming to rest due a several reasons ranged from debris penetrating the world and falling forever to more process issue steaming from human error due to a poor authoring process, and a lack of ownership for the material system.  Having artists hand tag materials with complex physics variables was not the best way to get results.I ended up working with the engineers and artists to redesign and implement the entire physics system and content authoring process. We changed the destruction system to spawn instance objects instead of custom unique debris, which in turn lead to changes in how art and design authored the levels and assets with instance geometry but kept the unique look when walls were destroyed. This also led to changes in how the material system was architected and data was managed by separating physics effects and gameplay materials. Global materials were also defined to cover 90% of the possible cases from a centralized location, and we added a new default/undefined physics material that was easy to use for debugging purposes.
  
  
• Various art and design related issues:
  
  
• Overzealous on use of physics / destructible objects – This caused problems when too many of the 3000 poly debris spawned. Worked with artists on making sure we spawned less stuff and lowered the poly count. Devised a “rule of mass conservation” for destruction art where all of the mass should be accounted for, but you can make a lot it dust and smaller particles. Also added vaporization parameter for designers to control the amount of debris released. 
• Overuse of post screen processing effects that blew the draw call limits – This problem only revealed itself when finalized art with all of the different materials came into the scene that in conjunction with the post processing caused the total draw call count to sky rocket.  This was resolved by redesigning the post processing effects that required less rendering passes. 
• Too many textures – Reduced the variety of AI archetypes being used at the same time that blew the texture budget causing the system to page to disk and load the new textures.
• Too many ray casts – Caused by a combination of too many shotgun-wielding AI all firing at the same time. A little tricky to isolate as the types of AI and weapon load outs that spawn were random. Reduced the number of shotgun AI that could spawn and the number of ray casts each shotgun fired. 
• In addition to the cases described above, I have seen a variety of other causes for frame rate issues such as overblown draw call count, unnecessarily high poly count, improperly placed occlusion planes, poor use of BSP portals, bad level flow for a sectorization system, overloaded particle systems, overly detailed sound volumes, fill rate issues due to large transparent particles, excessive use of dynamic lights, sounds being played that couldn’t be heard, etc. 

Linear vs. Open World

Linear	Open World
Creating the vision (+) Generally a simpler planning process for the end-to-end experience that can be storyboarded, planned, communicated and pitched to executives and team before production begins. Achieving team buy-in and agreement on what you are going to make can make the production process a lot smoother. (-) While it makes the production process smoother, it can certainly be more monotonous and devoid of creative decisions later on.	Creating the vision (-) A lot harder to communicate non-linear structures since it is more about putting simple rules into place that when put altogether create unpredictable experiences depending on how the player is interacting with them.  It takes a lot more time to prototype and to achieve buy-in for the game vision. The pressure from the team and execs to prove the fun increases exponentially the longer this process takes. (+) Once the core game systems and architecture are in place, often production experiences a hyper rapid improvement on the quality of the fun factor due to the ability to adapt and iterate.
Core gameplay and validation (+) Easier to provide a high quality and consistent delivery of story, action and pacing. (-) Control can be a disadvantage as it is a lot harder to create and contain a player in a scripted environment without breaking the suspension of disbelief. Scripting can become overly complicated very quickly, leading to odd cases that you hadn’t considered (the more you try to control the player, the more events are likely to break). (+) Testing gameplay is more complete as it’s more about the minute-to-minute play-through and functionality. The ability to focus on one path makes it easier to validate the fun factor and design goals from beginning to end. (-) As the amount of gameplay hours offered increases, so does the amount of time for it to be validated.	Core gameplay and validation (-) Harder to write and create a cohesive narrative experience when player choice has a huge impact on the order of the content they receive. Often auxiliary systems need to be designed and put in place to achieve the desired behavior. Good pacing can be achieved, but with the additional challenge of player data collection and creating global managers.  (+) Generally more interesting to make from a system design perspective as the game features have more depth and breadth in the game mechanics and their interactions. Fun problems to solve and challenges to overcome.  (-) Open world games often provide so much gameplay that it becomes impractical or impossible to test every minute of the experience. (+) Since systems are designed more as rules to be interacted with, there is less minute-to-minute gameplay to validate, but instead simple global test cases that validate functionality and spot breakages.
Production (-) While planning offers a certain level of production security, when things aren’t working or are not fun, often the underlying systems and code are too rigid to change. Effectively painting the designer into a corner and making iterating costly or impractical. (+) Features and levels can be created with minimal dependencies that can be added or subtracted. If an idea or feature doesn’t work, it can be cut or replaced without compromising the project or deadlines. It also has a production advantage of being able to scale the team up/down with cheaper less experienced labor as needed to work on modules in parallel or even outsourced to another studio. (-) While it’s easier to outsource larger sections of the game, this also requires more management overhead for external communications, maintaining quality and integration of the work.	Production (+) Good systems and architected dependencies can make the data management for tuning and balancing the rules of the game world easier allowing improved iteration time and ultimately a more polished experience with a smaller design team. An additional benefit is the flexibility to change as the design requirements and scope evolve over the course of development. (+)/(-) The quality of the core game requires key highly experienced and technical leadership in order to make sure the systems are properly architected. Often if a key vision keeper or talent leaves in the middle of the project it can have a massive impact on the production time and vision. (-) Often impossible using traditional content production methods to create enough unique content that matches the amount gameplay hours that can be expected.
Player experience (+) When a goal is provided it’s easier for the player to understand and strive for it. Getting lost or falling off course is a lot harder due to the lack of options. In addition, minimalistic UI and game feedback lead to a more immersive and less gamey experience. (-) Linear experience often can alienate players that enjoy exploration and discovery. (-) Lack of choice and decisions made by the player can feel artificial and contrived as they are forced down a path that they personally wouldn’t take. (+) Less filler content required allows the development team to focus on more unique memorable moments where the player experiences significantly less repetitive gameplay. (-) Difficult to maintain an immersive experience as the player probes the boundaries of the space, the façade is easily revealed as a movie set.  (+)/(-) Linear games do not offer significant opportunity for replay-ability.  Once a player has completed the story, the motivation to return to the game is lost entirely.	Player experience (+)/(-) The abundance of side content that can be interacted with can distract a player from the core story experience. This distraction may be welcomed by the player, but may also have the side effect of making them lose their way if not properly managed. (+) An open world offers a massive playground to explore, giving the player control and ownership over their experiences and solutions to challenges. (+) Players have more decision making moments that not only define their character, but also impact the living world around them. (-) Unfortunately most open world games have levels that are inherently hard to navigate due to highly reused art and content, requiring the use of abundant UI and gamey navigational aids. (+) Because the rules of the world are not rigid, players often discover and enjoy unintended, emerging gameplay that were not intentionally designed. (+) Players can enjoy new adventures and their own unique stories within the same game, even after completing the story.

Containing Players in an Open World Game

The real interesting solutions to these problems lie within the specific contexts of the game, the behavior change you are looking for in the player, and their emotional response.  Here are some high-level methods with context-based examples for encouraging players to stay within a certain area of an open-world game:

Audio-visual feedback cues – Reinforce desired behaviors through music, sounds, loss of visibility due to environmental changes, directional arrows, etc.

EXAMPLE: In a zombie game during a day/night cycle, the simple use of light and dark can be used to effectively control player movement within a larger area. Additional cues can be utilized to invoke feelings of fear or horror by introducing sound cues outside of the light zones to indicate impending danger resulting in death.

Progression / difficulty – Creating a core progression system where a player cannot physically handle the difficulty beyond a certain point to encourage the player to stay in a zone until they progress

EXAMPLE: A level-based shooter where experience is earned and used to obtain progressively better equipment in order to overcome the challenges preventing the player from leaving the space.

Limitation in resources – A game system or temporary mechanic that physically limits the player’s range (eg. endurance, oxygen, fuel, ammo, etc.)

EXAMPLE: An exotic gameplay scenario that requires a player to hold their breath in a smoke-filled environment, physically limiting the range they can travel.  Combined with the right set of audio-visual feedback cues can effectively create a sense of anxiety and stress.  As a designer, you would obviously want to keep this tactic in limited durations so as not to cause player fatigue.

Breadcrumbs and carrots – Feedback cues that steer the player to the desired destination area

EXAMPLE: In a car combat game, a player may need to kill an objective in a certain area.  Players enjoy intense and compelling combat when they are inside the zone, and experience lulls when they deviate outside of it. Whenever the player leaves the area, new combat targets (ie. carrots) are spawned traveling towards the core objective to lead the player back into the action.

Anchoring / tethering – A specific location that missions/objectives require you to visit to either progress or get rewards that help you progress

EXAMPLE: Imagine a game where the player takes the role of a cop and is assigned to a specific HQ where he receives all of his dispatches and gear required to venture in the area.  Whenever it is time for the player to move on, he gets transferred to a new department or HQ.  This context makes sense, feels logical and maintains a suspension of disbelief for this kind of tethering model.

Attachment / investment – The player is given or earns a location that he can invest in. This doesn’t have to be stationary; it could be mobile if the context fits.

EXAMPLE: Imagine a game where the player can choose to build and upgrade a fort or lair with gameplay benefits as well as personalized cosmetic features that can be used for bragging rights.  This is a slight deviation from the anchoring/tethering mechanic where the player is forging an attachment to the anchor via the investment of time, emotionally through personal expression, or to use it as an edge in gameplay. A mobile example could be investing in a large ship that travels from island to island.