§ Advances to competitive Pokémon
Shoddy Battle was released by Cathy and bearzly on July 7, 2007 and it ushered in a number of significant advances in the Pokémon community, changing the face of competitive Pokémon in ways that have been continued by subsequent simulators. This page will describe some of Shoddy Battle's contributions to the competitive Pokémon community.
Table of contents:
- Free, open source, and cross-platform software
- Scalability
- Extension and customisation
- Statistical data and user ranking
- Status
You can click on the § symbols to return to the table of contents.
On another page, you can also read about the history of Shoddy Battle.
§ Free, open source, and cross-platform software
Shoddy Battle was and is free and open source software, which means that the source code was and is available for users to review, learn from, modify, and redistribute pursuant to the applicable free software licence.
In July 2007, a free and open source Pokémon simulator was a fairly revolutionary idea. NetBattle, which was the main simulator used for many years before Shoddy Battle, was closed source and this led to a number of problems, including the following:
- The entire Pokémon community was completely dependent on the NetBattle authors to maintain the software. If the authors lost interest, the Pokémon community would be helpless (which had in fact happened; NetBattle was no longer being maintained). Similarly, if the NetBattle authors had a philosophical objection to a certain feature, it would not be implemented.
- NetBattle had many security flaws stemming from its closed source nature. The authors had assumed the client would not be modified and accordingly had performed game logic in the client that should have been done on the server (i.e. the trusted client problem).
- It was not possible to extend NetBattle in order to implement customised Pokémon mechanics. Similarly, the NetBattle codebase could not be used as the base for other Pokémon projects.
- It was difficult to verify whether NetBattle's Pokémon game mechanics were accurate because Pokémon is a very probabilistic game and the only reliable way to check what probabilities are being used is to inspect the source code.
- NetBattle only ran on Microsoft® Windows® and could not be ported because of the lack of source code.
Shoddy Battle was free and open source from the start. Users were encouraged to inspect the source code and learn from it. Even users who did not contribute were able to use the source code to better understand the Pokémon game mechanics or to detect errors. Additionally, Shoddy Battle was initially coded almost entirely in Java and was able to run on all common operating systems including Microsoft® Windows®, Apple® Mac OS X®, and GNU/Linux.
The fact that Shoddy Battle was free and open source led to a variety of long-lasting positive effects:
- The Pokémon community was no longer dependent on any particular person to maintain the simulator. All users were free to learn from the source and contribute to it.
- Shoddy Battle had a proper client-server network architecture which was secure and did not suffer from obvious flaws. The fact that the source code was open encouraged people to try to find security flaws (and some users did try to do this), which allowed flaws to be found and corrected faster, rather than being undetected until exploited.
- Shoddy Battle was a much more accurate simulation than any previous simulator because users could review the source code for errors rather than having to run tests. Additionally, errors could be fixed faster because more users could solve them.
- The availability of the source code encouraged users to extend the program to add new features. For example, DougJustDoug extended Shoddy Battle to support the Create-a-Pokémon Project.
- Other Pokémon projects were able to use the Shoddy Battle codebase to provide Pokémon battle mechanics. For example, several Pokémon-themed MMORPGs used the Shoddy Battle codebase. Open source code also made it easier for other projects to interoperate with Shoddy Battle.
The open source nature of Shoddy Battle was so beneficial for the Pokémon community that all subsequent Pokémon simulators have been free and open source. In fact, within weeks of Shoddy Battle's release in July 2007, the developers of upcoming rival simulators announced that their formerly closed source simulators would instead be open source. As of December 2012, the current most popular simulator, Pokémon Showdown, is also free and open source.
§ Scalability
The 4th generation of Pokémon games began with the release of Pokémon Diamond and Pearl in September 2006 in Japan and April 2007 in North America. Prior to the 4th generation, online competitive Pokémon was a relatively niche activity. However, the release of the 4th generation games brought unprecedented levels of traffic into the fold of online competitive Pokémon.
Shoddy Battle was prepared to handle the rapid increase in traffic. It supported hundreds of concurrent users (there were typically around 500 users online at a time for most of its life) and it was able to host hundreds of thousands of battles per month. Its high scalability meant it was able to handle ever-increasing traffic, far above what was initially contemplated when the program was written and far above what previous simulators were able to handle.
The following graph shows the number of battles played per month on the main server (the most popular server) between October 2007 and July 2010, inclusive:
Battles played per month on the main server (view data)
As can be seen from the graph, the amount of traffic increased rapidly to reach over six times the initial levels (which is the level of traffic that was that contemplated during development). At its peak, around 700,000 battles were played per month on the main server — around one battle starting every four seconds. Shoddy Battle served the community admirably by handling these high levels of traffic. Between October 2007 and July 2010, there were around 12.3 million battles played on the main server.
It is also worth noting that the main server was not the only server. There were a number of other large servers, such as the Create-a-Pokémon Project server and the Pokemonexperte server, as well as a variety of smaller niche servers.
In the years following Shoddy Battle, the popularity of online competitive Pokémon has continued to grow and subsequent simulators have had to continue the tradition of high scalability set by Shoddy Battle.
§ Extension and customisation
Unlike previous simulators, Shoddy Battle was designed from the ground up to be extensible and customisable by users.
Modification of the game
In Shoddy Battle, the individual moves, abilities, items, rules (called "clauses"), etc. were implemented using a flexible and intuitive API in order to facilitate modification, extension, and the creation of new moves, etc. The framework also allowed for the creation of entirely new Pokémon characters. Some changes could be made with no programming, and even when programming was required, it was simple and within reach of non-programmers. Many users created modified Pokémon games. The most popular sub-community centred on modifying on the game was the Create-a-Pokémon Project, but there were other groups and individuals as well.
The client-server protocol was designed so that the standard client could be used to play on a server running a modified Pokémon game. This allowed people who modified the program to share their work without their users having to download any additional software. The standard Team Builder allowed users to build teams designed to work on any modified Pokémon game.
Modularity
Shoddy Battle was designed in a modular fashion so that one part of the program could be used without the others. For example, several other projects used Shoddy Battle's implementation of the Pokémon game mechanics, including PokeNet Global, a now-defunct MMORPG.
The modular nature of Shoddy Battle meant that it was relatively easy to write tools that interoperate with the software. For example, Martin wrote a tool to import teams from a plain text format. Cathy wrote a tool to generate teams based on the information in a community-maintained repository of information on Pokémon game strategy. (Unfortunately, this tool is no longer available as of December 2012.) Users released a number of interesting tools over the years.
Bots
In contrast to previous simulators, the network protocols for Shoddy Battle and Pokémon Lab were well-documented (and the source code was available) and simple to use. In addition, Shoddy Battle and Pokémon Lab came with a bot framework that allowed users to readily write bots in any one of several languages (Java, Python, etc.) to join the server and interact with users.
Over the years, users wrote bots to do the following sorts of tasks:
- automate common moderation tasks, such as delivering warnings;
- autonomously moderate the chat, e.g., by removing spammers;
- organise tournaments by creating pairings, directing people which matches to play, keeping track of wins (by spectating the matches), and announcing a winner;
- provide information to users about a particular server's special features;
- play battles with users.
One impressive bot is Technical Machine, an AI project by david stone that plays intelligent Pokémon matches on Pokémon Lab (and some other simulators) using a minimax-based algorithm.
Outlook
Shoddy Battle set the bar high for extensibility and customisation features. Later simulators have provided similar facilities in order to satisfy user need; for example, Pokémon Showdown also supports modified Pokémon games.
§ Statistical data and user ranking
Shoddy Battle was the beginning of a new era of Pokémon based heavily on statistical data.
Shoddy Battle was the first simulator to provide statistical information on what Pokémon and strategies were used by players. Before Shoddy Battle, there was no statistical data of this nature available. Shoddy Battle was also the first simulator to provide players with a rated matchmaking facility (the "ladder").
Shoddy Battle's focus on statistics had some significant and long-lasting effects on the community.
Matchmaking and user rating
Previous simulators had included basic mechanisms to keep track of how many matches a person had won or lost. However, Shoddy Battle was the first simulator to take a systematic and statistical approach to user rating. The Shoddy Battle matchmaking and rating system included the following features, all of which were novel for competitive Pokémon at the time.
Rating system
Rather than a simple count of wins and losses, Shoddy Battle kept track of player skill using the Glicko2 rating system. Rather than conceptualising a player's skill as a particular number, each player was assigned a probability distribution; a player's "played skill" realised in any particular game was understood as a random variable distributed according to a normal distribution characterised by two parameters: μ (mean) and σ (standard deviation).
The value of these parameters was adjusted based on game outcomes to reflect both the player's average "played skill" in a particular match (μ) as well as the level of uncertainty in that value (σ). For the purpose of ranking players, Shoddy Battle used a conservative estimate of the player's average "played skill" (defined as μ - 4 * σ). The value of σ also increased with player inactivity, reflecting an increased uncertainty in the average rating, but also serving to encourage people to play regularly, increasing activity on the server.
The sophisticated rating system provided for a more competitive environment and contributed to the popularity of the ladder system.
Selection of opponents
Shoddy Battle did not allow players to select their own opponents for rated matches. Rather, match pairings were made "randomly" based on the relative ratings of players.
Ideally, people would be matched to somebody of similar skill to them because the game outcome of such a pairing provides more information about the skill of each player than does a match between unequal opponents. In a match between unequal opponents, the result gives very little information because it is already known that one player is much better than the other and the outcome does not help determine the degree to which that is the case (without attempting to quantify "how badly" somebody lost, which has a variety of practical and theoretical flaws, and which Shoddy Battle did not attempt to do).
Aside from the rating-theoretic benefit of matching players "randomly" based on relative ratings, the system also produced matches that would hopefully be more fun for the players. In addition, the system prevented cheating by way of fixing match outcomes because a player would not be able to guarantee a match against the collaborator in such a scheme.
Uniformity of ruleset
In Shoddy Battle, rating statistics were "local" to a particular ruleset. In other words, for each ruleset, a player had a different set of rating statistics. The purpose of this was to avoid the effect that playing under different rules might have on a player's "played skill". In theory, we might imagine that there is some underlying notion of "skill" which is predictive of skill in any ruleset; however, measuring such an underlying notion of skill has many pitfalls and typically results in meaningless ratings. Accordingly, Shoddy Battle localised rating statistics to a particular ruleset.
This system also helped players find matches quickly because it reduced the variation in rulesets being played. In Pokémon, a team tends to be designed for a specific ruleset, and the ladder system made it much easier to find a match under the desired ruleset.
Type of Pokémon statistics collected and reported
Shoddy Battle collected statistics on what Pokémon and strategies players were using in their battles. The presence of the matchmaking and rating system meant that usage statistics for ladder games could be assumed to be representative of competitive play, because all players had an incentive to play to win.
At first, the statistics collected and reported were fairly basic and consisted of information on how often each Pokémon was used in each ruleset as a team-member and as a lead (initial Pokémon). Even these basic statistics were unprecedented in the competitive Pokémon community and had immediate practical applications and theoretical consequences (discussed below). As time went on, the statistics collected and reported became increasingly detailed and informative.
The statistics for Cathy's server from October 2007 to July 2008 are archived on this website.
Later, statistics were computed and maintained by other people.
In addition to the basic statistics on the popularity of each species of Pokémon, the statistics collected and reported over the years included the following:
- detailed statistics on the popular choices of stat distribution, nature, item, ability, and moves for each individual Pokémon;
- teammate statistics providing information on which other Pokémon were likely to be on the same team as a given Pokémon;
- typical movesets (groups of moves appearing together) for each Pokémon;
- centralisation and diversity statistics providing information on how varied the gameplay in a particular ruleset was;
- switching statistics providing information on which Pokémon were likely to switch into or be switched into particular other Pokémon.
Various other Pokémon statistics were computed from these and reported for further analysis.
Statistics and tiering
One of the major applications of the statistics collected by Shoddy Battle (both the user rating statistics and the Pokémon statistics) was tiering. In the context of competitive Pokémon, "tiering" refers to the creation of different several rulesets by banning particular Pokémon from each ruleset.
Construction of a mode of play featuring less commonly seen Pokémon
One problem that tiering solves is the fact that there are hundreds of Pokémon, but most of them are not usable in the standard game because they are too weak to be effective. By removing the most commonly seen Pokémon ("OU" or Overused Pokémon) from the game, a different game is created ("UU" or Underused) which provides an alternative mode of play.
Prior to the advent of Pokémon statistics, the construction of tiers had been based on intuition rather than any sort of mathematical metric. The introduction of usage statistics allowed the construction of tiers based on actual usage data, so that the UU ruleset would actually feature less commonly seen Pokémon, which was the intent.
Banning of exceedingly strong Pokémon
A second function of tiering is to remove Pokémon that are deemed "too strong" for a particular ruleset. Pokémon removed from the standard game for this reason were referred to as "Ubers", whereas Pokémon removed from the UU game for this reason were referred to as "BL".
Shoddy Battle's statistics helped in the determination of which Pokémon were "too strong" in a couple ways:
- Uncommonly used Pokémon
If a Pokémon was alleged to be "too strong", one would expect that it would be commonly used because people would be taking advantage of it. If an allegedly too strong Pokémon was not being used very much, it would cast doubt on the claim that it was in fact too strong, subject to any explanation that might be proffered for this discrepancy.
- Analysis of the challenged Pokémon
The Pokémon statistics provided detailed information on how the challenged Pokémon was actually being used, which was valuable information for determining whether that Pokémon was "too strong".
- Determination of qualified users
Ultimately, the question of whether a Pokémon was "too strong" and ought to be banned had to be resolved by some form of vote. In the past, the question of who should be allowed to vote on such a thing was resolved by the community leaders hand-picking people they considered skilled. However, this system excluded a lot of skilled players. The advent of the Shoddy Battle rating system allowed the voter pool to be based on players who had proven themselves skilled through the ladder system, as opposed to through social connections. This made the decision-making process more transparent and fair.
Statistics and strategy
The statistics provided by Shoddy Battle also affected strategy in the Pokémon game itself.
One of the major skills involved in the game of Pokémon is "prediction"; that is, determining what the opponent is going to do based on imperfect and incomplete information. The existence of a treasure trove of statistical information about what people commonly do in a similar situation provides useful information to assist in predicting what one's opponent is likely to do in a particular battle. Of course, the opponent is also aware of the statistical data and as such may alter her strategy accordingly, which must be taken into account as well. David Sirlin refers to this phenomenon as Yomi Layers and it is a hallmark of complex competitive games.
Moving beyond human players, the statistical information also made it possible to write sophisticated Pokémon-playing AI bots whose decisions of what moves to make were informed by the Pokémon statistics. The most sophisticated AI bot to date is Technical Machine, which relies heavily on a variety of different Pokémon statistics in order to predict teams, evaluate game states, and so on.
Outlook
Shoddy Battle significantly changed the landscape of the competitive Pokémon community by providing player ratings and Pokémon statistics. Statistics have become a central aspect of the competitive Pokémon community and now figure prominently in all manner of competitive Pokémon discussions and in the formulation of strategy. All subsequent simulators have continued to provide statistical facilities, inspired by the features in Shoddy Battle.
§ Status
With the conclusion of the 4th generation, Shoddy Battle has been superseded by other simulators such as Pokémon Showdown. Shoddy Battle remains notable for the changes it effected in the Pokémon community. Later Pokémon simulators have followed the path set by Shoddy Battle and continue to expand on the above advances. On another page, you can also read about the history of Shoddy Battle.
If you enjoyed using Shoddy Battle or have questions or comments, please feel free to email Cathy and bearzly at developers@pokemonlab.com and let them know. They appreciate hearing from you. After Pokémon, Cathy went on to litigate and win a landmark judgment for transgender equality rights. You can read about that at cathyjf.com.