• Please make sure you are familiar with the forum rules. You can find them here: https://forums.tripwireinteractive.com/index.php?threads/forum-rules.2334636/

The Life (And Death) Of A Bug

Yoshiro

Senior Community Manager
Staff member
  • Oct 10, 2005
    14,495
    1
    4,490
    East Coast
    Hello everyone!

    First and foremost we would like to highlight that we believe we have found and fixed the cause of many (if not all) the prone related jitter/movement issues. While the team is continuing on to work towards our other target goals (of performance, balance and bugs) before we start the next beta wave we want to once again look at some of the specifics of this work, today focusing on bugs.

    Helping Us Help You - Discover and Reproduction
    The first two steps in getting a bug fixed are to discover the bug and, critically, find a way to reproduce it. When it comes to software development this happens both internally with the QA (Quality Assurance) team as well as reports from the users (thats you!).

    You may ask "why do we rely on user reports?", and that is a good question. While we wish we knew about every bug, it can take 100s of thousands of hours in the game to actual identify obscure bugs, that may happen only in specific combinations of actions or of hardware/software configs. That is where you have all come into the equation, around the closed beta sessions - even if we had the whole dev team playing full time, it would still take us actual years to clock up that many hours of play.

    But knowing a bug exists is only the start of the journey. To get it fixed, we have to be able to (reliably) reproduce it, both so we can home in on the root cause, as well as prove that we've actually fixed it. The root cause of a bug in a game can be almost anything. A few examples of this (that we've seen) includes but is not limited to:
    • Another system providing bad data to the system where the bug is happening
      • This can be both direct and indirect as data travels between systems in the game making it very complex to track down!
    • Timing specific errors
      • Button or event sequences carried out in just the right order at sometimes split second timing requirements causing a system to be left without information it needs or in a bad state
    • Client and Server sync
      • The players local game is running a simulation of the server state and updating as needed when it is sent commands and information by the server. But sometimes events can fire out of sequence between the two and if not properly captured and corrected (by the server) can lead to even worse problems
    • And Third party software (not the game itself) including:
      • Operating System and Prerequisite software updates and corruption
      • Security software (firewalls)
      • And many others (including some streaming software, something that only becomes more and more popular among users!)

    Having a reliable reproduction case allows our QA team and developers to use additional tools that can't be run during normal gameplay (a debugger tool can often make a game run at 30 seconds per frame) to take a very detailed look at code execution and can help pinpoint those root causes.

    Estimation and Bug Triage
    Once a reliable reproduction case has been found, the programming team puts an estimate on time it would take to fix the bug. These estimates are then used during Bug Triage. For us here at Tripwire and Antimatter, triage is usually a weekly event (but closer to a launch it becomes a daily event) where the production team, QA team, community team and project lead decide how to best make use of the time left before a project launch or update. Each person in the studio has a known amount of time and they use the Bug Triage to fill that time and decide which bugs won't make it and push them to a later update. The roles of each of the teams/people involved looks something like this:
    • QA - Provides further information about bugs if needed or take requests to get more information or better reproduction steps
    • Community - Makes the case for top community issues
    • Production - Ensures that people have a decent chance of actually get the work done in the time available
    • Project Lead - Makes the final call on prioritisation

    Fix Attempt and Test
    Once a programmer has been assigned to work on a bug during triage they will make an attempt to fix it. After this fix attempt is made the bug goes back to the QA team to verify that the fix works and to check the system (and surrounding systems) to see if additional issues have been introduced in the fix attempt.

    In the days leading into a release or update the QA team does a full game pass trying to make sure no new bugs have been introduced and that all tested fixes have made it into the release candidate build.

    Thank you for your interest in the game (and the game development process if you made it this far!)
     
    Thanks for insights into the bug fixing process.

    Very happy to hear about this focus on bug fixing and performance over last couple weeks. It was kind of the primary issue holding back full enjoyment of the game during last couple betas.

    Being unable to properly use prone was a particular annoyance.

    Look forward to next beta/release.
     
    Upvote 0