In the first two posts on the series we learned how to set up the Advanced Combat Tracker and evaluate dungeon runs. Today we are going to look at another very popular way to utilize the Combat Log: Dummy Tests. Several areas in Neverwinter such as Dread Ring or the Trade of Blades feature Training Dummies that players can hit infinitely. This makes them ideal to test builds and record and compare DPS meters.
To understand the value of these tests we need to talk about experimental theory a bit. Any empiric study faces the challenge to balance between internal and external validity. Internal validity is an estimation how accurately an observed effect can be credited to its assumed cause. External validity describes to which extent results can be held true for other cases and situations.
As you might imagine our recorded dungeon runs do not offer enough internal validity in terms of builds. Too many other stuff factors into your DPS. Even doing two runs with the exact same build can lead to vastly different numbers, because you have no chance controlling all conditions like group composition or the type of mobs you face.
This is where dummy tests can help setting up your ideal rotation and build. Assuming that no other players are hitting the dummies or providing buffs, you will always find the same condition while running your tests. So you can be pretty sure that recorded DPS with different rotations, builds or gear are a good representation of actual performance. You can even switch between single target and AoE simulations by adjusting the amount of dummies you hit.
Let’s go kill some dummies!
The approach is pretty simple. You set up your build, start hitting the dummies for about five minutes and record your DPS using the ACT. Then you respec your character and repeat. You can then use the parses to compare your performance. I personally even took it a step further and use macroed rotations, because it rules out any human errors. Also the tests can become pretty boring. A solid series can easily contain a few dozen tests. We’re talking about different Weapon Enchantments, Feats and Powers, possibly even gear. With a macro you can just run the tests and do something else in the meantime. I would also recommend doing this on the preview server, because you are less likely to run into other players occupying the dummies and respecs don’t cost anything.
There are flaws
Dummy tests are an efficient way to evaluate builds, but they are not without flaws. Going back to the external validity you can’t be certain that the difference in DPS you observe in your dummy tests also equals the difference in the field. The advantage why we were choosing the dummy tests in the first place now has the opposite effect: You simply can’t simulate the changing conditions of dungeon runs. This is exactly the challenge of internal vs. external validity I was referring to. Dummies are not the standard mobs you are going to face. They have no damage or control resistances. So Control Wizards with Chilling Presence will benefit from its damage bonus longer than usual. And like Kaelac found out and Sharpedge has mentioned some debuffs are mitigated by damage resistance multiple times.
The best way to accurately measure DPS would be to do the exact same run with the exact same group over and over and over again for each build you want to test. Good luck finding such a group. Or in other words: It’s simply impossible. What you will do is achieving the maximum amount of internal validity by doing dummy tests and then take your results out on the field. In most cases you will find that what you tested on the dummies matches performance in dungeons or PVP. In case there are any discrepancies, you can still go back to the drawing board.
Despite some disadvantages dummy tests are the best option to measure different builds. And it doesn’t stop with DPS by the way. You could also set up rotations for optimal debuff stacking and such. It’s a great foundation for theorycrafting and even enables players to compare classes with each other.
I hope you enjoyed the article! In case there are any questions or remarks, make sure to post them in the comments below or the corresponding thread on our message board!