[Natsumi]

Beta testers, one of the most useful tools for developers, and essential for quality assurance. They are a necessity for any self-respecting developer. However, as I've started to notice over the years, it seems that the Sonic hacking scene is full of incompetent, clueless and utterly useless beta testers. Aside from a few selection of people, all I get is “it's good”, “I like it”, and “nice work” when I'm really looking for criticism to improve my work. It, indeed, seems that a popular belief says that beta testing is all fun and game.

It's not.

When we ask you to beta test, we aren't looking for you to suck our e-dicks. We rather ask you to help us find flaws in our projects. Bugs, bad level design, something that isn't executed well, or even just general ideas for improvements. Unfortunately, that's not what most people are after (helping us), they just seem to be wanting to play something for fun. But the reality is, nobody wants to just share you an early beta of a hack for you to play before everyone else can just for the heck of it. Beta testing is you playing the product over and over and over to find flaws and report them back to the developer(s).

Now! If I made you feel guilty for your complete incompetence at being helpful, good! That was the intention. But fear not, because I will guide you become the opposite: useful! So, without any further ado, let's start!


Step 0: Can You Be a Beta Tester?

Before we start with the actual help, there are a few things about being a beta tester that are obligatory to for you to be able to do, and without being able to fulfill all of them you should reconsider your capability for the job.

• Time Devotion:
  Being an efficient and helpful beta tester means you must devote a great portion of your free time to the job. It is usually good to be available more often than once per week, and at least an hour or two at a time. It depends entirely on the developer(s), project, and possibly other any other beta testers involved. But generally speaking, two or more times per week, for few hours at once, should be a good average.
• Non-disclosure obligation:
  Imagine a social worker and client relationship. The social worker is legally obliged to not disclose certain types of information about the client to anyone without permission from the client. The very same thing applies to the beta tester and developer relationship. While this, as far as to my knowledge, cannot be legally enforced, it is generally accepted as a rule within workplaces and non-workplaces alike. This means, not even telling your best friend about the most insignificant detail, unless consent by the developer(s) is given.
• Patience:
  Like with anything, patience is necessary with beta testing as well. The developer(s) may want to test or adjust a feature, and that may need frequent updates. If you are in this kind of situation, it may frustrate you a bit, but remember you are only helping by providing feedback and answering questions quickly.
• Criticism:
  This is the main thing all beta testers need to master. If you want to help the product to be successful, you need to be able to know what is good, what is acceptable, and what is outright garbage. I often see people, especially young individuals, lacking the ability to criticize something that is clearly subpar, and in the same breath they can shun something that is actually pretty good, just because they don't want to mess with their previous opinion. This is completely unacceptable of any beta tester, and in fact it's usually the best to strive for a mostly objective opinion about something. Now, it's also good to insert your personal opinion to a degree, obviously. (Unless you genuinely think that Sonic I AM AN UNFUNNY SHITSTAIN's visuals are acceptable for anything else but a joke. Then you should give up trying.)
• Experience:
  Having knowledge of the genre and/or style a product is aiming for is important, so that you can give the developer(s) nudges to push them on the right track. This may be the result of anything from you having played few games alike or even developing your own before. Even if you do not feel you have good grasp on similar games, you can always just play a few to hopefully get an idea to what you're jumping into. I will later discuss more Sonic hacking related experiences that should help with that.
• Passion for Gaming:
  Unless you absolutely love gaming, playing games, and criticizing them, you will sooner or later get tired of beta testing and stop being helpful. It is impossible to universally get enjoyment or even a sense of satisfaction from beta testing, unless you're the kind of person to explore every nook and cranny of a game until you found every last thing. If you already love hunting bugs for games like Sonic, or even games in general, then it will be more likely you keep an interest in beta testing for a long time, as it's kind of an ever-changing puzzle of challenges of you finding every flaw possible.

If you have fulfilled most or even all of the above, you should be good to go! If not, maybe educate yourself a little; research, listen to critics, etc. These should help you to pick up on most necessary things.


Step 1: Tools

Now, I can't really speak any further than our dear Sonic hacking community, because any different work environment or platform will require their specific sets of tools.

Emulators:
Depending on the task you are doing, or the nature of the testing, some emulators will be more useful than others, and while these aren't rules to follow, they certainly are strong recommendations from my perspective, and are tools I use for said purposes. Note: It is suggested to have multiple emulators available to test on for different purposes.

• Kega Fusion:
  This is really good emulator for testing playability. Not so much to test strength of code, but how the game plays, as it is most common emulator among casual players and likely the main emulator the general public will use once your product is out. If you create something for the sole purpose of having many people play it, you are almost forced to make sure it runs at least on this one.
• Regen:
  Another common emulator for people who play casually; but more so for other hackers. Not only that, but it is vastly more accurate to real hardware in comparison to Fusion, which intentionally isn't as accurate (to be lighter to run, especially on low end machines). Regen is good for testing most of the code you'll come across. However, keep in mind there are always those few people who actually play on real hardware, and testing your work on Regen for that is not fool-proof. Also remember that sound emulation in Regen is not fully accurate.
• Exodus:
  By far the most accurate emulator to date. Although it also doesn't fully replace hardware, and is very CPU-intensive. For anyone able to run it well, it's definitely the option to go for when testing complex code.
• Flashcarts:
  While not emulators per se, these in combination with real Mega Drives, are by far the best platform to test hacks on. While it limits your options on screen capture, as well as the experience the common user would have (and hence is not suggested to use as the only testing environment), it will help any developer the most, especially keeping their software compatible. The most suggested flashcart is Mega Everdrive, even if it has proven to be not fully accurate, and is rather expensive, it's the best means we have for hardware testing that I know of.

Screen Capture:


If you want to be a useful tester, you will also want to have a way to capture what you are doing on your end, videos or still images. These methods should allow high quality capture from your computer. Note: Never save the images as JPGs; it is a lossy compression and with small enough images it will make them not only larger than PNG or BMP, but also worsen the quality. PNG and BMP are suggested (though the latter only if you can't be arsed).

• Print Screen:
  This is a key in your keyboard that allows you to capture image from your entire screen and put it in your clipboard, or while holding ALT and pressing Print Screen to only get an image from active window. This is a quick and effective way of taking an image. And as a bonus, imgur.com allows you to directly paste it into the site and upload directly. Some keyboards may also use the abbreviations, and MACs may not contain this key at all.
• Built-in screenshot features:
  These are emulator-specific features which allow to output correct aspect ratio and size images from your emulator window directly. While it's a little slower to do, it will also give a better quality than the PrintScreen technique, and will make pixel-accurate outputs.
• Open Broadcast Software:
  OBS is a handy tool for recording any window or area, with any sound from your stereo mix or microphone. While this may not give optimal quality for your emulator, it will allow to quickly capture any emulator with fair ease. An alternative is Gyazo.
• Gens Rerecording:
  This is easily the best way to record videos of you playing a hack or a game on the Mega Drive, Mega CD or 32X. It gives high quality AVI output, allows to record a playback movie for you to later decide to record or not record an AVI out of it, and is a good for Tool-Assisted Speedruns. This is not a good emulator for accurate testing however, so sometimes an additional tool may be needed to catch a bug that may not occur in Gens Rerecording.
• Text Editor:
  You will also need a text editor. Preferably one which produces clean ASCII text output, not any “advanced text formats” (such as RTF, DOC, etc.). This will help you when taking notes and sharing said notes with the developer.

Step 2: Beta Testing for the First Time

Eventually the time will come that you are asked to test something. Or the opposite happens, your request to receive help from someone was accepted. So... what should you do? Well, perhaps there isn't a straightforward answer, but I can share my recommendations.

Try to fetch as much information about the project as you can. What are its goals? What it is supposed to feel like? What does the developer want to achieve with it? These are the type of questions you should be asking. But, try to avoid getting any spoilers. Why? Because the first thing you should do is having a blind playthrough. This is likely the most helpful information for any developer, as most people will be playing with not much knowledge of the product in question, so the first impression is important. It is also good idea to get to know a little bit about who you are working for. Not like personal details, but more along the lines of their personality and work motives. It should help you later in case of disagreement or conflict.


Step 3: Different Beta-Testing Methods

There is quite a bit of variation involved when it comes to ways of beta testing, and some are more helpful than others. I can't give you an exact plan of what is right and what isn't, but I can give you vague concepts to mix and match, and develop your own methods based on them.

• Take Notes:
  Taking notes in your text editor of choice, listing pros and cons, or maybe just your general thought, issues, etc.
• Screenshots and Explanation:
  Another method is showing a screenshot of something you see, and then explaining why you took it and what the developer should pay attention to. Be it a bug, something off with a thing, or even you noticing something positive and surprising! It's also sometimes easier to just show a picture than explaining stuff with words.
• Video Play:
  Or, you can record yourself playing the hack, whether to find bugs, or just to do gameplay, and show this to the developer to analyze what a casual playthrough may look like.
• Review:
  This would you be you, doing a concise or not review about the hack, pointing out your thoughts and maybe listing a few things of what you would like to see in the future.
• Report:
  You doing a short and on-to-the-point observation about the hack, pointing out flaws you find along the way.

Speaking of flaws, don't be afraid to point out everything, no matter how minor it may seem, or even if you have pointed it out before and it has been neglected. But it is also good to learn the difference between engine issue and user-made issues. Someone new, who doesn't quite understand how the engine works yet, may not be able to fix those engine issues, but merely be able work around them. Usually good developers want to know about issues, even how minor or obscure they may be. But it's good to prioritize issues that are larger and easier to trigger, and move to more and more difficult and obscure ones as more issues get fixed.

It's also good to mention about how you represent your thoughts. It is important that you don't sugarcoat things much, but that you also don't be too harsh on the person either. It may be hard to find a good middle road between the two, and can affect how your suggestions are taken. Usually, it's helpful to try to be neutral about it. It's also good to note, that the developer may have bad response to even reasonable criticism. If they don't think it's important to fix something that clearly needs to be fixed, you should show them how much it matters, even if by annoying them about it a little. If they still refuse to listen to you, or even take offense, it's good to reconsider if you should work for them anymore.


Step 4: My Last Stupid Thoughts

Here are a few bullet points about how I often work:

• Videos about new features:
  Whenever I'm told to check out a new feature, or major change, I strive to make a video about me testing it, to show how I would approach it as a player.
• Reporting Issues:
  I'm often just lazy, take a screenshot, and write short text about what I think is wrong in my picture. If I can't get a snap, then I try to explain by text only, but I avoid doing that.
• Finding Bugs:
  I often intentionally try stupid, absurd and unexpected things, just to see if the game would handle it well, or not. Even trying known tactics to make the game freak out (Such as common glitches from the original games or previously found bugs). I do it over and over again to ensure nothing slips by. Of course, sometimes the more obscure things will.
• Dedication:
  When I'm beta testing for something, I easily spend hours, even for something that isn't too mandatory. I try to do many different things to put the game to test. It takes a lot of effort and sometimes just may be boring, but getting complimented for being helpful is easily worth it. Plus, it's fun if I can find something ridiculous.
• Interaction:
  I also talk to the developer frequently and suggest things to them what I think could enhance the product, as well as offer ideas as to how to solve issues. Probably not everyone appreciates this, but I like to do so anyway.

Step 5: Final Words

Overall, being a beta tester is definitely not an easy job to do, but it is really useful one. Don't worry if you feel like you aren't helping much, because the opposite is true. You will be irreplaceable for the development team at some point. But don't let it get to you, because you aren't the only one capable to do the same job.

Most importantly, don't make beta testing a competition!

Oh, and thanks for KingOfHarts and especially Selbi for helping me make this.

This post has been edited by Green Snake: 25 November 2015 - 06:56 PM

[Linaru]

An excellent summary of a beta testers job. hopefully people will take it on board.

[nineko]

Having OCD, I can be a quite effective beta tester and anyone who got their hacks beta tested by me can confirm how much a pain in the ass I can be, sometimes I provided dozens of screenshots about the most obscure bugs in existence. I like to do things which shouldn't be done, if you saw some of my playthroughs on Youtube you know that I always tend to exploit bugs and whatnot. Sometimes I even found some bugs in the original games before they were documented by anyone (the MGZ2 boss in act 1 comes to mind, or that LZ2 skip which is now used in most TASes (I found it way before they did, too bad I didn't document it, either)).

tl;dr I can be a brutal beta tester if you want

[Mr Lange]

I'm going to add that you can be a tremendous asset to beta testing if you have at least some understanding of how computers and video games function. This kind of knowledge will lend a sense of what to test for and how, and make it easier to interpret things that are happening which also means better communication with the developers. However, not every beta tester should be this way; the best group of testers should include a mix of people with working knowledge of video games and technical virgins. You want people who are playing the game with the mindset of a normal player with no technical knowledge tainting their experience and behavior, as defining the game experience is more than just fixing bad code. But at the same time, you want people who know the kinds of specific things to look for, a mindset of poking for numbers and triggers, and able to communicate what things are happening under the surface.

This post has been edited by Mr Lange: 25 November 2015 - 09:38 PM

[KingofHarts]

I regret not being able to finish looking over this with you GS, unfortunately nature of the beast lately where I am at...

That said, this is well written and I hope it reaches as many people as possible. Thumbs up

[Jase]

Man I'd love to have a fangame beta tester that fits this description.

[.Luke]

I have a Shantae fan game that I'm starting for real now, so once I start asking around for testers, I'm totally linking to this thread, Green Snake. Lots of helpful stuff in your post! It's going to save many of us a lot of time writing posts in a similar vein.

By the way, what's a good GIF recorder? Sometimes that can be more convenient than a video file for showing bugs in action. Recordit and gifrecorder are the first results I'm seeing, but I have no idea how good they are.

Mr Lange, on 25 November 2015 - 09:38 PM, said:

I'm going to add that you can be a tremendous asset to beta testing if you have at least some understanding of how computers and video games function. This kind of knowledge will lend a sense of what to test for and how, and make it easier to interpret things that are happening which also means better communication with the developers. However, not every beta tester should be this way; the best group of testers should include a mix of people with working knowledge of video games and technical virgins. You want people who are playing the game with the mindset of a normal player with no technical knowledge tainting their experience and behavior, as defining the game experience is more than just fixing bad code. But at the same time, you want people who know the kinds of specific things to look for, a mindset of poking for numbers and triggers, and able to communicate what things are happening under the surface.

This. All I'd ask in any beta tester for a game I'm working on is a basic understanding of these things. (Although I'd love to see how many different ways someone like nineko would destroy a game. )

I had a guy during the early stages of the Shantae project, who's Intel CPU was so odd that I sometimes needed to rewrite code to get things working properly on his end, (It was a good thing, because it meant a broader range of compatibility.) and at the same time, he was completely, totally daft on how games work underneath, despite being what you'd call a game critic. It took me a week to explain to him that I had the Harpy Form's movement speeds mostly accurate; the animations that day just weren't timed yet to match the speed. He could only see the animation and not the real speeds Shantae was moving at, ugh.

So I agree a healthy balance of both average players and technical geniuses are helpful. The clever geniuses looking for things to break with an evil grin may miss something the average person lacking the mindset of a programmer wouldn't see, and visa versa. I've had enough experiences to know to be more strict about who helps me test stuff.

[Irixion]

Very comprehensive. To add to what MarkeyJester said, you could also have terrible luck (like me) and have glitches find you. Randomly at first, and then you try and figure out how to recreate them

[Mr Lange]

.Luke, on 27 November 2015 - 07:03 PM, said:

By the way, what's a good GIF recorder? Sometimes that can be more convenient than a video file for showing bugs in action. Recordit and gifrecorder are the first results I'm seeing, but I have no idea how good they are.

I've been using gyazo, wonderful utility. Allows you to do instant screen caps and short screen recordings (a few seconds each) by selecting portions of a screen, and the render is immediately uploaded and hosted on the gyazo site which can be hotlinked. Recordings are rendered in both gif and mp4 at once, with the mp4 version being higher quality. This utility has been invaluable in testing and showing devs anything and everything, and of course is useful for many other miscellaneous things. And yes, it's free.

.Luke, on 27 November 2015 - 07:03 PM, said:

he was completely, totally daft on how games work underneath, despite being what you'd call a game critic

Oh these are just my favorite kinds of people.

This post has been edited by Mr Lange: 28 November 2015 - 06:18 AM

[.Luke]

You can imagine the immense frustration I experienced in that case. I could write a long list of all the things he misreported as bugs; that particular example was his pinneacle of unhelpfulness. Developers beware! Everyone's a critic, but not always a critical thinker.

As for gyazo, that's an awesome feature set. Once the Internet's setup properly here, (On 3G.) I'm downloading it first thing. That could have been helping me with any number of things related to development months ago!

[MarkeyJester]

I by no means wish to sound negative, I can see you have put quite some time and effort into presenting this tutorial. It is clean, efficient, and quite self explanatory. Form follows function.

Through experience however, we know that this tutorial will have about a 10% or less impact on the actions of beta tester wannabes, because they are no different to hack/game wannabes. If these wannabes are ignorant enough to not know how to beta test effectively, then they are ignorant enough to miss this thread or even read it. Even if the thread were to become a sticky thread, you're still looking at a replica of Jayextee's So, you want to start hacking Sonic?, and Jman's You all suck at programming, of which still to this day hasn't counteracted the problems they were meant to solve (in fairness, some of it is related to progress and change in information, but that's besides the point).

I would suggest making a guide for those who are "looking" for beta testers (the developers themselves), like for example, "how to spot a decent tester on the board via their posts and the points they make". Maybe have a list of selected people (with their permission) who are decent at beta testing, and are currently taking requests. The developers themselves are more interested in getting it right, and will be less ignorant and actually read (in full) the tips that are presented to them, after all, the developers want to fix the serious bugs, not just hand them out willy nilly to any tom, dick or harry full of bugs for fun (unless there is a tom, dick or harry on this board that is actually decent at beta testing).

[Natsumi]

I know there is no way to get lazy or incompetent kids to essentially not become this, but I also know there are people who want to be helpful but simply have no clue about how, and I hope at least that part of people can take something out of this. Furthermore there are people who may be helpful, but could always be better at this, or just learn few more techniques. Also its a good text to link to anyone who asks me, or I ask them, to beta test for me, and ask to show, based on this, what they could do with some other project. This would practically show me if they are just incompetent and not worth my time, if they end up not putting forth any effort. As far as the latter is concerned, I'd love to steer the tutorial more to that direction, but alas, I don't think I would know how to do in efficient way. Because funnily enough, I've never quite found anyone just from the boards that are helpful. Luckily I ran into someone at IRC, but that's about all the useful beta testers I've had for months.

[winterhell]

Guess I'll have my 2 cents input here.

For a tester, it helps very much if they have previous involvement in making a bigger-ish project, lets say 10 000+ lines of code) and preferably have been programming there.
Be sure to try every menu, button and feature you can. Next step is to be creative and try bizzare combinations, rare/edge cases.

Examples: What happens when I try to delete the object while I'm dragging and dropping it? What if I try to paste it after its been deleted? If I click on this popup/dropdown menu and there is a button under it, would both activate? Changing the edit mode to collision changes the palette to grayscale - what if I try to change it back to colour and draw the collision pieces like that?
I can target only enemies with this splash damage rocket, but what if I have allies in the splash radius, would they get hurt?
And 1 example from Dota 2: 2 allied units try to get on the opposite side of the forest and there is a shortcut tunnel. They approach it and they enter it simultaneously, blocking each other, so they try to find another way and they both unblock the path, then see its unblocked and... this goes on for a while.

I'd put testing in 3 categories:
a) platform testing. Does the game run on this and that configuration. For a Mega Drive rom sure, does it run on hardware, which emulators. For a PC game the variations are usually the amount of RAM, OS version and video card type - AMD, Intel or Nvidia and how old. Usually if it works on the older it'll run on the newer of the same brand.

b) full scale try anything - this would be trying odd cases like with the examples above, and also trying a playthrough. In the past decade many AAA high budget games I've played have exhibited multiple game breaking bugs that kill you or something on the first run, and not even odd bugs that happen on a blue moon but every time you go through that section.
If a cold turkey person playing the game for the first time, goes into such bugs on a playthrough, the game should be illegal to be released to the public, especially for 60 bucks with first day DLC. I mean come on! Have at least 3 new people play it hassle-free before release.

c) 'I worked on this feature/bug, try to see if its fixed or if it breaks something'. Basically make any combinations involving this feature. In rarer cases it might trigger a problem if you play with it first, then go to another feature or vice versa.

If you want to be a helpful tester and see a bug, take notes on how you managed to make it. Video recording or some kind of replay feature definitely helps. Then try to repeat it, and try to reduce the steps necessary to reproduce it, narrowing it down. See if it breaks only in a particular location/situation, or does it work in other places as well.
When you post a bug report, be sure to describe the problem in couple of fragments: what should happen, what does happen, how do you make it happen, screenshot / video.
Then of course try to assist the programmer with c)
If possible, before submitting the bug report, see if others haven't submitted it before, if its a known bug, and even if its already fixed in a newer version that you don't have. If its an odd bug that others couldn't reproduce then by all means try to provide new information that helps. Maybe you have a different system, you encountered the bug in a different situation, or have an easier way to reproduce it.

For projects with 1 or only a few programmers you might not even need tester(s) for b) and c)
People who love the project tend to test the things they are making and by using 1-5% of the coding time for testing you'll catch 95-99% of the bugs.
Unless its a multiplayer game and your boss explicitly forbids you that 15 daily minutes of 'fooling around'.

This post has been edited by winterhell: 29 November 2015 - 07:17 AM

[KingofHarts]

MarkeyJester, on 29 November 2015 - 04:04 AM, said:

Even if the thread were to become a sticky thread, you're still looking at a replica of Jayextee's So, you want to start hacking Sonic?, and Jman's You all suck at programming, of which still to this day hasn't counteracted the problems they were meant to solve

For what its worth... that first thread is what I read that finally got me into Sonic hacking in 2012... I was always intimidated by the thought before I finally started looking into that thread, and going from there. I never saw Jman's thread, seeing it now... I'm really happy I did ultimately read it. Thank you for linking them.

This post has been edited by KingofHarts: 29 November 2015 - 02:39 PM

[Shadow Fire]

Bethesda's beta testers should read this. Maybe their games would then increase in quality and wouldn't result in the necessity of being fixed by the fans.