A/B Tests

A/B testing allows developers to run a controlled experiment between two or more versions of something in their game to determine which one is more effective. Your audience is segmented into the control group (current performance – no changes) and your test group.

A/B testing is a great way to find the best pricing and game difficulty or test any of your hypotheses.

Before using, all A/B Tests must be added to the table.

Parameter	Description
Name	The name of the Test.
Groups	Each user gets one random group.
Target Audience	The portion of your total users participating in the test.
Users Type	Defines the kind of users to target the test: New, Old, or All. If the app is opened for the first time, the user is considered a new one until he restarts the app.
Concurrent Test	Such test ignore the fact that some other tests can be active. All concurrent test will be activated for your users and won't affect Non-concurrent tests.
Conditions	Conditions that are required to start the A/B Test. After A/B Test starts for a player, it won't be stopped for him even if the conditions turn to False.
PreInitLaunch	Defines if the A/B Test can be launched during PreInit and before Balancy OnReadyCallback is executed. Read more here

Balancy ensures that each user is not simultaneously participating in two A/B Tests (unless you are running Concurrent Tests). If a user has a running A/B Test, he is skipping all other A/B Tests, even if they target 100% of the audience. Once the A/B Test is finished, the user can join a new one.

A/B tests don't need priority. When the game starts, and a user doesn't have any running A/B Tests, the user collects all Active A/B Tests and picks just one of them, according to the chance.

There are currently 3 ways to work with A/B testing:

1. Conditions

2. Use the A/B Test Node in Visual Scripting

   

3. Make your logic

Fast A/B Testing

1. Find any record in Balancy you want to run A/B Test on.

2. Click on the 3 dots next to the Document Id, and select Create A/B Test.

   

3. Select parameter you want to A/B Test from the list of available parameters:

   

4. Create A/B Test.

A/B Test lifecycle

The main rules for each user individually:

• Only one non-concurrent Test can be active at the same time, unless you activate Tests manually.
• If a Test passed its Condition and wasn't activated due to the User Type or Audience Size, the test will be ignored by this user forever.
• Concurrent Tests exist in a separate world, it is possible to have many active concurrent tests.

Finish A/B-test

When you want to finish A/B Test and apply results:

1. Open A/B Test info modal.

2. Press "Finish Collecting Users". After that, users will not be able to join this A/B Test.

   

3. You might want to keep it for some time to collect more data from participants.

4. Stop A/B Test. This will stop collecting data from participants.
5. Choose variant you want to roll out to all users.
6. Save.

Now all overrides for this variant will be applied to documents parameters itself for all users.

You still need to deploy changes after that, so users will see new values.

Deploy and migration!

You have to do all changes on dev environment and then migrate to stage/production. Read launch check list for better understanding data migration process.

External References

For your convenience there is a section for external references to that A/B Test to find what other documents might use it in their setup.

Visual Scripting References shows all scripts where this A/B Test is mentioned in A/B Test node.

Other References shows all other documents where this A/B Test is used in Conditions.

A/B Test Analysis

Balancy automatically tracks the performance of A/B Tests and provides you with analytics.

1. Find the A/B Test you need and expand its view:

   

2. Analyse the performance of each variants:

   

Section for programmers

After the initialization of Balancy package, you can get all current A/B tests and their variants for the user.

var allTests = Balancy.LiveOps.ABTests.GetAllTests();
Debug.LogWarning("All Tests: " + allTests.Count);
foreach (var test in allTests)
    Debug.Log("Name = " + test.AbTest.Name + "; Variant = " + test.Variant.Name + "; isFinished = " + test.Finished);

//You can manually launch a test and place the user in the specified variant:
LiveOps.ABTests.StartABTestManually(test, variant);

You can send all the A/B Tests' data to your own BI or analytics system:

namespace Balancy
{
    public class SmartObjectsEventsExample : ISmartObjectsEvents
    {
        ...

        public void OnAbTestStarted(LiveOps.ABTests.TestData abTestInfo)
        {
            Debug.Log("=> OnAbTestStarted: " + abTestInfo?.AbTest?.Name + " ; Variant = " + abTestInfo?.Variant?.Name);
            //TODO Send AB Test info to your custom analytics
        }

        public void OnAbTestEnded(LiveOps.ABTests.TestData abTestInfo)
        {
            Debug.Log("=> OnAbTestEnded: " + abTestInfo?.AbTest?.Name);
            //TODO Send AB Test info to your custom analytics
        }
    }
}

External Tests

If you want to manually launch A/B Tests and define the variants assigned to each player using your algorithms or any 3rd party solution, follow the next steps:

1. When creating A/B Test add an Empty condition with false value, automatically making A/B Test unavailable for all players.

   

2. Invoke the following code to start A/B Test and assign a variant manually:

   var allABTest = DataManager.SmartObjects.Analytics.ABTests;
   var myTest = allABTest[0];//find the AB Test you want to start for this player
   var myVariant = myTest.Variants[0];//find the variant you need
   
   LiveOps.ABTests.StartABTestManually(myTest, myVariant);