A/B Tests

A/B testing allows developers to run a controlled experiment between two or more versions of something in their game to determine which is more effective. Your audience is segmented into the control group (current performance – no changes) and your test group.

A/B testing is a great way to find the best pricing and game difficulty or test any of your hypotheses.

Before using, all AB Tests must be added to the table.

Name	Description
Name	The name of the Test.
Groups	Each user gets one random group.
Target Audience	The portion of your total users participating in the test.
Users Type	Defines the kind of users to target the test: New, Old, or All. If the app is opened for the first time, the user is considered a new one until he restarts the app.
Concurrent Test	Such test ignore the fact that some other tests can be active. All concurrent test will be activated for your users and won't affect Non-concurrent tests.
Conditions	Conditions that are required to start the A/B Test. After A/B Test starts for a player, it won't be stopped for him even if the conditions turn to False.

Balancy ensures that each user is not simultaneously participating in two A/B tests (unless you are running Concurrent Tests). If a user has a running A/B test, he is skipping all other A/B tests, even if they target 100% of the audience. Once the A/B test is finished, the user can join a new one.

A/B tests don't need priority. When the game starts, and a user doesn't have any running A/B tests, the user collects all Active A/B tests and picks just one of them, according to the chance.

There are currently 3 ways to work with A/B testing:

1. Conditions

2. Use the AB Test Node in Visual Scripting

   

3. Make your logic

Fast A/B Testing

1. Find any record in Balancy you want to run A/B test on.

2. Click on the 3 dots next to the Document Id, and select Create A/B Test.

   

3. Select parameter you want to A/B Test:

   

4. Create A/B Test.

A/B Test Analysis

Balancy automatically tracks the performance of A/B Tests and provides you with analytics.

1. Find the A/B Test you need and expand its view:

   

2. Analyse the performance of each variants:

   

Section for programmers

After the initialization of Balancy package, you can get all current A/B tests and their variants for the user.

var allTests = Balancy.LiveOps.ABTests.GetAllTests();
Debug.LogWarning("All Tests: " + allTests.Count);
foreach (var test in allTests)
    Debug.Log("Name = " + test.AbTest.Name + "; Variant = " + test.Variant.Name + "; isFinished = " + test.Finished);

//You can manually launch a test and place the user in the specified variant:
LiveOps.ABTests.StartABTestManually(test, variant);

You can send all the A/B Tests' data to your own BI or analytics system:

namespace Balancy
{
    public class SmartObjectsEventsExample : ISmartObjectsEvents
    {
        ...

        public void OnAbTestStarted(LiveOps.ABTests.TestData abTestInfo)
        {
            Debug.Log("=> OnAbTestStarted: " + abTestInfo?.AbTest?.Name + " ; Variant = " + abTestInfo?.Variant?.Name);
            //TODO Send AB Test info to your custom analytics
        }

        public void OnAbTestEnded(LiveOps.ABTests.TestData abTestInfo)
        {
            Debug.Log("=> OnAbTestEnded: " + abTestInfo?.AbTest?.Name);
            //TODO Send AB Test info to your custom analytics
        }
    }
}

External Tests

If you want to manually launch AB Tests and define the variants assigned to each player using your algorithms or any 3rd party solution, follow the next steps:

1. When creating AB Test add an Empty condition with false value, automatically making AB Test unavailable for all players.

   

2. Invoke the following code to start AB Test and assign a variant manually:

   var allABTest = DataManager.SmartObjects.Analytics.ABTests;
   var myTest = allABTest[0];//find the AB Test you want to start for this player
   var myVariant = myTest.Variants[0];//find the variant you need
   
   LiveOps.ABTests.StartABTestManually(myTest, myVariant);