Skip to content

A/B Tests

A/B testing allows developers to run a controlled experiment between two or more versions of something in their game to determine which one is more effective. Your audience is segmented into the control group (current performance – no changes) and your test group.

A/B testing is a great way to find the best pricing and game difficulty or test any of your hypotheses.

Before using, all A/B Tests must be added to the table.

Parameter Description
Name The name of the Test.
Groups Each user gets one random group.
Target Audience The portion of your total users participating in the test.
Users Type Defines the kind of users to target the test: New, Old, or All. If the app is opened for the first time, the user is considered a new one until he restarts the app.
Concurrent Test Such test ignore the fact that some other tests can be active. All concurrent test will be activated for your users and won't affect Non-concurrent tests.
Conditions Conditions that are required to start the A/B Test. After A/B Test starts for a player, it won't be stopped for him even if the conditions turn to False.
PreInitLaunch Defines if the A/B Test can be launched during PreInit and before Balancy OnReadyCallback is executed. Read more here

Screenshot

Balancy ensures that each user is not simultaneously participating in two A/B Tests (unless you are running Concurrent Tests). If a user has a running A/B Test, he is skipping all other A/B Tests, even if they target 100% of the audience. Once the A/B Test is finished, the user can join a new one.

A/B tests don't need priority. When the game starts, and a user doesn't have any running A/B Tests, the user collects all Active A/B Tests and picks just one of them, according to the chance.

There are currently 3 ways to work with A/B testing:

  1. Conditions
  2. Use the A/B Test Node in Visual Scripting

    Screenshot

  3. Make your logic

Fast A/B Testing

  1. Find any record in Balancy you want to run A/B Test on.
  2. Click on the 3 dots next to the Document Id, and select Create A/B Test.

    Screenshot

  3. Select parameter you want to A/B Test:

    Screenshot

  4. Create A/B Test.

A/B Test lifecycle

The main rules for each user individually:

  • Only one non-concurrent Test can be active at the same time, unless you activate Tests manually.
  • If a Test passed its Condition and wasn't activated due to the User Type or Audience Size, the test will be ignored by this users forever.
  • Concurrent Tests exist in a separate world.

Screenshot

Finish A/B-test

When you want to finish A/B Test and apply results.

  1. Open A/B Test info modal.
  2. Press "Finish Collecting Users".

    Screenshot

  3. Stop A/B Test.

  4. Chose variant you want to roll out to all users.
  5. Save.

Now all overrides for this variant will be applied to documents parameters itself for all users.

You still need to deploy changes after that, so users will see new values.

!!! warning "Deploy and migration"! You have to do all changes on dev environment and then migrate to stage/production. Read launch check list for better understanding data migration process.

A/B Test Analysis

Balancy automatically tracks the performance of A/B Tests and provides you with analytics.

  1. Find the A/B Test you need and expand its view:

    Screenshot

  2. Analyse the performance of each variants:

    Screenshot

Section for programmers

After the initialization of Balancy package, you can get all current A/B tests and their variants for the user.

var allTests = Balancy.LiveOps.ABTests.GetAllTests();
Debug.LogWarning("All Tests: " + allTests.Count);
foreach (var test in allTests)
    Debug.Log("Name = " + test.AbTest.Name + "; Variant = " + test.Variant.Name + "; isFinished = " + test.Finished);

//You can manually launch a test and place the user in the specified variant:
LiveOps.ABTests.StartABTestManually(test, variant);

You can send all the A/B Tests' data to your own BI or analytics system:

namespace Balancy
{
    public class SmartObjectsEventsExample : ISmartObjectsEvents
    {
        ...

        public void OnAbTestStarted(LiveOps.ABTests.TestData abTestInfo)
        {
            Debug.Log("=> OnAbTestStarted: " + abTestInfo?.AbTest?.Name + " ; Variant = " + abTestInfo?.Variant?.Name);
            //TODO Send AB Test info to your custom analytics
        }

        public void OnAbTestEnded(LiveOps.ABTests.TestData abTestInfo)
        {
            Debug.Log("=> OnAbTestEnded: " + abTestInfo?.AbTest?.Name);
            //TODO Send AB Test info to your custom analytics
        }
    }
}

External Tests

If you want to manually launch A/B Tests and define the variants assigned to each player using your algorithms or any 3rd party solution, follow the next steps:

  1. When creating A/B Test add an Empty condition with false value, automatically making A/B Test unavailable for all players.

    Screenshot

  2. Invoke the following code to start A/B Test and assign a variant manually:

    var allABTest = DataManager.SmartObjects.Analytics.ABTests;
    var myTest = allABTest[0];//find the AB Test you want to start for this player
    var myVariant = myTest.Variants[0];//find the variant you need
    
    LiveOps.ABTests.StartABTestManually(myTest, myVariant);