Evolving Experiments and Analytics: Segmentation, Experiment Targeting, Early Access Program, and More

Roblox has upgraded Experiments and Analytics for developers Developers can now target experiments to specific player groups, serve different config values based on player type, and get alerts if an experiment starts hurting key metrics. Read more: https://t.co/N9SSozNjqQ https://t.co/GGzvI6n0Cw

This topic was automatically opened after 10 minutes.

This is a fantastic update to Experiments and Analytics. Conditional Configs and Experiment Targeting in particular give creators significantly more control over how changes are rolled out. Being able to tailor configurations and experiments around new vs. returning users, engagement level, acquisit

Our studio’s (and many others’) current workflow for pushing out updates is: test in a private testing place only accessible to devs & testers during an update, put the game into “maintenance” by setting the public game to private and only letting testers in let testers playtest the public game

That’s actually a pretty interesting use-case. I’m wondering though, what made you host testing on the same universe as the main game?

Will the updated segments be made available in AnalyticsService:GetPlayerSegmentsAsync

theyre not the same universe; theyre separate game ids. having a testing place is so we can iterate rapidly, while the maintenance period is to make sure theres no drift or critical bugs caused by prod data when loading in.

Oh yeah sorry, I meant for the maintenence period specifically. Pyseph: make sure theres no drift or critical bugs caused by prod data when loading in. Have you considered doing it in a private place under the same universe? It still kind of messes up some universe metrics like Session time by place

Why not use feature flags? Or, slightly riskier, publish your update and then simply monitor game health and quickly revert if anything looks wrong? Taking your game offline to test every update seems like a very strange choice.

Amazing update! By the way if any staff seeing this post: Stolen experience

Holy.. it is just that easy.. I would thought this would involve more extra steps but nope. It’s incredibly easy to find and not difficult to setup and with very useful metrics. This is a very, very good addition.

why are Custom Events no longer a tab on analytics? Is there a reason why it was removed, and is it being added back?

There’s a myriad reasons involved behind this decision that I don’t have the energy to explain in detail

Yeah, that’s the same question I had. I feel like most games have separate places for testing (like as Bee Swarm Simulator for instance) to avoid messing around with the main game