A/B Testing платформа

A/B Testing платформа позволяет проводить контролируемые эксперименты для принятия data-driven решений о продуктовых изменениях.

Шаг 1: Требования

Функциональные требования

1. Создание экспериментов с несколькими вариантами (A, B, C...)
2. Детерминированное распределение пользователей по вариантам
3. Несколько одновременных экспериментов без конфликтов
4. Real-time и batch сбор метрик
5. Статистический анализ результатов
6. Градуальная раскатка (1% -> 5% -> 50% -> 100%)

Нефункциональные требования

1. Latency определения варианта < 5ms
2. Consistency: пользователь всегда видит один вариант
3. Масштабирование до 100K+ QPS для assignment
4. Точность метрик: статистическая значимость

Шаг 2: High-Level архитектура

┌──────────┐     ┌──────────────────┐     ┌──────────────────────────┐
│  Client  │────>│  Assignment      │────>│  Experiment Config       │
│  (App)   │     │  Service         │     │  (Feature Flags)         │
└──────────┘     └────────┬─────────┘     └──────────────────────────┘
                          │
               ┌──────────▼──────────┐
               │  Event Tracking     │
               │  (Kafka)            │
               └──────────┬──────────┘
                          │
               ┌──────────▼──────────┐
               │  Analytics Pipeline │
               │  (Flink/Spark)      │
               └──────────┬──────────┘
                          │
               ┌──────────▼──────────┐
               │  Results Dashboard  │
               │  (Statistical       │
               │   Analysis)         │
               └─────────────────────┘

Шаг 3: Детальный дизайн

3.1 Детерминированное назначение варианта

<?php

declare(strict_types=1);

final class ExperimentAssigner
{
    public function __construct(
        private readonly ExperimentConfigStore $configStore,
        private readonly \Redis $redis,
    ) {}

    /**
     * Assign user to experiment variant
     * MUST be deterministic: same user always gets same variant
     */
    public function assign(string $userId, string $experimentId): AssignmentResult
    {
        $config = $this->configStore->getExperiment($experimentId);

        if ($config === null || !$config->isActive) {
            return AssignmentResult::notInExperiment();
        }

        // 1. Check if user is in the experiment's target audience
        if (!$this->isEligible($userId, $config)) {
            return AssignmentResult::notEligible();
        }

        // 2. Deterministic hash-based assignment
        $bucket = $this->hashToBucket($userId, $experimentId);

        // 3. Map bucket to variant based on traffic allocation
        $variant = $this->bucketToVariant($bucket, $config);

        if ($variant === null) {
            return AssignmentResult::notInExperiment(); // Not in traffic allocation
        }

        // 4. Log assignment event
        $this->logAssignment($userId, $experimentId, $variant);

        return new AssignmentResult(
            inExperiment: true,
            variant: $variant,
            experimentId: $experimentId,
        );
    }

    /**
     * Deterministic hashing: user always gets same bucket
     * Using murmur3-like approach for uniform distribution
     */
    private function hashToBucket(string $userId, string $experimentId): int
    {
        // Combine user ID and experiment ID for independence between experiments
        $seed = "{$experimentId}:{$userId}";
        $hash = crc32($seed);

        // Map to 0-9999 range (10000 buckets)
        return abs($hash) % 10000;
    }

    /**
     * Map bucket number to experiment variant
     *
     * Example: 50/50 split
     *   buckets 0-4999    -> variant "control"
     *   buckets 5000-9999 -> variant "treatment"
     *
     * Example: 80/10/10 split
     *   buckets 0-7999    -> variant "control"
     *   buckets 8000-8999 -> variant "treatment_a"
     *   buckets 9000-9999 -> variant "treatment_b"
     */
    private function bucketToVariant(int $bucket, ExperimentConfig $config): ?string
    {
        $cumulative = 0;

        foreach ($config->variants as $variant) {
            $cumulative += $variant->trafficPercentage * 100; // Convert to bucket range

            if ($bucket < $cumulative) {
                return $variant->name;
            }
        }

        return null; // Bucket not allocated to any variant
    }

    private function isEligible(string $userId, ExperimentConfig $config): bool
    {
        // Check targeting rules (country, platform, user segment, etc.)
        if ($config->targetingRules === null) {
            return true;
        }

        // Example: whitelist/blacklist, percentage rollout
        return true; // Simplified
    }

    private function logAssignment(string $userId, string $experimentId, string $variant): void
    {
        // Async logging for analytics
        $this->redis->xAdd('experiment:assignments', '*', [
            'user_id' => $userId,
            'experiment_id' => $experimentId,
            'variant' => $variant,
            'timestamp' => microtime(true),
        ]);
    }
}

3.2 Experiment Configuration

<?php

declare(strict_types=1);

final readonly class ExperimentConfig
{
    public function __construct(
        public string $id,
        public string $name,
        public bool $isActive,
        /** @var ExperimentVariant[] */
        public array $variants,
        public ?TargetingRules $targetingRules,
        public string $layer, // Isolation layer for mutual exclusion
        public \DateTimeImmutable $startDate,
        public ?\DateTimeImmutable $endDate,
    ) {}
}

final readonly class ExperimentVariant
{
    public function __construct(
        public string $name,         // "control", "treatment_a"
        public float $trafficPercentage, // 0.0 - 1.0
        public array $parameters,    // Feature flag values for this variant
    ) {}
}

final class ExperimentConfigStore
{
    public function __construct(
        private readonly \PDO $db,
        private readonly \Redis $redis,
    ) {}

    public function getExperiment(string $id): ?ExperimentConfig
    {
        // Cache for performance (< 5ms requirement)
        $cached = $this->redis->get("experiment:{$id}");

        if ($cached !== false) {
            return unserialize($cached);
        }

        $stmt = $this->db->prepare(
            'SELECT e.*, json_agg(v.*) as variants
             FROM experiments e
             JOIN experiment_variants v ON v.experiment_id = e.id
             WHERE e.id = :id
             GROUP BY e.id'
        );
        $stmt->execute(['id' => $id]);
        $row = $stmt->fetch(\PDO::FETCH_ASSOC);

        if ($row === false) {
            return null;
        }

        $config = $this->mapToConfig($row);
        $this->redis->setex("experiment:{$id}", 60, serialize($config));

        return $config;
    }

    /**
     * Get all active experiments for a layer
     */
    public function getActiveExperiments(string $layer = 'default'): array
    {
        $cacheKey = "experiments:active:{$layer}";
        $cached = $this->redis->get($cacheKey);

        if ($cached !== false) {
            return unserialize($cached);
        }

        $stmt = $this->db->prepare(
            'SELECT e.id FROM experiments e
             WHERE e.is_active = true AND e.layer = :layer
             AND e.start_date <= now()
             AND (e.end_date IS NULL OR e.end_date > now())'
        );
        $stmt->execute(['layer' => $layer]);

        $ids = $stmt->fetchAll(\PDO::FETCH_COLUMN);
        $this->redis->setex($cacheKey, 30, serialize($ids));

        return $ids;
    }
}

3.3 Feature Flag Integration

<?php

declare(strict_types=1);

final class FeatureFlagService
{
    public function __construct(
        private readonly ExperimentAssigner $assigner,
        private readonly ExperimentConfigStore $configStore,
    ) {}

    /**
     * Get feature flag value for a user (experiment-aware)
     */
    public function getValue(string $userId, string $flagName, mixed $default = null): mixed
    {
        // Find active experiment for this flag
        $experiments = $this->configStore->getActiveExperiments();

        foreach ($experiments as $experimentId) {
            $config = $this->configStore->getExperiment($experimentId);

            // Check if any variant controls this flag
            foreach ($config->variants as $variant) {
                if (isset($variant->parameters[$flagName])) {
                    // Assign user to variant
                    $assignment = $this->assigner->assign($userId, $experimentId);

                    if ($assignment->inExperiment && $assignment->variant === $variant->name) {
                        return $variant->parameters[$flagName];
                    }
                }
            }
        }

        return $default;
    }

    /**
     * Check boolean feature flag
     */
    public function isEnabled(string $userId, string $flagName): bool
    {
        return (bool) $this->getValue($userId, $flagName, false);
    }
}

3.4 Event Tracking и Statistical Analysis

<?php

declare(strict_types=1);

final class ExperimentAnalyzer
{
    /**
     * Calculate statistical significance using Z-test
     */
    public function analyzeConversionRate(
        int $controlSamples,
        int $controlConversions,
        int $treatmentSamples,
        int $treatmentConversions,
    ): AnalysisResult {
        $controlRate = $controlConversions / max(1, $controlSamples);
        $treatmentRate = $treatmentConversions / max(1, $treatmentSamples);

        // Pooled proportion
        $pooled = ($controlConversions + $treatmentConversions)
            / ($controlSamples + $treatmentSamples);

        // Standard error
        $se = sqrt($pooled * (1 - $pooled) * (1/$controlSamples + 1/$treatmentSamples));

        // Z-score
        $zScore = $se > 0 ? ($treatmentRate - $controlRate) / $se : 0;

        // P-value (two-tailed, approximation)
        $pValue = $this->calculatePValue($zScore);

        // Relative lift
        $lift = $controlRate > 0 ? ($treatmentRate - $controlRate) / $controlRate : 0;

        // Confidence interval for the difference (95%)
        $marginOfError = 1.96 * $se;

        return new AnalysisResult(
            controlRate: round($controlRate, 4),
            treatmentRate: round($treatmentRate, 4),
            lift: round($lift * 100, 2),
            pValue: round($pValue, 4),
            isSignificant: $pValue < 0.05,
            confidenceInterval: [
                round(($treatmentRate - $controlRate) - $marginOfError, 4),
                round(($treatmentRate - $controlRate) + $marginOfError, 4),
            ],
            controlSamples: $controlSamples,
            treatmentSamples: $treatmentSamples,
        );
    }

    /**
     * Minimum sample size calculator
     */
    public function minimumSampleSize(
        float $baselineRate,
        float $minimumDetectableEffect, // e.g., 0.02 for 2% lift
        float $significanceLevel = 0.05,
        float $power = 0.80,
    ): int {
        // Z-scores for significance and power
        $zAlpha = 1.96;  // 95% confidence
        $zBeta = 0.84;   // 80% power

        $p1 = $baselineRate;
        $p2 = $baselineRate + $minimumDetectableEffect;

        $n = (($zAlpha * sqrt(2 * $p1 * (1 - $p1))
            + $zBeta * sqrt($p1 * (1 - $p1) + $p2 * (1 - $p2))) ** 2)
            / (($p2 - $p1) ** 2);

        return (int) ceil($n);
    }

    private function calculatePValue(float $z): float
    {
        // Approximation of complementary error function
        $absZ = abs($z);
        $t = 1.0 / (1.0 + 0.2316419 * $absZ);
        $d = 0.3989422804 * exp(-$absZ * $absZ / 2.0);
        $p = $d * $t * (0.3193815 + $t * (-0.3565638
            + $t * (1.781478 + $t * (-1.821256 + $t * 1.330274))));

        return 2.0 * $p; // Two-tailed
    }
}

final readonly class AnalysisResult
{
    public function __construct(
        public float $controlRate,
        public float $treatmentRate,
        public float $lift,
        public float $pValue,
        public bool $isSignificant,
        public array $confidenceInterval,
        public int $controlSamples,
        public int $treatmentSamples,
    ) {}
}

Шаг 4: Isolation Layers

Для одновременных экспериментов без конфликтов:

Layer "UI":        Experiment A (button color)
                   Experiment B (layout)

Layer "Backend":   Experiment C (algorithm)
                   Experiment D (cache strategy)

User hashing is independent per experiment,
but experiments in same layer are mutually exclusive.

Шаг 5: Градуальная раскатка

Возможные вопросы интервьюера

1. Почему детерминированное хеширование?

   • Пользователь всегда видит один вариант (consistency)
   • Не нужно хранить assignment -- вычисляется на лету
   • Перезапуск сервиса не меняет распределение

2. Как избежать sample ratio mismatch?

   • Мониторинг ratio между вариантами
   • Alert если отклонение > 1%
   • Проверка хеш-функции на равномерность

3. Как проводить A/B тесты на малом трафике?

   • Bayesian analysis вместо frequentist
   • Sequential testing
   • Multi-armed bandit для optimization

4. Как обрабатывать network effects?

   • Cluster randomization
   • Geo-based splitting
   • Synthetic control groups