Видеохостинг

Проектирование платформы для загрузки, хранения и стриминга видео. Ключевые инженерные задачи: транскодирование, adaptive bitrate streaming, CDN.

Шаг 1: Требования

Функциональные требования

1. Загрузка видео (до 10 GB)
2. Стриминг видео (adaptive bitrate)
3. Поддержка разных разрешений (360p, 480p, 720p, 1080p, 4K)
4. Thumbnails генерация
5. Поиск и рекомендации видео
6. Комментарии, лайки, просмотры

Нефункциональные требования

1. Минимальный buffering при просмотре
2. Быстрый старт воспроизведения (< 2 секунды)
3. Масштабирование до 1B DAU (просмотр), 10M DAU (загрузка)
4. Высокая доступность стриминга (99.99%)

Шаг 2: Оценка нагрузки

Шаг 3: High-Level архитектура

┌──────────┐    ┌────────────────┐    ┌─────────────────────────────────┐
│  Client  │───>│  Upload API    │───>│  Upload Service                 │
│ (Upload) │    │                │    │  ┌──────────┐  ┌──────────────┐ │
└──────────┘    └────────────────┘    │  │ Chunked  │─>│  Object      │ │
                                      │  │ Upload   │  │  Storage     │ │
                                      │  └──────────┘  └──────┬───────┘ │
                                      └────────────────────────┼─────────┘
                                                               │
                                                    ┌──────────▼──────────┐
                                                    │  Transcoding Queue  │
                                                    │  (Kafka)            │
                                                    └──────────┬──────────┘
                                                               │
                                                    ┌──────────▼──────────┐
                                                    │  Transcoder Workers │
                                                    │  (FFmpeg)           │
                                                    └──────────┬──────────┘
                                                               │
┌──────────┐    ┌────────────────┐    ┌────────────────────────▼──────────┐
│  Client  │───>│  CDN           │───>│  Video Storage                   │
│ (Watch)  │    │  (Edge Cache)  │    │  (S3-compatible)                 │
└──────────┘    └────────────────┘    └──────────────────────────────────┘

Шаг 4: Детальный дизайн

4.1 Upload Service

<?php

declare(strict_types=1);

final class VideoUploadService
{
    private const MAX_FILE_SIZE = 10 * 1024 * 1024 * 1024; // 10 GB
    private const CHUNK_SIZE = 5 * 1024 * 1024;            // 5 MB

    public function __construct(
        private readonly ObjectStorage $storage,
        private readonly VideoRepository $repository,
        private readonly QueuePublisher $queue,
        private readonly ThumbnailGenerator $thumbnails,
    ) {}

    /**
     * Initiate resumable upload
     */
    public function initiateUpload(UploadRequest $request): UploadSession
    {
        // Validate
        if ($request->fileSize > self::MAX_FILE_SIZE) {
            throw new FileTooLargeException('Maximum file size is 10 GB');
        }

        $videoId = bin2hex(random_bytes(16));
        $uploadId = $this->storage->initiateMultipartUpload(
            bucket: 'raw-videos',
            key: "{$videoId}/original",
        );

        // Create video record
        $this->repository->create(
            id: $videoId,
            title: $request->title,
            userId: $request->userId,
            status: VideoStatus::UPLOADING,
        );

        return new UploadSession(
            videoId: $videoId,
            uploadId: $uploadId,
            chunkSize: self::CHUNK_SIZE,
            totalChunks: (int) ceil($request->fileSize / self::CHUNK_SIZE),
        );
    }

    /**
     * Upload a chunk
     */
    public function uploadChunk(
        string $uploadId,
        int $partNumber,
        string $data,
    ): void {
        $this->storage->uploadPart(
            uploadId: $uploadId,
            partNumber: $partNumber,
            data: $data,
        );
    }

    /**
     * Finalize upload and trigger transcoding
     */
    public function finalizeUpload(string $videoId, string $uploadId): void
    {
        // Complete multipart upload
        $this->storage->completeMultipartUpload($uploadId);

        // Update status
        $this->repository->updateStatus($videoId, VideoStatus::PROCESSING);

        // Publish transcoding job
        $this->queue->publish('video.transcode', [
            'video_id' => $videoId,
            'source_key' => "{$videoId}/original",
            'source_bucket' => 'raw-videos',
        ]);

        // Generate thumbnails (parallel job)
        $this->queue->publish('video.thumbnails', [
            'video_id' => $videoId,
            'source_key' => "{$videoId}/original",
        ]);
    }
}

4.2 Transcoding Pipeline

<?php

declare(strict_types=1);

final class TranscodingWorker
{
    private const PROFILES = [
        '360p'  => ['width' => 640,  'height' => 360,  'bitrate' => '800k',  'audio' => '96k'],
        '480p'  => ['width' => 854,  'height' => 480,  'bitrate' => '1400k', 'audio' => '128k'],
        '720p'  => ['width' => 1280, 'height' => 720,  'bitrate' => '2800k', 'audio' => '128k'],
        '1080p' => ['width' => 1920, 'height' => 1080, 'bitrate' => '5000k', 'audio' => '192k'],
    ];

    private const SEGMENT_DURATION = 6; // seconds for HLS

    public function __construct(
        private readonly ObjectStorage $storage,
        private readonly VideoRepository $repository,
        private readonly string $tempDir,
    ) {}

    public function transcode(array $job): void
    {
        $videoId = $job['video_id'];
        $sourceKey = $job['source_key'];

        try {
            // 1. Download source video
            $localSource = $this->downloadSource($sourceKey, $job['source_bucket']);

            // 2. Probe video metadata
            $metadata = $this->probeVideo($localSource);

            // 3. Transcode to each profile
            $manifests = [];
            foreach (self::PROFILES as $quality => $profile) {
                // Skip if source is lower quality
                if ($metadata['height'] < $profile['height']) {
                    continue;
                }

                $outputDir = "{$this->tempDir}/{$videoId}/{$quality}";
                mkdir($outputDir, 0755, true);

                // Transcode to HLS segments
                $this->transcodeToHls($localSource, $outputDir, $profile, $quality);

                // Upload segments to storage
                $this->uploadSegments($videoId, $quality, $outputDir);

                $manifests[$quality] = "{$videoId}/hls/{$quality}/playlist.m3u8";
            }

            // 4. Generate master playlist
            $masterPlaylist = $this->generateMasterPlaylist($manifests);
            $this->storage->put(
                'processed-videos',
                "{$videoId}/hls/master.m3u8",
                $masterPlaylist,
            );

            // 5. Update video status
            $this->repository->updateStatus($videoId, VideoStatus::READY);
            $this->repository->updateMetadata($videoId, [
                'duration' => $metadata['duration'],
                'width' => $metadata['width'],
                'height' => $metadata['height'],
                'formats' => array_keys($manifests),
            ]);

        } catch (\Throwable $e) {
            $this->repository->updateStatus($videoId, VideoStatus::FAILED);
            throw $e;
        } finally {
            // Cleanup temp files
            $this->cleanup("{$this->tempDir}/{$videoId}");
        }
    }

    private function transcodeToHls(
        string $input,
        string $outputDir,
        array $profile,
        string $quality,
    ): void {
        $cmd = sprintf(
            'ffmpeg -i %s -vf scale=%d:%d -b:v %s -b:a %s '
            . '-hls_time %d -hls_list_size 0 -hls_segment_filename '
            . '%s/segment_%%04d.ts %s/playlist.m3u8 2>&1',
            escapeshellarg($input),
            $profile['width'],
            $profile['height'],
            $profile['bitrate'],
            $profile['audio'],
            self::SEGMENT_DURATION,
            $outputDir,
            $outputDir,
        );

        exec($cmd, $output, $exitCode);

        if ($exitCode !== 0) {
            throw new TranscodingException(
                "FFmpeg failed for {$quality}: " . implode("\n", $output)
            );
        }
    }

    private function generateMasterPlaylist(array $manifests): string
    {
        $lines = ['#EXTM3U'];

        foreach ($manifests as $quality => $path) {
            $profile = self::PROFILES[$quality];
            $bandwidth = (int) str_replace('k', '000', $profile['bitrate']);

            $lines[] = sprintf(
                '#EXT-X-STREAM-INF:BANDWIDTH=%d,RESOLUTION=%dx%d',
                $bandwidth,
                $profile['width'],
                $profile['height'],
            );
            $lines[] = "{$quality}/playlist.m3u8";
        }

        return implode("\n", $lines);
    }

    private function probeVideo(string $path): array
    {
        $cmd = sprintf(
            'ffprobe -v quiet -print_format json -show_format -show_streams %s',
            escapeshellarg($path),
        );

        exec($cmd, $output);
        $data = json_decode(implode('', $output), true);

        $videoStream = null;
        foreach ($data['streams'] as $stream) {
            if ($stream['codec_type'] === 'video') {
                $videoStream = $stream;
                break;
            }
        }

        return [
            'duration' => (float) ($data['format']['duration'] ?? 0),
            'width' => $videoStream['width'] ?? 0,
            'height' => $videoStream['height'] ?? 0,
            'codec' => $videoStream['codec_name'] ?? 'unknown',
        ];
    }
}

4.3 Streaming Service

<?php

declare(strict_types=1);

final class VideoStreamingService
{
    public function __construct(
        private readonly VideoRepository $repository,
        private readonly CdnUrlGenerator $cdn,
        private readonly ViewTracker $views,
    ) {}

    /**
     * Get streaming URL for a video
     */
    public function getStreamUrl(string $videoId, string $userId): StreamInfo
    {
        $video = $this->repository->find($videoId);

        if ($video === null || $video->status !== VideoStatus::READY) {
            throw new VideoNotAvailableException("Video not available: {$videoId}");
        }

        // Generate CDN-signed URL
        $masterUrl = $this->cdn->generateSignedUrl(
            path: "{$videoId}/hls/master.m3u8",
            expiresIn: 3600 * 4, // 4 hours
        );

        // Track view (async)
        $this->views->trackView($videoId, $userId);

        return new StreamInfo(
            videoId: $videoId,
            manifestUrl: $masterUrl,
            duration: $video->metadata['duration'],
            availableQualities: $video->metadata['formats'],
        );
    }
}

Шаг 5: Adaptive Bitrate Streaming

Client network speed detection:
  Fast (>5 Mbps)   -> 1080p
  Medium (2-5 Mbps) -> 720p
  Slow (1-2 Mbps)   -> 480p
  Very slow (<1)     -> 360p

HLS segment switching:
  Segment 1: 1080p (network good)
  Segment 2: 1080p
  Segment 3: 720p  (network degraded, automatic switch)
  Segment 4: 720p
  Segment 5: 1080p (network recovered)

Шаг 6: Масштабирование

Возможные вопросы интервьюера

1. Почему HLS, а не прогрессивный download?

   • Adaptive bitrate: автоматическое переключение качества
   • Начало просмотра без ожидания полной загрузки
   • Эффективное кэширование сегментов на CDN

2. Как оптимизировать стоимость хранения?

   • Удаление raw video после транскодинга
   • Lifecycle policies: hot (SSD) -> warm (HDD) -> archive (Glacier)
   • Удаление редко просматриваемых разрешений

3. Как обеспечить copyright protection?

   • DRM (Digital Rights Management): Widevine, FairPlay
   • Content fingerprinting (Content ID)
   • Signed URLs с ограниченным TTL

4. Как ускорить time-to-playback?

   • Preload первых нескольких сегментов
   • Start с низкого качества, затем upscale
   • CDN warm-up для популярных видео