module Bakelite::Docs::E_SCALE::LARGE_ASSET_STREAMING

Overview

Large Asset Streaming and Scale (100MB & 1GB)

Bakelite is engineered from the ground up to handle massive assets—ranging from 100MB video/audio files to multi-gigabyte open-world game maps and terrain databases—with constant O(chunk_size) heap memory usage.

Executive Summary & Key Topics

Topic Method / Anchor Description
Constant Memory Bounds .topic_01_memory_bounds How Bakelite guarantees fixed RAM consumption regardless of file scale.
100MB and 1GB Scale Verification .topic_02_scale_testing Deterministic synthetic testing apparatus and automated CI benchmarks.

Defined in:

bakelite/docs/e_scale/large_asset_streaming.cr

Class Method Summary

Class Method Detail

def self.topic_01_memory_bounds : Nil #

Constant Memory Bounds: How Bakelite guarantees fixed RAM consumption regardless of file scale.

Traditional embedding solutions load entire files into memory buffers. When dealing with large assets (such as 100MB video clips or 1GB texture packs), this leads to drastic memory spikes and out-of-memory crashes on resource-constrained platforms.

Bakelite's store primitive and Bakelite::FileIO solve this through active chunk sliding buffers:

  • Files are partitioned into discrete compressed chunks (e.g. 64KB).
  • At any point in time, only the single active chunk containing the current read position resides uncompressed in RAM.
  • Sequential reads across a 1GB file execute in continuous 64KB slices with zero heap allocations.
  • Random seeking across chunk boundaries simply invalidates the active chunk and loads the target chunk, requiring only O(log N) binary search iterations over 64-bit offsets.

def self.topic_02_scale_testing : Nil #

100MB and 1GB Scale Verification: Deterministic synthetic testing apparatus and automated CI benchmarks.

Bakelite includes dedicated automated test suites (spec/scale_spec.cr) verifying:

  • 100MB Real Disk Test: Generates a 104,857,600-byte file on disk, stream-packs 1,600 compressed chunks into a .bkl container, performs arbitrary seeks across 25MB, 50MB, 75MB, and near-EOF boundaries, and verifies 128KB multi-chunk spanning reads.
  • 1GB Scale Verification: Tests 1,073,741,824 bytes spanning 16,384 chunks, proving 64-bit offset integrity, binary container trailers at EOF, and deep seeks to 512MB and 768MB with bit-perfect chunk marker verification.
  • Compile-Time Guard: Files exceeding 10MB passed to bake trigger compile-time warnings advising the developer to switch to store to prevent binary executable bloat.