module Bakelite::Docs::B_ARCHITECTURE::CHUNKING_AND_IO

Overview

Chunking and Streaming IO

Bakelite achieves constant O(chunk_size) memory usage during file reading by partitioning large files into discrete, individually compressed chunks and streaming them on demand using Bakelite::FileIO.

Executive Summary & Key Topics

Topic Method / Anchor Description
Partitioning and Compression .topic_01_chunking_mechanism How files are split and verified.
Bakelite::FileIO Semantics .topic_02_streaming_file_io Transparent seeking and reading through standard Crystal IO.

Defined in:

bakelite/docs/b_architecture/chunking_and_io.cr

Class Method Summary

Class Method Detail

def self.topic_01_chunking_mechanism : Nil #

Partitioning and Compression: How files are split and verified.

When a file is stored using store, Bakelite's Chunker divides the raw stream into fixed segments (default 64KB). Each chunk contains:

  • 64-bit absolute file offset
  • 32-bit compressed length
  • 32-bit uncompressed length
  • 32-bit CRC32 checksum
  • Compressed byte payload (Deflate, Gzip, Zlib, or raw)

During verification (bakelite verify), each chunk's CRC32 checksum is checked against the decompressed bytes, ensuring bit-perfect integrity.


def self.topic_02_streaming_file_io : Nil #

Bakelite::FileIO Semantics: Transparent seeking and reading through standard Crystal IO.

Bakelite::FileIO inherits directly from IO, enabling it to be passed into standard Crystal parsers (JSON.parse, YAML.parse, image loaders, XML parsers).

Key IO operations:

  • read(slice): Reads arbitrary byte lengths across multiple chunk boundaries.
  • seek(offset, whence): Supports IO::Seek::Set, IO::Seek::Current, and IO::Seek::End.
  • Active chunk caching: Only decompresses a chunk when the read pointer lands inside it, caching the active chunk for sequential reads.