module Bakelite::Chunker

Overview

Utility module for splitting large files into compressed chunks.

Defined in:

bakelite/chunker.cr

Class Method Summary

Class Method Detail

def self.chunk_data(data : Bytes, chunk_size : UInt32 = 65536_u32, compression : CompressionType = CompressionType::Deflate, base_offset : Int64 = 0_i64) : Array(Chunk) #

Splits input byte slice into an array of compressed chunks of chunk_size.


def self.chunk_stream(io : IO, total_size : Int64, chunk_size : UInt32 = 65536_u32, compression : CompressionType = CompressionType::Deflate, base_offset : Int64 = 0_i64, &block : Chunk -> ) : Tuple(Array(Chunk), UInt32) #

Reads an arbitrary IO stream in chunks of chunk_size and compresses each chunk. Yields each chunk to the provided block as soon as it is generated, maintaining strict O(chunk_size) peak RAM usage even for multi-gigabyte streams. Returns a Tuple containing the Array of Chunk descriptors and the total uncompressed CRC32.


def self.chunk_stream(io : IO, total_size : Int64, chunk_size : UInt32 = 65536_u32, compression : CompressionType = CompressionType::Deflate, base_offset : Int64 = 0_i64) : Tuple(Array(Chunk), UInt32) #

Overload for chunk_stream when block is not provided


def self.compress_slice(slice : Bytes, compression : CompressionType) : Bytes #

Compresses a raw byte slice using the requested algorithm.