Deterministic Data Scanning for Resource-Constrained Systems with Cribra

Sensitive-data detection is often framed as a pattern-matching problem. In practice, the difficult part is execution. A service may need to inspect data while it is arriving, while a document is being edited, or under a memory budget much smaller than the input. Each situation changes what the engine may retain, when it may report a result, and what happens when a limit is reached.

Cribra is an embeddable, synchronous Rust engine for deterministic sensitive-data detection, validation, classification, reporting, and explicit transformation. The central design question is not only which rules it can evaluate, but how it makes execution predictable when the source is large, fragmented, or changing. It answers that question with explicit contracts: a semantic reference, admission rules, bounded resource accounting, and failure behavior callers can reason about.

Determinism starts with the execution model#

Cribra’s core performs no I/O and no network access. It does not reach out to a filesystem, database, or remote service while evaluating a source. The Rust core also forbids unsafe code. These are more than implementation preferences. They remove entire categories of behavior that would otherwise make results depend on the environment rather than on the input and configuration.

A synchronous, I/O-free core is easier to reason about. Execution does not interleave with an event loop or background task. There is no hidden retry, network timeout, or dependence on a remote rules service. The engine receives a source and a configuration, performs its work, and returns a result. That is a useful foundation for deterministic detection: the execution path is explicit, and the boundary between the engine and the host application is visible.

The core’s scope is also explicit. It detects sensitive data, validates candidates, classifies findings, produces reports, and can perform transformations when the caller asks for them. Transformation is not an implicit side effect of scanning. It is a separate, explicit operation, which matters when a detection pass must not alter the data being inspected.

Within that model, whole-source scanning is the semantic reference. Behavior on a complete source defines the result semantics against which other execution modes are understood. Streaming and incremental operation are not separate definitions of correctness. They are ways of executing while preserving the contract established by the whole-source path.

That reference matters for resource-constrained systems. It is tempting to treat streaming or incremental mode as a different product with weaker guarantees. Cribra instead treats the execution mode as an operational choice. The question becomes whether a source can be processed in that mode, under those budgets, without changing the meaning of the result.

Streaming without partial answers#

Streaming in Cribra is synchronous and opt-in. The caller supplies arbitrary ordered byte chunks, and the engine processes them as a stream. “Arbitrary” matters. A chunk boundary is not guaranteed to fall on a convenient character boundary. A UTF-8 scalar may be split across two chunks. A CRLF sequence may also be split, with the carriage return at the end of one chunk and the line feed at the start of the next.

Those are not edge cases for the caller to fix. They are part of the streaming contract. A correct implementation must carry enough state across chunks to preserve the same semantic view of the source that a whole-source scan would have. Streaming is therefore not a loop around a per-chunk function. The engine treats the concatenation of chunks as the source, not as a sequence of independent inputs.

The project README’s streaming example makes this concrete: a literal rule for SECRET receives TOKEN=SEC and RET as two chunks of one stream:

use cribra::{Rule, Scanner, Severity, StreamLimits};

let scanner = Scanner::builder()
    .rule(Rule::literal("credential", "SECRET", Severity::High))
    .build()?;

let limits = StreamLimits::default()
    .with_max_candidates(Some(10_000))
    .with_max_retained_results(Some(1_000));

let mut session = scanner.stream_with_limits(limits)?;
session.push(b"TOKEN=SEC")?;
session.push(b"RET")?;
let report = session.finish()?;

Both pushes feed one session: the first chunk ends with SEC, the second begins with RET, and a successful finish() returns the complete report without exposing an intermediate result.

Reporting is equally explicit. A successful finish() returns a complete ScanReport. No partial report is exposed while the stream is open. Progress and outcome are separate: the engine may do incremental work internally, but the public result appears only when the stream completes successfully. If it cannot complete, the caller does not have to reconcile a nearly complete report with an error.

Streaming also has an admission step. Unsupported configurations are rejected before source execution begins. This is preferable to discovering halfway through a large input that the selected configuration cannot be represented faithfully. Admission makes the boundary visible up front.

That boundary currently has an important limitation: complete built-in portfolios are not supported in streaming mode. An embedder needs to know that the streaming surface is narrower than the whole-source surface, and that admission will reject configurations outside it. Validate the chosen configuration against the streaming documentation before designing a pipeline around it.

Streaming is therefore a contract with three parts. Input is an ordered sequence of arbitrary byte chunks. The engine is responsible for boundary correctness, including UTF-8 and CRLF splits. Output is all-or-nothing: a successful finish yields a complete report, and no partial report is exposed. Admission protects the contract by rejecting unsupported configurations before any source is consumed.

Budgets, not memory folklore#

Streaming exposes three optional limits: max_candidates, max_retained_results, and max_stream_storage_bytes. They are logical execution limits, not CPU-time limits or total-heap limits. max_candidates limits cumulative raw rule emissions, including candidates rejected by validation. max_retained_results applies to accepted records and structural review records retained before final normalization. max_stream_storage_bytes bounds the logical executor payload.

That distinction deserves emphasis. Logical executor storage accounting is not a total heap memory cap. Allocator overhead, spare capacity, compiled automata, and other workspaces are excluded. A deployment that treats a logical storage budget as a guarantee about resident memory is measuring the wrong thing.

The right mental model is a set of accounted budgets inside a larger memory picture. They make specific kinds of growth explicit and enforceable, but they are not a substitute for measuring the whole process: the engine-level budget says what the engine will account for, while process-level measurement says what the deployment actually costs. They are part of the execution contract, not a claim of “bounded memory.”

max_candidates therefore counts rejected candidates too; max_retained_results is not a count of final findings; and max_stream_storage_bytes is logical payload. None of the three is a CPU-time budget or a total-heap cap. Test the resulting trade-offs against the actual workload rather than assuming them from the names.

Incremental scanning as a transaction#

Incremental scanning addresses a different problem. Instead of receiving a source in chunks, the engine owns a complete UTF-8 document and maintains report updates as the document changes. The report is revisioned and transactional, so updates are organized around committed states rather than an ever-changing partial view.

The transactional model matters for failure behavior. Resource and execution failures are terminal. For incremental scanning, a failure preserves the last committed snapshot. The caller is not left with a report that mixes results from before and after a failed update. Invalid edit inputs can be recoverable, which is a separate category from terminal execution failure. A malformed edit is a caller-facing input problem; a resource or execution failure ends the current execution path.

Incremental scanning has four optional limits: max_candidates, max_retained_cache_records, max_materialized_results, and max_transaction_storage_bytes. As with streaming, these are logical budgets. They bound specific dimensions of candidate work, retained cache records, materialized results, and transaction storage. They are not a claim that the entire process, including compiled automata and allocator behavior, fits inside those numbers.

Matching is deliberately uneven. Incremental matching localizes eligible literal and structural work, so some kinds of matching can be updated without re-evaluating the entire rule set over the entire document. But Prefix, Suffix, and regex matching retain an authoritative whole-rule fallback. The fallback is not a failure of the optimization; it is the correctness boundary. When localized reasoning is insufficient to establish the result, the engine falls back to the authoritative path.

This also rules out a convenient but false claim: there is no general sublinear update guarantee. Some edits, workloads, and rule shapes benefit from localization. Others fall back to whole-rule work. The correct expectation is workload-dependent performance, not a universal asymptotic improvement.

For an editor or interactive service, the committed snapshot is the source of truth. The caller should not assume that every edit costs less than a full scan. The engine preserves correctness while taking advantage of localized work where eligible. The caller budgets for the cases where it is not.

What the v0.7 numbers actually show#

Cribra’s v0.7 local incremental benchmark results are a useful antidote to sweeping performance claims. They compare v0.7 incremental edit latency against the v0.6 implementation for four local workloads. They do not compare incremental scanning against whole-source rescanning.

A negative change means less edit latency than the v0.6 implementation; a positive change means more. The dense case shows a substantial reduction. The mixed and sparse-64 cases show smaller reductions. The sparse-4-1k case is a regression of 3.60%.

That last row is not an embarrassment to be omitted. It is the most useful row for understanding the model. Incremental editing is not universally faster than the v0.6 implementation. Localization has a cost, and for a small or unfavorable workload that cost can exceed the savings. The table supports a narrow, honest conclusion: v0.7 incremental edit handling can produce meaningful latency reductions relative to v0.6 on some workloads, modest reductions on others, and a small regression on at least one local case. It says nothing about incremental scanning versus a whole-source rescan.

These are local v0.7 results, not a service-level guarantee. Reproduce them against representative data before using them to justify an architecture. The labels identify the benchmark cases; the numbers measure those cases, not every possible document or rule set.

The same discipline applies to resource limits. A benchmark says what happened under a particular configuration and machine. It does not predict what happens when the rule set, document shape, allocator behavior, or host runtime changes. Measure the deployment, not the slogan.

Deployment surfaces and artifact size#

Cribra’s optimized WebAssembly artifacts are small enough to be relevant for browser and edge deployments. With Brotli level 11, the measured transfer sizes are 294,690 bytes for Core and 311,469 bytes for Flow. These are transfer-size measurements. They are not runtime memory measurements, and they should not be interpreted as the heap the engine will use after download and decompression.

That distinction is easy to lose in a deployment discussion. A compressed artifact size affects download and cold-start transfer. It says nothing directly about compiled automata, allocator capacity, retained results, or the host application’s own memory. Resource planning needs both numbers: transfer size for distribution, and measured runtime memory for execution.

Core, Flow, and Fuse are documented combinations of existing Cargo features, not separate engines with separate semantics. Fuse is not a supported WASM profile, so a WebAssembly deployment should not assume that every documented feature combination is available there. Check the configuration qualification documentation rather than inferring support from a profile name.

One more boundary matters for embedders: streaming and incremental session APIs currently belong to the Rust core. They are not exposed by the CLI, the C ABI, or the WASM adapters. The core may support a mode, but the adapter surface may not yet expose it. Check the adapter surface in the repository before committing to an integration design.

The practical consequence is that “Cribra supports streaming” and “my deployment can use streaming” are different statements. The first concerns the engine’s capabilities. The second depends on the adapter, the configuration, and the admission rules. Keeping those statements separate prevents integration rework.

One core, multiple integration boundaries#

Cribra is written in Rust, but embedding it should not require every host application to be written in Rust. The engine is exposed through distinct integration surfaces: the Rust API, a C ABI, a WebAssembly adapter, and native-language bindings. These are interoperability boundaries around the same core, not independent scanning engines. Their supported operations and release maturity must be evaluated separately; a capability available in Rust does not automatically exist in every binding.

Swift bindings are being exercised in native integration work, and Kotlin/JVM bindings are being developed and validated for Android-facing use. Python and other language targets are part of the expanding binding effort, but this article does not treat them as generally available releases without a published, verified distribution. Similarly, GitHub Actions integration belongs to the developer-workflow direction; its exact released surface should be checked against the repository rather than assumed from the existence of the core. An MCP interface is planned, not shipped.

These boundaries have practical consequences beyond language choice. They determine packaging, platform compatibility, error propagation, memory ownership, and which operations a host can invoke. Adapter qualification is therefore part of product engineering, not merely a matter of generating foreign-function signatures.

Crystal as a cross-platform integration test#

Crystal, a privacy-focused application built around local sensitive-data scanning and redaction, is where Cribra is being tested in real product integration. In Crystal’s browser-side implementation, the WebAssembly adapter executes scanning locally rather than sending document contents to a remote scanning service. The native mobile work exercises Cribra through Swift on iOS and Kotlin on Android. This is ongoing production-oriented integration testing; it should not be read as a claim that every mobile distribution or every binding API is already generally available.

The value of this arrangement is architectural. Browser WebAssembly and native mobile bindings place different demands on the same Rust core: binary packaging, host memory ownership, lifecycle management, error reporting, and platform-specific integration. Using Cribra through Crystal makes those differences observable in an application rather than only in isolated adapter tests. It does not, by itself, prove complete cross-platform parity; equivalent results still require explicit tests using matched inputs, rules, and configurations.

Crystal Desktop is a future validation target. The intended Linux application would integrate through GTK, and the Windows application through WinUI, using Cribra’s C ABI as the interoperability boundary. Those desktop integrations are planned, not implemented. If delivered, they would extend the same core into additional native UI stacks without requiring the scanning engine to take responsibility for the host application’s I/O or interface.

A resource-constrained checklist#

The contracts above suggest a short checklist for teams embedding Cribra in constrained environments.

First, choose the execution mode from the source lifecycle, not from a benchmark table. If the source is complete and owned, whole-source or incremental scanning may be appropriate. If it arrives as ordered chunks, streaming is relevant, provided the configuration passes admission.

Second, decide whether the operation must preserve a complete document. Incremental scanning owns a complete UTF-8 document and maintains revisioned, transactional updates. That is a different operational model from consuming arbitrary chunks. The two modes are not interchangeable.

Third, set logical budgets deliberately and measure process memory separately. Treat max_stream_storage_bytes or max_transaction_storage_bytes as accounted dimensions, not total heap caps. Include allocator overhead, capacity, compiled automata, and host memory in the deployment measurement.

Fourth, test boundaries. For streaming, split inputs at UTF-8 scalar boundaries and inside CRLF sequences. The engine is designed for those cases, but the integration should still verify that the surrounding pipeline does not normalize, re-chunk, or truncate data in a way that changes the source.

Fifth, distinguish failure classes. Resource and execution failures are terminal. Incremental failures preserve the last committed snapshot. Invalid edit inputs can be recoverable. Error handling should reflect those categories rather than collapsing everything into a generic retry.

Finally, benchmark the actual workload. The v0.7 table includes both improvements and a regression against v0.6. That is what a credible performance model looks like. Use the configuration qualification guidance, reproduce measurements on representative data, and keep whole-source semantics as the reference when evaluating an optimization.

Conclusion#

Cribra’s design is best understood as an execution contract for deterministic sensitive-data scanning. The core is synchronous and performs no I/O or network access. The Rust core forbids unsafe code. Whole-source scanning defines the semantic reference. Streaming accepts ordered byte chunks, handles UTF-8 and CRLF boundaries, and returns a complete report only on a successful finish. Incremental scanning owns a complete UTF-8 document and commits revisioned report updates transactionally. Both modes expose logical limits, and those limits are explicitly not total heap memory caps.

The performance story is explicit. Incremental scanning localizes eligible literal and structural work, but Prefix, Suffix, and regex matching retain authoritative whole-rule fallback, and there is no general sublinear update guarantee. The v0.7 local benchmarks show meaningful incremental-edit latency gains relative to v0.6 in some cases and a small regression in another; they do not benchmark incremental scanning against whole-source rescans. Streaming and incremental session APIs are currently Rust-core features, not adapter features.

The portability story is similarly bounded. Cribra’s Rust core is exposed through C, WebAssembly, and evolving native-language bindings; Crystal is being used to exercise browser-side WASM and Swift/Kotlin mobile integration, with GTK/WinUI desktop targets planned through the C ABI. An integration in one host does not imply identical feature coverage or verified parity in every other host.

Those constraints are not weaknesses. They are the information an engineer needs to deploy the engine honestly. For resource-constrained systems, the useful question is not whether a scanning engine can claim to be fast or bounded in the abstract. It is whether its execution semantics, accounting model, and failure behavior are explicit enough to reason about before the first byte is processed.

References#

• Cribra repository
• docs/ARCHITECTURE.md
• docs/streaming.md
• docs/incremental.md
• docs/PERFORMANCE.md
• docs/0.7-configuration-qualification.md