datagen

CREATE TABLE simple (
    payload [ BIGINT | INTEGER ] NOT NULL PRIMARY KEY
) WITH (
  'materialized' = 'true',
  'connectors' = '[{"name":"data","transport":{"name":"datagen","config":{"workers":9,"plan":[{"fields":{"payload":{"strategy":"[ increment | uniform ]"}}}]}}}]'
);

original is v0.281.0 and roaring branch is the current platform runtime on 92e4a95. enable_roaring=true was set on all runs.

Bench Summary

• increment workloads improved on the roaring branch: better throughput and much lower memory.
• uniform workloads in the noise on throughput, memory.

Filter Hit/Miss Comparison:

Takeaways:

• higher miss rate is better here it means the filter is rejecting more lookups.
• In these runs, the measurable rejection work came from range for increment and bloom for uniform; roaring hit/miss counters themselves stayed 0 in the final samples.
• throuhgput increase most likely from building the bitmap filters faster than the bloom filters for increment workloads

postgres

original is v0.281.0 and roaring branch is the platform runtime on 92e4a95.

I loaded all four Postgres tables to 200,000,000 rows each. It's very slow.

Bench Summary

Filter Read
Miss rate here is misses / (hits + misses), so higher is better for the filter.

Takeaways:

• The postgres input connector is not able to saturate the circuit (buffered records 0).
• Random workloads are basically flat. One is slightly worse, one slightly better.
• Sequence workloads show roaring roaring branch helps with filter size.
• For sequence keys, the roaring branch switches filter representation from bloom to roaring and cuts filter size a lot, observed rejection already in the range filter. Roaring hit/miss counters stayed 0/0.
• For random keys, bloom is doing the rejection at ~99.99% miss rate in both versions.

deltalake

original is v0.281.0 and roaring branch is the platform runtime on the current manager.

CREATE TABLE int_random_keys (
    id INT NOT NULL PRIMARY KEY
) WITH (
  'materialized' = 'true',
  'connectors' = '[{"name":"data","transport":{"name":"delta_table_input","config":{"uri":"file:///mnt/data/deltalake/$table","mode":"snapshot","transaction_mode":"none"}}}]'
);

These were 300-second fda bench runs, and all four runs had enable_roaring=true.
int_random_keys: delta-rs table filled with random u32 keys
int_sequence_keys: delta-rs table built with sequentially filling keys

Bench Summary

For fixed 300s runs, absolute storage at stop time is influenced by how much work got processed in that window.

Filter Miss Table

Miss rate is misses / (hits + misses), so higher is better.

Read

• INT random is effectively flat. Both versions stay bloom-based, and the filter behavior is almost identical.
• INT sequence is the real win for the roaring branch: +21.4% throughput and -18.1% memory.
• On INT sequence, original stays bloom-based, while the roaring branch switches fully to roaring and shows a 100% roaring miss rate in the final sample.
• Absolute storage is higher on the roaring branch for INT sequence, but it also processed much more data in the same 300s window; normalized state-amplification is slightly better.

kafka

CREATE TABLE kafka_int_sequence_keys (
    id INT NOT NULL PRIMARY KEY
) WITH (
  'materialized' = 'true',
  'connectors' = '[{"name":"data","transport":{"name":"kafka_input","config":{"topic":"$topic","bootstrap.servers":"127.0.0.1:9092","start_from":"earliest","resume_earliest_if_data_expires":true,"poller_threads":10,"log_level":"info"}},"format":{"name":"avro","config":{"update_format":"raw","schema":"{\"type\":\"record\",\"name\":\"KafkaInt\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}","skip_schema_id":true}}}]'
);

• two topics: kafka_int_sequence_keys, kafka_int_random_keys
• This only ran at ~800k/s. Not maxing out the circuit

Bench Summary

Filter Miss Table

Main read:

• INT sequence is where roaring helps on Kafka+Avro: throughput is bottlenecked on kafka, memory drops a bit and storage/state amplification drop a lot.
• The filter behavior matches that: random stays bloom-based; sequence flips from bloom on original to roaring on the roaring branch.

This is currently configured as an enabled by default feature with option to disable it in dev-tweaks.
Alternative: disabled for now, enable selectively for some users.

Thoughts?

Thanks @blp great feedback, I addressed evyerthing except for the upsampling constant of 100. Will try to run the benchmark with lower values and see if it has an impact.

• It looks like some of our users disable bloom filters, need to make sure both bloom and bitmaps can be disabled at the same time

Most benchmarks are mixed. Is the main justification for this that it significantly speeds up ingest into tables whose PKs arrive in-order?

As long as the connector can max out the circuit it seems to be between 10-20% better wrg to tput (datagen+delta), and less memory (-28% datagen, -20% delta) for things that arrive mostly ordered.
When the connector isn't maxed out we probably would have to run longer to see effects on mem+tput.

#6007 (comment)

One question I don't know and haven't looked at is if also can benefits transactions commits if it has to build a spine and can do it in a way such new entries appear mostly sequential. Do you think it's plausible?

Most benchmarks are mixed. Is the main justification for this that it significantly speeds up ingest into tables whose PKs arrive in-order?

As long as the connector can max out the circuit it seems to be between 10-20% better wrg to tput (datagen+delta), and less memory (-28% datagen, -20% delta) for things that arrive mostly ordered. When the connector isn't maxed out we probably would have to run longer to see effects on mem+tput.

#6007 (comment)

I missed the numbers for memory. But yes, this is a significant improvement, if the input pattern is favorable. I don't know how frequently this happens in practice.

One question I don't know and haven't looked at is if also can benefits transactions commits if it has to build a spine and can do it in a way such new entries appear mostly sequential. Do you think it's plausible?

This will be the case if the output of an operator is indexed in the same way as the input, then you can expect outputs to be ordered. Also not sure how common this is. Normally outputs need to be reindexed.

This is currently configured as an enabled by default feature with option to disable it in dev-tweaks. Alternative: disabled for now, enable selectively for some users.

Thoughts?

Have you tested it on some customer workloads? If you feel you've done enough testing, it may be ok to enable by default.