Feature/mdf data sources by jonathanb-db · Pull Request #69 · databrickslabs/impulse

jonathanb-db · 2026-07-30T06:28:36Z

Summary

Adds 3 DataSources for MF4 binary file loading, required data readers as well as a quickstart guide ("QUICKSTART.md" in the data_sources subdirectory). This is the first of many planned data sources to be added to impulse. The introduced readers are optimized to work with spark and use a stripe-memory reading approach to optimize IO-calls and general throughput compared to other solutions available. Please review the "README.md" in the code directory for details.

Currently this feature is experimental, since some parts of the mf4 standard are not yet implemented, missing functionality shall be added in the future once required. For a full list of current limitations review "KNOWN_LIMITATIONS.md" in the sub directory.

A Databricks Industry Solution that uses these data sources for a end-to-end ingestion pipeline is coming soon.

Changes

Added 3 Data sources to load mf4 files into a spark data frame
Added required readers to interpret mf4

Test Plan

Unit tests added/updated
Manual testing completed
Documentation updated (if applicable)

Checklist

Code follows project style guidelines
Self-review completed
No new linter warnings introduced

tombonfert · 2026-07-31T11:25:21Z

-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/ee/67/531ea369ba64dcff5ec9c3402f9f51bf748cec26dde048a2f973a4eea7f5/annotated_types-0.7.0.tar.gz", hash = "sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89", size = 16081, upload-time = "2024-05-20T21:33:25.928Z" }
+source = { registry = "https://pypi-proxy.cloud.databricks.com/simple/" }
+sdist = { url = "https://pypi-proxy.cloud.databricks.com/packages/ee/67/531ea369ba64dcff5ec9c3402f9f51bf748cec26dde048a2f973a4eea7f5/annotated_types-0.7.0.tar.gz", hash = "sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89", upload-time = "2024-05-20T21:33:25.928Z" }


The databricks pypi-proxy is not reachable from the GH runner and this might be the reason why the lint check fails. Please check the uv.lock file of Impulse in the main branch: https://github.com/databrickslabs/impulse/blob/main/uv.lock

Thanks @tombonfert , just pushed with correct pypi

codecov · 2026-08-04T07:12:46Z

Codecov Report

❌ Patch coverage is 96.17580% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.72%. Comparing base (8457893) to head (3463716).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
src/impulse_data_sources/mdf/mdf4_reader.py	95.25%	9 Missing and 8 partials ⚠️
src/impulse_data_sources/mdf/mdf_blocks.py	94.23%	6 Missing and 11 partials ⚠️
src/impulse_data_sources/mdf/mdf_decode.py	95.63%	4 Missing and 9 partials ⚠️
src/impulse_data_sources/mdf/datasources.py	95.85%	7 Missing and 1 partial ⚠️
src/impulse_data_sources/mdf/arrow_emit.py	98.37%	3 Missing and 4 partials ⚠️
src/impulse_data_sources/mdf/bin_packer.py	97.05%	2 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #69      +/-   ##
==========================================
+ Coverage   88.67%   90.72%   +2.05%     
==========================================
  Files          60       69       +9     
  Lines        5008     6946    +1938     
  Branches      596      915     +319     
==========================================
+ Hits         4441     6302    +1861     
- Misses        461      497      +36     
- Partials      106      147      +41

Flag	Coverage Δ
data_sources	`96.17% <96.17%> (?)`
query_engine	`84.57% <ø> (ø)`
reporting	`94.22% <ø> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/impulse_data_sources/mdf/schemas.py	`100.00% <100.00%> (ø)`
src/impulse_data_sources/mdf/udf_helpers.py	`100.00% <100.00%> (ø)`
src/impulse_data_sources/mdf/bin_packer.py	`97.05% <97.05%> (ø)`
src/impulse_data_sources/mdf/arrow_emit.py	`98.37% <98.37%> (ø)`
src/impulse_data_sources/mdf/datasources.py	`95.85% <95.85%> (ø)`
src/impulse_data_sources/mdf/mdf_decode.py	`95.63% <95.63%> (ø)`
src/impulse_data_sources/mdf/mdf4_reader.py	`95.25% <95.25%> (ø)`
src/impulse_data_sources/mdf/mdf_blocks.py	`94.23% <94.23%> (ø)`

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tombonfert · 2026-08-07T08:28:06Z

can we integrate this part into the existing docs of Impulse?

it's integrated in the docs, shall we leave it in here as well for offline/code doc or remove and only have in the docs app?

lets remove it here and only have it in the docs folder.

tombonfert · 2026-08-07T08:28:27Z

please integrate into the existing docs of Impulse

it's integrated in the docs, shall we leave it in here as well for offline/code doc or remove and only have in the docs app?

lets have it just in the normal docs folder and remove it here please.

tombonfert · 2026-08-07T08:29:29Z

please create a new section in the existing docs of Impulse under https://databrickslabs.github.io/impulse/docs/references/data_sources ?

it's integrated in the docs, shall we leave it in here as well for offline/code doc or remove and only have in the docs app?

only in the general Impulse docs please.

tombonfert · 2026-09-04T07:15:45Z

-is the natural place to record them).
+Impulse ships an [MDF4 data source](../data_sources/mdf4.md) that reads raw
+`.mf4` files directly into Spark DataFrames — parsing MDF4 binary blocks
+across workers, no `asammdf` at runtime. It exposes the `mdf_signals`,


It is not required to mention asammdf here because it is not used by Impulse.

tombonfert · 2026-09-04T07:49:27Z

+    s_bit_count = ch_spec["bit_count"]
+    s_byte_offset = ch_spec["byte_offset"]
+    s_bit_offset = ch_spec["bit_offset"]
+    s_bytes = (s_bit_count + 7) // 8


Correctness: bit-offset ignored when computing byte width (bit-packed signals truncated)

The byte width here is (bit_count + 7) // 8, which ignores bit_offset. Any signal whose bits span an extra byte because of a non-zero bit_offset has its high bits truncated.

Example: an 8-bit unsigned signal at bit_offset=4 occupies bits 4..11 (2 bytes), but s_bytes = (8+7)//8 = 1 grabs only 1 byte; after >> 4 only the low 4 bits survive, so a value of 200 decodes to ~12.

The same defect appears in convert_values value width (~line 35) and extract_timestamps (~line 481). The aligned fast paths require bit_offset == 0, so every non-zero-bit-offset channel hits this path. Bit-packed sub-byte signals with a non-zero bit offset are the norm for automotive bus data (the stated primary use case), so this is high-impact. The current tests only cover cases where bit_offset + bit_count fits the rounded byte width, which masks the bug.

Suggested fix: compute the width from the span bit_offset + bit_count, i.e. (bit_offset + bit_count + 7) // 8.

Flagged by Isaac Review.

tombonfert · 2026-09-04T07:49:28Z

+        vals = vals & ((1 << bit_count) - 1)
+        sign_bit = np.uint64(1 << (bit_count - 1))
+        vals = np.where(
+            vals & sign_bit, vals.astype(np.int64) - (1 << bit_count), vals.astype(np.int64)


Correctness: signed 64-bit conversion wraps before subtracting 1 << bit_count

The signed conversion does vals.astype(np.int64) before subtracting (1 << bit_count). For bit_count == 64, the unsigned value wraps to int64 first and is then decremented by 2^64, producing a wildly wrong result.

Example: a signed 64-bit BE (or byte-misaligned / bit-offset) channel with value 0x8000000000000000 should decode to -9223372036854775808. Instead astype(int64) wraps to -9.2e18 and then subtracts 2^64, yielding ~-2.77e19 (object dtype), cast to float64 as a nonsense value. Verified empirically.

The LE-aligned 64-bit signed path uses a correct fast path, so only BE / misaligned / bit-offset signed-64 channels are affected.

Suggested fix: perform the sign correction in unsigned/Python-int space (or on the raw unsigned array) before casting, e.g. subtract 1 << bit_count where the top bit is set, then cast.

Flagged by Isaac Review.

tombonfert · 2026-09-04T07:49:30Z

+        raw_tab = np.array(cc_params[0::2])
+        phys_tab = np.array(cc_params[1::2])
+        return np.interp(values, raw_tab, phys_tab)
+    if cc_type == CC_TAB_NOINTERP:


Correctness: invalidation lost through table conversions (invalid samples gain fabricated values)

extract_signal applies invalidation (sets NaN) and then applies the channel conversion. For a CC_TAB_NOINTERP channel, searchsorted(raw_tab, nan) clips to the last table entry and np.where selects a real physical value, so a sample that was correctly NaN (invalidated) silently becomes a fabricated table value.

Verified empirically: input [nan, 1.0] -> [30.0, 20.0]. Any table-converted channel can turn invalidated samples into bogus real values (other CC types are affected to a lesser degree).

Suggested fix: preserve NaN through the conversion, e.g. mask out NaN inputs before searchsorted/np.where and re-apply NaN to those positions afterward.

Flagged by Isaac Review.

tombonfert · 2026-09-04T07:50:30Z

+
+Read ASAM **MDF4** measurement files (`.mf4`) as Spark DataFrames, or convert them
+straight to Delta Lake tables. The reader parses MDF4 binary blocks **directly** —
+`asammdf` is *not* a runtime dependency — so decoding parallelises across Spark


Please don't mention asammdf if it is not used at all (might confuse readers).

tombonfert · 2026-09-04T07:52:32Z

+`register_mdf_datasources` is the recommended entry point: it verifies the workspace
+client, tags API calls with `databricks-impulse` product info, and emits a
+lightweight telemetry beacon each time Spark plans partitions for a read. If you
+register the data-source classes manually instead, reads still work but no telemetry
+is sent.


this is not required in the docs.

tombonfert · 2026-09-04T08:24:04Z

+    F.col("channel_name").alias("value"),
+)
+
+channel_metrics = (


lets add channel_name (and if available device_name/bus_name) to the channel_metrics table. This way the channel_tags table is getting optional. Lets add this to the documentation.

tombonfert · 2026-09-04T08:31:11Z

revert those changes please (unrelated)

jonathanb-db added 4 commits July 2, 2026 09:09

Added mdf datasources

eaf84c6

added tests and documentation for mdf data sources

41bd77c

moved known limitations to its own md

7d58657

renamed impulse_ds => impulse_data_sources

5cc9587

jonathanb-db assigned tombonfert Jul 30, 2026

jonathanb-db requested a review from a team as a code owner July 30, 2026 06:28

code cleanup for pylint

04ab77c

tombonfert reviewed Jul 31, 2026

View reviewed changes

reset uv.lock

7e66ccf

jonathanb-db added 3 commits August 4, 2026 14:55

renamed and fixes for data source unit tests

7177e4c

Merge remote-tracking branch 'origin/main' into feature/mdfDataSources

450086d

added data_sources tests to acceptance test matrix

5577a5f

tombonfert reviewed Aug 7, 2026

View reviewed changes

Comment thread .isaac/config.json Outdated

tombonfert reviewed Aug 7, 2026

View reviewed changes

jonathanb-db added 2 commits August 14, 2026 09:44

added impulse datasources:mf4 docs

5314242

removed converter.py

3463716

tombonfert reviewed Sep 4, 2026

View reviewed changes

Comment thread docs/impulse/package-lock.json

tombonfert Sep 4, 2026

Copy link
Copy Markdown

Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert those changes please (unrelated)

Uh oh!

Conversation

jonathanb-db commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Test Plan

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tombonfert Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tombonfert Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tombonfert Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jonathanb-db commented Jul 30, 2026 •

edited

Loading

codecov Bot commented Aug 4, 2026 •

edited

Loading

tombonfert Sep 4, 2026 •

edited

Loading

tombonfert Sep 4, 2026 •

edited

Loading

tombonfert Sep 4, 2026 •

edited

Loading