🚸 Restrict "column 'col' not in dataframe" error to at most 10 columns

[Zethson]

Fixes #2547

Before

After

ValidationError: column 'date' not in dataframe. Columns in dataframe: ['transaction_amount_usd_cent', 'currency_name', 'transaction_id', 'transaction_date', 'transaction_time', 'merchant_id', 'merchant_name', 'merchant_category', 'payment_method', 'card_type', 'card_last_four', 'customer_id', 'customer_email', 'transaction_status', 'is_recurring', 'geography_country', 'geography_city', 'device_type', 'platform', 'is_international', 'exchange_rate', 'fees_applied']

ValidationError: column 'date' not in dataframe. Found 22 columns including: 'transaction_amount_usd_cent', 'currency_name', 'transaction_id', 'transaction_date', 'transaction_time', 'merchant_id', 'merchant_name', 'merchant_category', 'payment_method', 'card_type'...

This is in line with the rest of our UX where we also only show the 10 first hits. The difference becomes bigger of course the more columns we have. I observed that a user got spammed with a stupid amount of columns for their dataset which made the whole notebook unreadable.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.16%. Comparing base (cbe457a) to head (be56055).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2549      +/-   ##
==========================================
+ Coverage   92.15%   92.16%   +0.01%     
==========================================
  Files          60       60              
  Lines        9988    10023      +35     
==========================================
+ Hits         9204     9238      +34     
- Misses        784      785       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🚀 Deployed on https://67d02884cfb7a7a36f9762df--lamindb-qnwk.netlify.app

[sunnyosun]

This looks so complicated and so much code, is there a way you can utilize

lamindb/lamindb/models/_from_values.py

Line 301 in 1373325

def _format_values(

? Which was used everywhere to truncate logging list.

[Zethson]

@sunnyosun thank you. I don't know why I didn't think of it but it makes it much more concise.

[sunnyosun]

Could this be simplified to display_cols_str = _format_values(self._dataset.columns, n=10)?

[sunnyosun]

err_msg = f"column '{missing_col}' not in dataframe. {len(list(self._dataset.columns))} columns in dataframe: {display_cols_str}"

[sunnyosun]

what happens if more than 1 columns are not in dataframe?

[sunnyosun]

This is OK if you need it urgently, but it's a bit patchy, we should ideally have a parser for all pandera errors.

[Zethson]

No this is not urgent so I might revisit this at a later point with more general code.