Skip to content

Add paging for large data files #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
RandomFractals opened this issue Jan 4, 2022 · 3 comments
Closed

Add paging for large data files #60

RandomFractals opened this issue Jan 4, 2022 · 3 comments
Labels
feature

Comments

@RandomFractals
Copy link
Owner

RandomFractals commented Jan 4, 2022 •
edited
Loading

Switching to CSV data parsing with fs.ReadStream and Papa.parse in a worker thread added in #59 now loads larger data files with millions of rows and up to 1Gb of data in a reasonable time frame.

There seems to be an issue with Tabulator virtual DOM setup recorded in #58.

As a short term solution, we'll add paging for large data files to navigate large datasets in table view after data is parsed and loaded into memory.
@RandomFractals RandomFractals mentioned this issue Jan 4, 2022
Closed
RandomFractals added a commit that referenced this issue Jan 5, 2022
@RandomFractals
initial set of table paging UI config (#60)
bb9ecb6 
- add paging and page sizes config to Tabulator setup in tableView.js
- change data loading page size to 100K in tableView.ts
- clear loading data status message when all data is read and total rows counter is set
- add starting CSS for the Tabulator footer row in table-view.css
RandomFractals added a commit that referenced this issue Jan 5, 2022
@RandomFractals
set min pager size to 100 and initial pagination size to 1K rows (#60)
5752b72
RandomFractals added a commit that referenced this issue Jan 5, 2022
@RandomFractals
rework incremental paged data loading in table view (#58 and #60)
47f46ac
This was referenced Jan 6, 2022
Closed
Closed
RandomFractals added a commit that referenced this issue Jan 7, 2022
@RandomFractals
refactor paged data loading and passing to table view (#60)
6a907bb
RandomFractals added a commit that referenced this issue Jan 7, 2022
@RandomFractals
display the first data page on table view reload (#60)
97dcc0d
RandomFractals added a commit that referenced this issue Jan 7, 2022
@RandomFractals
add simple data page selector (#60)
1911465 
and comment out data load debug logging for now
@RandomFractals
Copy link
Owner Author

table view with new simple data page selector and 100K rows per page default:

image

RandomFractals added a commit that referenced this issue Jan 7, 2022
@RandomFractals
use replace data on reload and paging to preserve tabulator table col…
05bb254 
…umns config (#60)
@RandomFractals
Copy link
Owner Author

data loading stats and paging with larger data file (~1.1GB & ~5.3M rows):

tabular-data-5M-rows-paging

@RandomFractals
Copy link
Owner Author

Closing this for now. See new Limitations section in docs related to large data files streaming, pagination, display, tabulator sort and save data functionality:

https://github.com/RandomFractals/tabular-data-viewer#limitations

This was referenced Jan 9, 2022
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@RandomFractals