Skip to content

Switch to Papa Parse for CSV data loading and parsing in a worker thread #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
RandomFractals opened this issue Jan 1, 2022 · 3 comments
Closed

Switch to Papa Parse for CSV data loading and parsing in a worker thread #59

RandomFractals opened this issue Jan 1, 2022 · 3 comments
Labels
data Data task enhancement New feature or request

Comments

@RandomFractals
Copy link
Owner

Even though I got the data parsing part decently tuned with tableshema library and table.iter() setup in #50, I still need to do it in worker threads.

Papa Parse has that as one of the config options and looks more up to date and resilient with malformed CSV data files.

https://www.papaparse.com/
@RandomFractals RandomFractals added enhancement New feature or request data Data task labels Jan 1, 2022
@RandomFractals RandomFractals mentioned this issue Jan 1, 2022
Closed
@RandomFractals
Copy link
Owner Author

RandomFractals commented Jan 1, 2022 •
edited
Loading

This looks promising. Parsing CTA ridership CSV from #56 in a browser with Papa Parse:

papa-parse-demo

RandomFractals added a commit that referenced this issue Jan 1, 2022
@RandomFractals
add papaparse library to package.json for csv data loading (#59)
7c5075e
RandomFractals added a commit that referenced this issue Jan 1, 2022
@RandomFractals
switch to using Papa.parse with readable file stream for CSV data loa…
8370278 
…ding (#59)
RandomFractals added a commit that referenced this issue Jan 1, 2022
@RandomFractals
enable dynamic typing and skip empty lines when parsing csv data (#59)
a8731a7
@RandomFractals RandomFractals mentioned this issue Jan 4, 2022
Closed
@RandomFractals
Copy link
Owner Author

takes about 2 minutes to parse 1.1 Gb of data with dynamic typing. This can be further enhanced by turning off dynamic typing and let the grid handle data formatting on visible rows display ...

image

@RandomFractals
Copy link
Owner Author

This is done. Will handle loading all parsed data into table view incrementally with pagination in #60

This was referenced Jan 9, 2022
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
data Data task enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@RandomFractals