[ADD] Calculate memory of dataset after one hot encoding (pytorch embedding) #437
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ravinkohli
changed the base branch from
reg_cocktails_apt1.0+reg_cocktails_pytorch_embedding
to
reg_cocktails-pytorch_embedding
ravinkohli
force-pushed
the
reg_cocktails_apt1.0+reg_cocktails_pytorch_embedding_debug
branch
from
c2a98c9 to
f2f5f72
Compare
Comment on lines
59
to
60
| port=X['logger_port' | ||
| ] if 'logger_port' in X else logging.handlers.DEFAULT_TCP_LOGGING_PORT, |
There was a problem hiding this comment.
Suggested change
| port=X['logger_port' | |
| ] if 'logger_port' in X else logging.handlers.DEFAULT_TCP_LOGGING_PORT, | |
| port=X.get('logger_port', logging.handlers.DEFAULT_TCP_LOGGING_PORT), |
There was a problem hiding this comment.
Actually, we don't need this code to be merged, I'll remove this.
Comment on lines
+493
to
+494
| else: | ||
| multipliers.append(arr_dtypes[col].itemsize) |
There was a problem hiding this comment.
What happens in one-hot encoding when num_cat is larger than MIN_CATEGORIES_FOR_EMBEDDING_MAX
There was a problem hiding this comment.
they are not one hot encoded but rather sent to the PyTorch embedding module where there is implicit one-hot encoding.
| raise ValueError(err_msg) | ||
| for col, num_cat in zip(categorical_columns, n_categories_per_cat_column): | ||
| if num_cat < MIN_CATEGORIES_FOR_EMBEDDING_MAX: | ||
| multipliers.append(num_cat * arr_dtypes[col].itemsize) |
There was a problem hiding this comment.
Is it already guaranteed that all columns are non-object?
Otherwise, we should check it.
There was a problem hiding this comment.
yes its guaranteed that all columns are not object, moreover, they are also guaranteed to be np arrays, as this code is run after we have transformed the data using the tabular feature validator.
| if len(categorical_columns) > 0: | ||
| if n_categories_per_cat_column is None: | ||
| raise ValueError(err_msg) | ||
| for col, num_cat in zip(categorical_columns, n_categories_per_cat_column): |
There was a problem hiding this comment.
We could use sum(...) same as below. (optional)
Co-authored-by: nabenabe0928 <[email protected]>
theodorju
reviewed
Jul 16, 2022
autoPyTorch/pipeline/components/preprocessing/tabular_preprocessing/TabularColumnTransformer.py
Outdated
Show resolved
Hide resolved
ravinkohli
commented
Jul 16, 2022
autoPyTorch/pipeline/components/preprocessing/tabular_preprocessing/TabularColumnTransformer.py
Outdated
Show resolved
Hide resolved
…ssing/TabularColumnTransformer.py
ravinkohli
added a commit
that referenced
this pull request
Aug 16, 2022
…edding) (#437) * add updates for apt1.0+reg_cocktails * debug loggers for checking data and network memory usage * add support for pandas, test for data passing, remove debug loggers * remove unwanted changes * : * Adjust formula to account for embedding columns * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * remove unwanted additions * Update autoPyTorch/pipeline/components/preprocessing/tabular_preprocessing/TabularColumnTransformer.py Co-authored-by: nabenabe0928 <[email protected]>
ravinkohli
added a commit
that referenced
this pull request
Oct 25, 2022
…edding) (#437) * add updates for apt1.0+reg_cocktails * debug loggers for checking data and network memory usage * add support for pandas, test for data passing, remove debug loggers * remove unwanted changes * : * Adjust formula to account for embedding columns * Apply suggestions from code review Co-authored-by: nabenabe0928 <[email protected]> * remove unwanted additions * Update autoPyTorch/pipeline/components/preprocessing/tabular_preprocessing/TabularColumnTransformer.py Co-authored-by: nabenabe0928 <[email protected]>
ravinkohli
deleted the
reg_cocktails_apt1.0+reg_cocktails_pytorch_embedding_debug
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to improve the approximate memory usage of a dataset by considering the dataset after transforming with one-hot encoding. Based on our experiments (reg cocktails ablation study), we have observed that columns with a high cardinality of their categorical features tend to explode when they are one-hot encoded. Moreover, even with the addition of PyTorch embedding (removing the need to one hot encode all categorical columns), we observe that excessive memory is used while building the neural network.