Repository navigation
[TRACKER][BUG] Integer-based indexing causing failures when data grows very large. #2459
Description
Below is a reproducible example on the 0.15 nightly of cuML w/ CUDA 10.2.
There are several estimators that fail with either illegal memory access was encountered or invalid configuration argument, both of which seem to indicate an integer overflow might be occurring when representing the size of the underlying array.
import rmm
rmm.reinitialize(managed_memory=True, pool_allocator=False)
import cupy as cp
cp.cuda.set_allocator(rmm.rmm_cupy_allocator)
import numpy as np
a = np.random.random((1000000, 5000))
from cuml.decomposition import PCA
pca = PCA()
pca.fit(a)Here's the exception
[W] [10:33:38.606739] Expected column ('F') major order, but got the opposite. Converting data, this will result in additional memory utilization.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "cuml/decomposition/pca.pyx", line 394, in cuml.decomposition.pca.PCA.fit
RuntimeError: Exception occured! file=/conda/conda-bld/libcuml_1591208799167/work/cpp/src_prims/common/buffer_base.hpp line=55: FAIL: call='cudaStreamSynchronize(_stream)'. Reason:an illegal memory access was encountered
Obtained 19 stack frames
#0 in /raid/cnolet/miniconda3/envs/rapidgenomics2/lib/python3.7/site-packages/cuml/common/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9Exception16collectCallStackEv+0x3e) [0x7fb84624613e]
#1 in /raid/cnolet/miniconda3/envs/rapidgenomics2/lib/python3.7/site-packages/cuml/common/pointer_utils.cpython-37m-x86_64-linux-gnu.so(_ZN8MLCommon9ExceptionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x80) [0x7fb846246c50]
#2 in /raid/cnolet/miniconda3/envs/rapidgenomics2/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN8MLCommon11buffer_baseIdNS_15deviceAllocatorEEC2ESt10shared_ptrIS1_EP11CUstream_stm+0x164) [0x7fade2e2d314]
#3 in /raid/cnolet/miniconda3/envs/rapidgenomics2/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN2ML6pcaFitIdEEvRKNS_15cumlHandle_implEPT_S5_S5_S5_S5_S5_S5_RKNS_9paramsPCAEP11CUstream_st+0x103) [0x7fade30da133]
#4 in /raid/cnolet/miniconda3/envs/rapidgenomics2/lib/python3.7/site-packages/cuml/common/../../../../libcuml++.so(_ZN2ML6pcaFitERNS_10cumlHandleEPdS2_S2_S2_S2_S2_S2_RKNS_9paramsPCAE+0x5a) [0x7fade30cf07a]
#5 in /raid/cnolet/miniconda3/envs/rapidgenomics2/lib/python3.7/site-packages/cuml/decomposition/pca.cpython-37m-x86_64-linux-gnu.so(+0x12a8f) [0x7fb843a62a8f]
#6 in python(_PyObject_FastCallKeywords+0x15c) [0x560920306bec]
#7 in python(+0x181661) [0x560920307661]
#8 in python(_PyEval_EvalFrameDefault+0x48a2) [0x56092034d762]
#9 in python(_PyEval_EvalCodeWithName+0x255) [0x56092029f505]
#10 in python(PyEval_EvalCode+0x23) [0x5609202a08f3]
#11 in python(+0x227692) [0x5609203ad692]
#12 in python(+0xf0758) [0x560920276758]
#13 in python(PyRun_InteractiveLoopFlags+0xeb) [0x5609202768fd]
#14 in python(+0xf0998) [0x560920276998]
#15 in python(+0xf12ac) [0x5609202772ac]
#16 in python(_Py_UnixMain+0x3c) [0x5609203b8aec]
#17 in /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fb870cc1b97]
#18 in python(+0x1d773d) [0x56092035d73d]
In the above example, I used a size of 1Mx5000, which is >> 2^31 number of samples. I also tried with 1Mx2500, which is also > 2^31.
Just to rule out the possibility that this error only occurs in the case of oversubscribing the GPU memory, I tried with 1Mx2100, (which is <2^31 but still requires >32gb of GPU memory to train):
>>> import rmm
>>> rmm.reinitialize(managed_memory=True, pool_allocator=False)
0
>>> import numpy as np
>>> a = np.random.random((1000000, 2100))
>>> import cupy as cp
>>> cp.cuda.set_allocator(rmm.rmm_cupy_allocator)
>>> from cuml.decomposition import PCA
>>> pca = PCA()
>>> pca.fit(a)
[W] [10:46:02.336113] Expected column ('F') major order, but got the opposite. Converting data, this will result in additional memory utilization.
PCA(copy=True, handle=<cuml.common.handle.Handle object at 0x7f21bafe3ed0>, iterated_power=15, n_components=1, random_state=None, svd_solver='auto', tol=1e-07, verbose=2, whiten=False, output_type='numpy')As a result of the behavior outlined above, I have a very strong suspicion the are integers being used to represent array sizes that should be promoted to size_t. This should also affect TruncatedSVD.
Keeping a list to track the places where this occurs so far:
- PCA
- TSVD
- Logistic Regression
- Linear Regressoin
- K-Means
Activity
cjnolet commentedon Jun 22, 2020
Logistic regression seems to throw an error before the algorithm even executes and it seems to be from CuPy:
cjnolet commentedon Jun 23, 2020
I executed the
1Mx5ktraining and inference on several more models:Failed:
Did not fail:
[-][BUG] UVM failures when data grows very large. [/-][+][BUG] Integer-based indexing causing failures when data grows very large. [/+]cjnolet commentedon Jun 27, 2020
I believe the solution for most of these should be to convert any and all variables representing number of array elements (e.g., n_rows * n_cols) to
size_t.github-actions commentedon Feb 17, 2021
This issue has been marked rotten due to no recent activity in the past 90d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.
github-actions commentedon Feb 17, 2021
This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.
[-][BUG] Integer-based indexing causing failures when data grows very large. [/-][+][TRACKER][BUG] Integer-based indexing causing failures when data grows very large. [/+]cjnolet commentedon Jun 11, 2021
Testing these on 2108 w/ 1Mx2.1k (float32):
Using 64-bit array lengths to increase scale of pca & tsvd (#3983)
cjnolet commentedon Aug 24, 2021
I really thought PCA/TSVD had been fixed in 21.08 but it does appear the issue still lingers:
Definitely still seems to be an overflow somewhere. I'm guessing in a 32-bit int.
wphicks commentedon Jan 24, 2022
Related: #4105
Fix RF integer overflow (#4563)
Using 64-bit array lengths to increase scale of pca & tsvd (NVIDIA#3983)
Fix RF integer overflow (NVIDIA#4563)