sfm::reconstruct error with python error: (-215:Assertion failed) k == STD_VECTOR_MAT || k == STD_ARRAY_MAT in function 'getMatRef'

[themantalope]

Hi Everyone,

I'm messing around with the sfm module and doing a 3D reconstruction from a video file. I've compiled the OpenCV project with the contributed modules on my machine (Mac OS X 10.11.6, python 3.6 Anaconda) with python bindings for sfm. Compilation was successful, and I am able to successfully run the example_sfm_scene_reconstruction program.

For the problem I'm having, the general program is as follows:

1. Extract the location of ORB features for the first 8 images from the video
2. Compile the features into a python list, where each element of the list is a 2XN (where N is the number of features detected in the image, typically 100 points so 2X100) numpy array.
3. Load the camera calibration matrix from a previously stored file
4. Call the sfm::reconstruct(InputArrayOfArrays points2d, OutputArray Ps, OutputArray points3d, InputOutputArray K, bool is_projective) via the python binding.

Code is as follows

# get all the keypoints, save them in a pandas dataframe
all_keypoints = []

for idx, im in tqdm(zip(range(idx_start, idx_stop),images),desc='gettingkeypoints',total=len(images)):
#     kps = fast_detector.detect(im)
    kps = orb_detector.detect(im)
    for kp in kps:
        all_keypoints.append({'frame':idx, 'x':kp.pt[0], 'y':kp.pt[1]})

# get the keypoints for the first 8 frames
samped_frames = list(set(kp_df.loc[:, 'frame'].values.tolist()))
samped_frames = samped_frames[0:8]
kps_for_recon = []
for idx in samped_frames:
    kps_data = kp_df.loc[kp_df['frame']==idx, ['x','y']]
    kps_for_recon.append(np.matrix(kps_data.values.transpose().astype(np.float32).copy()))
    
# get the camera calibration data
lcm = pd.read_csv(left_cam_cal_file, index_col=0)
lcm = lcm.values.astype(np.float32).copy()

# run scene reconstruction

cv2.sfm.reconstruct(kps_for_recon, lcm, None, None, True)

The code starts to run, then I get an error:
error: (-215:Assertion failed) k == STD_VECTOR_MAT || k == STD_ARRAY_MAT in function 'getMatRef'

Based on how I understand the sfm::reconstruct function works, I think the error is coming from the line linked below.

opencv_contrib/modules/sfm/src/reconstruct.cpp

Line 141 in 21e1c8b

Mat(P).copyTo(Ps.getMatRef(i));

Camera matrix:

1685.72249 | 0.000000 | 689.894140
0.00000 | 1749.486859 | 767.726025
0.00000 | 0.000000 | 1.000000

Any help much appreciated. I'm really not sure where the error is coming from.

[berak]

i don't have the sfm module (or python bindings for that), but:

if lcm is your camera Mat, it should go into the 4th argument(K), not the 2nd (Ps).
maybe you should use "named" arguments here, like:

ps, pts3d = cv2.sfm.reconstruct(kps_for_recon, K=lcm, is_projective=True)

can you also try a : help(cv2.sfm.reconstruct) , and show us the output ?
(so we can see, which signature was generated exactly)

[themantalope]

Hey @berak

Here is the output of help(cv2.sfm.reconstruct):

Help on built-in function reconstruct:

reconstruct(...)
    reconstruct(points2d, K[, Ps[, points3d[, is_projective]]]) -> Ps, points3d, K
    .   @brief Reconstruct 3d points from 2d correspondences while performing autocalibration.
    .   @param points2d Input vector of vectors of 2d points (the inner vector is per image).
    .   @param Ps Output vector with the 3x4 projections matrices of each image.
    .   @param points3d Output array with estimated 3d points.
    .   @param K Input/Output camera matrix \f$K = \vecthreethree{f_x}{0}{c_x}{0}{f_y}{c_y}{0}{0}{1}\f$. Input parameters used as initial guess.
    .   @param is_projective if true, the cameras are supposed to be projective.
    .   
    .   This method calls below signature and extracts projection matrices from estimated K, R and t.
    .   
    .   @note
    .   - Tracks must be as precise as possible. It does not handle outliers and is very sensible to them.

Changing the input arguments to named arguments resulted in the same error:

cv2.sfm.reconstruct(kps_for_recon, K=lcm,is_projective=True)
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-96-c9a12494b6e8> in <module>()
----> 1 cv2.sfm.reconstruct(kps_for_recon, K=lcm,is_projective=True)

error: OpenCV(4.0.0-pre) /Users/<path>/opencv/modules/core/src/matrix_wrap.cpp:1735: error: (-215:Assertion failed) k == STD_VECTOR_MAT || k == STD_ARRAY_MAT in function 'getMatRef'

[berak]

oh, thanks for the report !
(ok, K is mandatory, so the args get reordered, and your initial call is the same as with named args.)

hmm, you might be the 1st person ever, to try this from python, unfortunately.

any chance you still have the opencv build folder ? looking at build/modules/python_bindings_generator/pyopencv_generated_funcs.h (and finding the reconstruct() wrapper) might be helpful.

[themantalope]

Ok found it, it's interesting:

static PyObject* pyopencv_cv_sfm_reconstruct(PyObject* , PyObject* args, PyObject* kw)
{
    using namespace cv::sfm;

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    Mat Ps;
    PyObject* pyobj_points3d = NULL;
    Mat points3d;
    PyObject* pyobj_K = NULL;
    Mat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }
    PyErr_Clear();

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    UMat Ps;
    PyObject* pyobj_points3d = NULL;
    UMat points3d;
    PyObject* pyobj_K = NULL;
    UMat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }

    return NULL;
}

It appears that the order of the keyword arguments in the python wrapper don't align with the order of arguments (passed?) to the C++ sfm::reconstruct function. However, this doesn't have any bearing on functionality and another error is thrown. Based on what I see, it looks like the python wrapper correctly maps the python keyword arguments to the C++ arguments.

I'm wondering if the problem is that I'm using the latest version of OpenCV (4.0.0-pre) and OpenCV contrib modules, and the behavior of some of the core functions is a bit different.

[berak]

oh, that is really helpful !

It appears that the order of the keyword arguments in the python wrapper don't align with the order of arguments (passed?) to the C++ sfm::reconstruct function.

the python wrapper script has an internal rule for this:

• InputArray's are mandatory, and thus go to the front of the args list
• InputOutputArray's are mandatory, too, and also get returned, so they go in the middle
• OutputArrays (usually) do not need to get initialized, thus go to the end of the list (and get returned from the python function)

I'm wondering if the problem is that I'm using the latest version of OpenCV (4.0.0-pre)

i don't think so. imho, there's a bug in the python generator(script), and not much changed there between 3.4 and 4.0

but here's the bug !

    PyObject* pyobj_Ps = NULL;
    Mat Ps;

remember the errormsg ? it should be vector_Mat Ps, not Mat Ps here !
(vector_Mat is just some lube/typedef, to make it smoothly though the python based parser)

you can easily reproduce it with 4 lines of c++:

    Mat v; // should be vector<Mat> but isn't.
    OutputArray o(v); // entering reconstruct() from python wrappers
    o.create(5,1,5); // assuming: nviews==5, depth==CV_32F
    Mat m = o.getMatRef(0);

[themantalope]

@berak

Ok, so how does this get fixed? Which line of code in the reconstruct function should be changed?

[berak]

Ok, so how does this get fixed?

good one ;)

i have no (immediate) idea.
no, the problem is not one of the reconstruct() overloads (or in c++)

we'll have to dive in here: https://github.com/opencv/opencv/tree/master/modules/python/src2

https://docs.opencv.org/master/df/da2/tutorial_py_table_of_contents_bindings.html

and again, you seem to be the 1st person to try this from python, aka -- "our man on the moon" !

[themantalope]

Ok, so based on this issue, I changed the CV_EXPORTS macro in my reconstruct.hpp file to CV_EXPORTS_W:

CV_EXPORTS_W //changed to allow python bindings to be built
void
reconstruct(InputArrayOfArrays points2d, OutputArray Ps, OutputArray points3d, InputOutputArray K,bool is_projective = false);

Is there another macro that needs to be changed to the Ps gets converted to the proper object type?

I could also try to change which overloaded method of reconstruct gets exported....

[berak]

hmm, what if you change the c++ sig to:

void sfm::reconstruct(InputArrayOfArrays points2d, OutputArrayOfArrays Ps, OutputArray points3d, InputOutputArray K, bool is_projective);

( OutputArrayOfArrays should generate a vector instead of a single Mat (OutputArray) in python )

? (and rebuild the whole schlepp)

(same might be needed for points3d, idk.)

[themantalope]

@berak well the points3d should just be the points corresponding to the reconstructed 3d scene, so I would expect them to be output in one array. The Ps should be the [R|t] matricies corresponding to the (relative) rotations and translations of the camera for each picture used in reconstruction.

Once more, unto the breach (of recompiling)!

[berak]

yea, true.

all i was trying to point out was:

if the python generator script encounters an OutputArrayOfArrays , it will generate a vector<Mat> (correct for Ps), if it sees an OutputArray only it will generate a Mat (problem)

so, well, shift in the diagnosis again: imho, the c++ sig is wrong, so the py generator does the wrong thing

(and the Ps prob has to be mended, before going on to pts3d)

[themantalope]

Ok, so I made the changes suggested and rebuilt. There are now modifications to the pyopencv_generated_funcs.h file:

static PyObject* pyopencv_cv_sfm_reconstruct(PyObject* , PyObject* args, PyObject* kw)
{
    using namespace cv::sfm;

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    vector_Mat Ps;
    PyObject* pyobj_points3d = NULL;
    Mat points3d;
    PyObject* pyobj_K = NULL;
    Mat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }
    PyErr_Clear();

    {
    PyObject* pyobj_points2d = NULL;
    vector_Mat points2d;
    PyObject* pyobj_Ps = NULL;
    vector_Mat Ps;
    PyObject* pyobj_points3d = NULL;
    UMat points3d;
    PyObject* pyobj_K = NULL;
    UMat K;
    bool is_projective=false;

    const char* keywords[] = { "points2d", "K", "Ps", "points3d", "is_projective", NULL };
    if( PyArg_ParseTupleAndKeywords(args, kw, "OO|OOb:reconstruct", (char**)keywords, &pyobj_points2d, &pyobj_K, &pyobj_Ps, &pyobj_points3d, &is_projective) &&
        pyopencv_to(pyobj_points2d, points2d, ArgInfo("points2d", 0)) &&
        pyopencv_to(pyobj_Ps, Ps, ArgInfo("Ps", 1)) &&
        pyopencv_to(pyobj_points3d, points3d, ArgInfo("points3d", 1)) &&
        pyopencv_to(pyobj_K, K, ArgInfo("K", 1)) )
    {
        ERRWRAP2(cv::sfm::reconstruct(points2d, Ps, points3d, K, is_projective));
        return Py_BuildValue("(NNN)", pyopencv_from(Ps), pyopencv_from(points3d), pyopencv_from(K));
    }
    }

    return NULL;
}

And Psis now a vector mat. However, re-running the python code still gives the same error....

[themantalope]

The other weird thing I'm now noticing is in the example scene reconstruction C++ example:

  bool is_projective = true;
  vector<Mat> Rs_est, ts_est, points3d_estimated;
  reconstruct(images_paths, Rs_est, ts_est, K, points3d_estimated, is_projective);

Here they are creating the Rs and ts vectors of matricies, and using them in the reconstruct function. However even in that version of the reconstruct function, shouldn't they be OutputArrayOfArrays?

The function declaration in the header file reconstruct.hpp is as such:

CV_EXPORTS
void
reconstruct(const std::vector<String> images, OutputArray Rs, OutputArray Ts,
            InputOutputArray K, OutputArray points3d, bool is_projective = false);

Additionally the reconstruction example works on my machine. Not sure what the exact issue is here.

[themantalope]

Ok, strange behavior but I got it to work. I had to change the reconstruct.hpp file as such:

CV_EXPORTS_W //changed to allow python bindings to be built
void
reconstruct(InputArrayOfArrays points2d, OutputArrayOfArrays Ps, OutputArrayOfArrays points3d, InputOutputArray K, bool is_projective = false);

Seems like the object type of the input argument wasn't correct, and hence the python bindings were generating the incorrect object types for the function. I've updated the code on my forked version of the repo. @berak are you a maintainer for the sfm module in the opencv_contrib repo?

[berak]

no, i'm not the maintainer of this, i'm only out for a nice "detective story" , now & then.

do you feel apt to make a pr with the changes ? if so, -- please do ! (your bug, your fix, your honour.)

(and only 1 overload (the one marked CV_EXPORTS_W) will be wrapped into python/java/matlab, so only that one needs to be fixed, imho.)