order of board.chessboardCorners != order of aruco.interpolateCornersCharuco

[edmund3333]

System information (version)

• OpenCV => opencv-contrib-python 4.3.0.36
• Operating System / Platform => Win 10
• Env: Python 3.7

Detailed description

not sure if its a bug, but when calling "aruco.detectMarkers" i get the correct order of markers but after calling "aruco.interpolateCornersCharuco" the order of the chessboard corners has changed and is not consitent with the corner order i get with "board.chessboardCorners".
This is why i cannot call "cv2.findHomography" with the chessboardCorners from "aruco.interpolateCornersCharuco" and "board.chessboardCorners".

In the attached image "charuco.jpg" you can see the blue marker ids and the yellow chessboard ids and that the chessboard ids start on the other side of the charuco board.

I think the chessboard ids should start at the same side as the marker ids and the "board.chessboardCorners" ids.

Steps to reproduce

import cv2
parameters = cv2.aruco.DetectorParameters_create()
#creat board
aruco_dict = cv2.aruco.Dictionary_get(cv2.aruco.DICT_4X4_50)
board = cv2.aruco.CharucoBoard_create(11, 8, 40, 30, aruco_dict)

img = cv2.imread(r'cam1_img1.png')

img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
marker_corners_c, marker_ids, rejectedImgPoints = cv2.aruco.detectMarkers(img_gray, aruco_dict, parameters=parameters)
cv2.aruco.drawDetectedMarkers(img, marker_corners_c, marker_ids)
cv2.aruco.drawDetectedMarkers(img, rejectedImgPoints, borderColor=(100, 0, 240))

[retval, chessboard_corners_c, chessboard_ids_c] = cv2.aruco.interpolateCornersCharuco(marker_corners_c, marker_ids, img_gray, board)

cv2.aruco.drawDetectedCornersCharuco(img,chessboard_corners_c, chessboard_ids_c,  cornerColor=(0, 215, 255))
cv2.imshow('camera_image', img)
cv2.waitKey(0)

#board image

#board image with markers(blue) and chessboard corners (yellow) (the ids start with 0 at different positions on the board)

[edmund3333]

i found out that this also effects "cv2.aruco.calibrateCameraCharuco" because inside this function _board->chessboardCorners[pointId] is called to find the object corners of the chessboard.
https://github.com/opencv/opencv_contrib/blob/dfb6d9bf7164fb64a045d96335c7f27fc83bd4a1/modules/aruco/src/charuco.cpp

[edmund3333]

it seems that the order of markers is not currect in the "CharucoBoard::create" method.

i created a new board with "cv2.aruco.CharucoBoard_create" and then plotted the "board.chessboardCorners" with the index of their position in the list and the "board.objPoints" with their "board.ids" as id and got following result. It seems that the marker positions are flipped around the horizontal middle line of the image.

Steps to reproduce:

import numpy as np
aruco_dict = cv2.aruco.Dictionary_get(cv2.aruco.DICT_4X4_50)
board = cv2.aruco.CharucoBoard_create(11, 8, 40, 30, aruco_dict)
img_board = board.draw((440, 320), marginSize=0, borderBits=1)

img_board_color = cv2.cvtColor(img_board, cv2.COLOR_GRAY2BGR)

cs_corners_o = np.array([np.array([[a[0], a[1]]]) for a in board.chessboardCorners])
cs_corner_ids = np.array([[a] for a in np.arange(cs_corners_o.shape[0])])
cv2.aruco.drawDetectedCornersCharuco(img_board_color, cs_corners_o, cs_corner_ids, cornerColor=(0, 215, 255))

objPoints = []
for marker in board.objPoints:
newPoint = np.array([[[a[0], a[1]] for a in marker]])
objPoints.append(newPoint)

cv2.aruco.drawDetectedMarkers(img_board_color, objPoints, board.ids)

cv2.imshow('img_board_color', img_board_color)

cv2.waitKey(0)

[edmund3333]

discussion https://stackoverflow.com/questions/63210515/homography-from-charuco-image

[edmund3333]

to clarify:

"CharucoBaord_create" starts with the charuco marker idx 0 at the bottom left and increases to the top right. The charuco corners start with idx 0 at the top left and increase to the bottom right.

"detectMarkers" starts with the charuco marker idx 0 at the top left and increases to the bottom right. The charuco corners start with idx 0 at the bottom left and increase to the top right.

import cv2
import numpy as np

squaresX = 11
squaresY = 7


aruco_dict = cv2.aruco.Dictionary_get(cv2.aruco.DICT_4X4_50)
board = cv2.aruco.CharucoBoard_create(squaresX, squaresY, 40, 30, aruco_dict)
img_board = board.draw((40*squaresX, 40*squaresY), marginSize=0, borderBits=1)

# plot board_data
cs_corners_o = np.array([np.array([[a[0], a[1]]]) for a in board.chessboardCorners])
cs_corner_ids = np.array([[a] for a in np.arange(cs_corners_o.shape[0])])

img_board_color = cv2.cvtColor(img_board, cv2.COLOR_GRAY2BGR)
cv2.aruco.drawDetectedCornersCharuco(img_board_color, cs_corners_o, cs_corner_ids, cornerColor=(0, 255, 0))

# remove third dimension from points
objPoints = []
for marker in board.objPoints:
    newPoint = np.array([[[a[0], a[1]] for a in marker]])
    objPoints.append(newPoint)

cv2.aruco.drawDetectedMarkers(img_board_color, objPoints, board.ids)
cv2.imshow('data from: CharucoBoard_create', img_board_color)

# plot detected data
img_board_color2 = cv2.cvtColor(img_board, cv2.COLOR_GRAY2BGR)
img_bord_gray = cv2.cvtColor(img_board_color2, cv2.COLOR_BGR2GRAY)

parameters = cv2.aruco.DetectorParameters_create()
marker_corners_c, marker_ids, rejectedImgPoints = cv2.aruco.detectMarkers(img_bord_gray, aruco_dict, parameters=parameters)
cv2.aruco.drawDetectedMarkers(img_board_color2, marker_corners_c, marker_ids)

[retval, chessboard_corners_c, chessboard_ids_c] = cv2.aruco.interpolateCornersCharuco(marker_corners_c, marker_ids, img_bord_gray, board)
cv2.aruco.drawDetectedCornersCharuco(img_board_color2, chessboard_corners_c, chessboard_ids_c,  cornerColor=(0, 0, 255))

cv2.imshow('data from: drawDetectedMarkers', img_board_color2)

cv2.waitKey(0)

reason why the order of the charuco corners is wrong:
In the "_drawPlanarBoardImpl" function the y coordiantes of the charuco markers get horizontally mirrored by "pf.y = (1.0f - pf.y / sizeY) * float(out.rows);" but the charuco corner positions are not updated to this change.

code from void _drawPlanarBoardImpl in aruco.cpp:

        // transform corners to markerZone coordinates
        for(int j = 0; j < 3; j++) {
            Point2f pf = Point2f(_board->objPoints[m][j].x, _board->objPoints[m][j].y);
            // move top left to 0, 0
            pf -= Point2f(minX, minY);
            pf.x = pf.x / sizeX * float(out.cols);
            pf.y = (1.0f - pf.y / sizeY) * float(out.rows);
            //pf.y = pf.y / sizeY * float(out.rows);
            outCorners[j] = pf;
        }