Latest updates to Checked C Convert tool

[Machiry]

This pull request includes a lot of changes to cconv tool.

We fixed various issues, added a lot (318) of test cases, improved the reliability and accuracy of the tool done over past 2 months.

[Machiry]

@dtarditi We removed the dependency of boost. This pull request is ready for review.

[mgrang]

Thanks for removing the dependency on boost. LGTM.
@dtarditi Would you like to take a final look at the PR?

[dtarditi]

I ran our automated testing for this PR and testing for Window x86 failed. 245 of the converter tests failed. There are errors involving confusion of long and int. There are also some improper type casts. It looks like some problems occur when running the tool and some problems occur when compiling the output of the tool.

2020-08-26T23:47:24.7721050Z ********************
2020-08-26T23:47:24.8170989Z Testing: 0 ..
2020-08-26T23:47:24.8172881Z FAIL: Clang :: CheckedCRewriter/arrcallee.c (1658 of 15640)
2020-08-26T23:47:24.8174232Z ******************** TEST 'Clang :: CheckedCRewriter/arrcallee.c' FAILED ********************
2020-08-26T23:47:24.8175413Z Script:
2020-08-26T23:47:24.8176074Z --
2020-08-26T23:47:24.8178098Z : 'RUN: at line 1'; cconv-standalone -alltypes D:\work\10\s\clang\test\CheckedCRewriter\arrcallee.c -- | d:\work\10\b\llvm-debug-x86-windows.obj\bin\filecheck.exe -match-full-lines -check-prefixes="CHECK_ALL","CHECK" D:\work\10\s\clang\test\CheckedCRewriter\arrcallee.c
2020-08-26T23:47:24.8181332Z : 'RUN: at line 2'; cconv-standalone D:\work\10\s\clang\test\CheckedCRewriter\arrcallee.c -- | d:\work\10\b\llvm-debug-x86-windows.obj\bin\filecheck.exe -match-full-lines -check-prefixes="CHECK_NOALL","CHECK" D:\work\10\s\clang\test\CheckedCRewriter\arrcallee.c
2020-08-26T23:47:24.8184359Z : 'RUN: at line 3'; cconv-standalone D:\work\10\s\clang\test\CheckedCRewriter\arrcallee.c -- | d:\work\10\b\llvm-debug-x86-windows.obj\bin\clang.exe -c -fcheckedc-extension -x c -o /dev/null -
2020-08-26T23:47:24.8185903Z --
2020-08-26T23:47:24.8186677Z Exit Code: 1
2020-08-26T23:47:24.8187278Z
2020-08-26T23:47:24.8188047Z Command Output (stdout):
2020-08-26T23:47:24.8188766Z --
2020-08-26T23:47:24.8189527Z $ ":" "RUN: at line 1"
2020-08-26T23:47:24.8190650Z $ "cconv-standalone" "-alltypes" "D:\work\10\s\clang\test\CheckedCRewriter\arrcallee.c" "--"
2020-08-26T23:47:24.8191830Z # command stderr:
2020-08-26T23:47:24.8193272Z D:\work\10\s\clang\test\CheckedCRewriter\arrcallee.c:18:23: error: typedef redefinition with different types ('unsigned long' vs 'unsigned int')
2020-08-26T23:47:24.8194541Z
2020-08-26T23:47:24.8195263Z typedef unsigned long size_t;
2020-08-26T23:47:24.8195986Z
2020-08-26T23:47:24.8196682Z ^
2020-08-26T23:47:24.8197376Z
2020-08-26T23:47:24.8198050Z 1 error generated.

[Machiry]

@dtarditi I think we fixed the issue. Can you please rerun the Windows tests?

[dtarditi]

@Machiry, I emailed you the results of automated testing on Friday against your most recent changes. I saw a bunch of failures that indicate the clang-cl.exe is being invoked on Windows. This is a compiler driver that emulates the ‘cl’ compiler command-line form Visual Studio. The -fcheckedc-convert-tool flag is being passed. There’s a special escape mechanism for flags that need to be passed through clang-cl to the underlying clang driver: -Xclang. So you need to do something like -Xclang -fcheckedc-convert-tool. You can run clang-cl /? To see the command line option.

2020-08-28T20:18:06.2051364Z -- Testing: 16024 tests, 24 threads --
2020-08-28T20:19:38.3894469Z Testing: 0 ..
2020-08-28T20:19:38.3898080Z FAIL: Clang :: CheckedCRewriter/arrbothmulti1.c (1662 of 16024)
2020-08-28T20:19:38.3904266Z ******************** TEST 'Clang :: CheckedCRewriter/arrbothmulti1.c' FAILED ********************
2020-08-28T20:19:38.3925579Z Script:
2020-08-28T20:19:38.3926357Z --
2020-08-28T20:19:38.3928102Z : 'RUN: at line 1'; cconv-standalone -base-dir=D:\work\10\s\clang\test\CheckedCRewriter -alltypes -output-postfix=checkedALL D:\work\10\s\clang\test\CheckedCRewriter\arrbothmulti1.c D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti2.c
2020-08-28T20:19:38.3930889Z : 'RUN: at line 2'; cconv-standalone -base-dir=D:\work\10\s\clang\test\CheckedCRewriter -output-postfix=checkedNOALL D:\work\10\s\clang\test\CheckedCRewriter\arrbothmulti1.c D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti2.c
2020-08-28T20:19:38.3933615Z : 'RUN: at line 3'; d:\work\10\b\llvm-debug-x86-windows.obj\bin\clang.exe -c D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti1.checkedNOALL.c D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti2.checkedNOALL.c
2020-08-28T20:19:38.3936725Z : 'RUN: at line 4'; d:\work\10\b\llvm-debug-x86-windows.obj\bin\filecheck.exe -match-full-lines -check-prefixes="CHECK_NOALL" --input-file D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti1.checkedNOALL.c D:\work\10\s\clang\test\CheckedCRewriter\arrbothmulti1.c
2020-08-28T20:19:38.3940223Z : 'RUN: at line 5'; d:\work\10\b\llvm-debug-x86-windows.obj\bin\filecheck.exe -match-full-lines -check-prefixes="CHECK_ALL" --input-file D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti1.checkedALL.c D:\work\10\s\clang\test\CheckedCRewriter\arrbothmulti1.c
2020-08-28T20:19:38.3943018Z : 'RUN: at line 6'; rm D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti1.checkedALL.c D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti2.checkedALL.c
2020-08-28T20:19:38.3945185Z : 'RUN: at line 7'; rm D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti1.checkedNOALL.c D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti2.checkedNOALL.c
2020-08-28T20:19:38.3946975Z --
2020-08-28T20:19:38.3947583Z Exit Code: 2
2020-08-28T20:19:38.3948162Z
2020-08-28T20:19:38.3948797Z Command Output (stdout):
2020-08-28T20:19:38.3949474Z --
2020-08-28T20:19:38.3950086Z $ ":" "RUN: at line 1"
2020-08-28T20:19:38.3952103Z $ "cconv-standalone" "-base-dir=D:\work\10\s\clang\test\CheckedCRewriter" "-alltypes" "-output-postfix=checkedALL" "D:\work\10\s\clang\test\CheckedCRewriter\arrbothmulti1.c" "D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti2.c"
2020-08-28T20:19:38.3953774Z # command stderr:
2020-08-28T20:19:38.3954784Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3955617Z
2020-08-28T20:19:38.3956612Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3957602Z
2020-08-28T20:19:38.3958508Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3959312Z
2020-08-28T20:19:38.3960251Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3961076Z
2020-08-28T20:19:38.3961994Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3962940Z
2020-08-28T20:19:38.3963896Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3964723Z
2020-08-28T20:19:38.3965138Z
2020-08-28T20:19:38.3965749Z $ ":" "RUN: at line 2"
2020-08-28T20:19:38.3967369Z $ "cconv-standalone" "-base-dir=D:\work\10\s\clang\test\CheckedCRewriter" "-output-postfix=checkedNOALL" "D:\work\10\s\clang\test\CheckedCRewriter\arrbothmulti1.c" "D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti2.c"
2020-08-28T20:19:38.3968972Z # command stderr:
2020-08-28T20:19:38.3969962Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3970793Z
2020-08-28T20:19:38.3971705Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3972544Z
2020-08-28T20:19:38.3973419Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3974296Z
2020-08-28T20:19:38.3975173Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3976044Z
2020-08-28T20:19:38.3976924Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3977784Z
2020-08-28T20:19:38.3978670Z warning: unknown argument ignored in clang-cl: '-fcheckedc-convert-tool' [-Wunknown-argument]
2020-08-28T20:19:38.3979565Z
2020-08-28T20:19:38.3980074Z
2020-08-28T20:19:38.3980728Z $ ":" "RUN: at line 3"
2020-08-28T20:19:38.3982357Z $ "d:\work\10\b\llvm-debug-x86-windows.obj\bin\clang.exe" "-c" "D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti1.checkedNOALL.c" "D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti2.checkedNOALL.c"
2020-08-28T20:19:38.3983991Z $ ":" "RUN: at line 4"
2020-08-28T20:19:38.3985835Z $ "d:\work\10\b\llvm-debug-x86-windows.obj\bin\filecheck.exe" "-match-full-lines" "-check-prefixes=CHECK_NOALL" "--input-file" "D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti1.checkedNOALL.c" "D:\work\10\s\clang\test\CheckedCRewriter\arrbothmulti1.c"
2020-08-28T20:19:38.3987651Z # command stderr:
2020-08-28T20:19:38.3988858Z Could not open input file 'D:\work\10\s\clang\test\CheckedCRewriter/arrbothmulti1.checkedNOALL.c': no such file or directory
2020-08-28T20:19:38.3989912Z
2020-08-28T20:19:38.3990386Z
2020-08-28T20:19:38.3991178Z error: command failed with exit status: 2
2020-08-28T20:19:38.3991956Z
2020-08-28T20:19:38.3992646Z --

[Machiry]

@dtarditi Fixed the warnings and verified that all tests pass on a Windows machine. Could you please check now?

[dtarditi]

This has passed automated testing. Thanks!