3C loses struct definition as part of a field nested in another struct definition

[mattmccutchen-cci]

Example reduced from libarchive:

struct outer {
  struct inner {
    int x;
  } i_arr[1];
};

int foo(struct outer *o) {
  return o->i_arr[0].x;
}

3c -alltypes produces:

struct outer {
  struct inner i_arr _Checked[1];
};

int foo(_Ptr<struct outer> o) {
  return o->i_arr[0].x;
}

We lost the definition of struct inner, so we get the following compile error:

nested-struct.checked.c:2:30: error: array has incomplete element type 'struct inner'
  struct inner i_arr _Checked[1];
                             ^

I believe this also explains at least some of the "incomplete definition of type" errors in libarchive, though I haven't confirmed that. I haven't looked into whether there are other variants of the example that trigger the bug without using an array.

It looks like 3C has code somewhere that splits struct inner { ... } i_arr[1]; into struct inner { ... }; and struct inner i_arr[1]; when it occurs at the top level, but this doesn't work for nested structs.

[john-h-kastner]

I haven't looked into whether there are other variants of the example that trigger the bug without using an array.

Yep

struct outer {
  struct inner {
    int x;
  } *i;
};

becomes

struct outer {
  _Ptr<struct inner> i;
};

[mattmccutchen-cci]

I updated the issue title to reflect that there are examples that do not require array fields and thus do not require -alltypes.

[kyleheadley]

This problem may be worse that we thought:

struct outer {
  struct inner {int x;} *i;
} a[4];

converts to

struct outer {
  _Ptr<struct inne};
struct outer a _Checked[4];

which isn't even correct syntax, or use of the name "inner".

[mattmccutchen-cci]

It looks like several other error message templates that currently occur in our benchmarks may be caused by this issue when the nested struct type has no name (not to be confused with an "anonymous" struct where the variable has no name and the fields enter the surrounding scope). With this slight modification of the original example:

struct outer {
  struct inner {
    int x;
  } i_arr[1];
};

int foo(struct outer *o) {
  return o->i_arr[0].x;
}

3c -alltypes gives:

struct outer {
  // Now it looks like `i_arr` is the type name rather than the field name, so we get:
  // error: expected member name or ';' after declaration specifiers
  // error: array has incomplete element type 'struct i_arr'
  struct i_arr _Checked[1];
};

int foo(_Ptr<struct outer> o) {
  // error: no member named 'i_arr' in 'struct outer'
  return o->i_arr[0].x;
}

[mattmccutchen-cci]

Summary of discussion from today's meeting:

I suggested that one approach we could take that should completely avoid this problem is to move all struct definitions to the top level and leave the field declarations just referring to them by name. Kyle pointed out that 3C tries to minimize its edits to the user's code and this would be a dramatic edit, but Mike says that nested struct definitions could be considered a poor coding practice that 3C is justified in preemptively cleaning up.

As Kyle pointed out, the next implementation problem is that the Clang API makes it straightforward for us to get the proper source locations to rewrite the inner struct fields in place or to move the entire inner struct definition as a chunk of text, but it would be tricky to do both in the same rewriting pass. I proposed a workaround of performing an initial pass that just moves all struct definitions to the top level and then re-parsing all the source files so we have the correct source locations to rewrite the fields of the moved struct definitions. This could all be done within a single run of the 3C executable. (If we wanted, 3C could avoid moving structs that it won't need to rewrite by solving for the annotations, figuring out which structs it will need to rewrite, moving them, and then starting over.)

Further ideas from me after the meeting:

We could consider additionally addressing #542 by inventing names for unnamed struct definitions and moving them in the same way.

But I wonder if we could just fix the rewriter to bring the struct body along with the type name wherever it needs to go in the rewritten code for the first variable declaration. For a multi-variable declaration, the subsequent variables can reference the struct type by name. For example, 3C could convert this:

struct mystruct {
  int *x;
} (*f)(struct { int *y; } *p), b;

to this:

_Ptr<struct mystruct {
  _Ptr<int> x;
} (struct { _Ptr<int> y; } *p)> f;
struct mystruct b;

Yes, it's valid Checked C to define a named struct type inside a function pointer type like this! It looks crazy, but the user is the one who decided to combine the struct and variable definitions in the first place. 🙂 This could work the same way whether the code is at the top level or nested in an outer struct. The same approach would work for single-variable declarations using unnamed struct types (in effect, automating in 3C what the 3C warning in #542 currently tells the user to do manually) and even for multi-variable declarations if we invent a name for the struct type.

Update: I realize that the "keep the body with the name" approach has a similar problem as the "move to the top level" approach in that we may have to rewrite the fields of the struct as well as the surrounding declaration, and it's unclear whether it would be feasible to break this into multiple passes as I proposed for the "move to the top level" approach. In simpler examples like the one above, we might be able to fiddle with the syntax surrounding the struct definitions while preserving the source locations of the struct fields so we can rewrite them in parallel. (In this case, we would add the _Ptr< before struct mystruct, delete the (*f), and add the > f after *p).) But this breaks down in extreme cases with multi-level function pointers, in which we may need to reverse the order of the inner and outer parameter lists. For example, this:

struct mystruct2 {
  int *x;
} (*(*g)(struct { double *z; } *q))(struct { int *y; } *p);

might have to become this:

_Ptr<_Ptr<struct mystruct2 {
  _Ptr<int> x;
} (struct { _Ptr<int> y; } *p)> (struct { _Ptr<double> z; } *q)> g;

Note that the order of y and z was reversed and we also rewrote both of their types. Realistically, if we want to produce valid output for these extreme inputs, we probably need to implement the ability to move struct definitions to the top level, and then it may not be worth the extra work to implement the "keep the body with the name" approach as an alternative for the simpler inputs.

[mattmccutchen-cci]

The latest plan is that I'm going to work on de-nesting structs, not Kyle.