XDP Hints Support through Device Binding

[jalalmostafa]

Closes #493

[jalalmostafa]

Hey @tohojo

The PR to support XDP hints is ready for review. The bpf.h has been updated to that of kernel 6.3 to get the BPF_F_XDP_DEV_BOUND_ONLY definition.

[tohojo]

Hmm, so I don't think the same approach we use for frags will work for the dev_bound flag, because the kernel treats it differently:

For frags, the kernel only considers the flag for the dispatches and doesn't care about any freplace programs that are attached to that main XDP program. Meaning that this is completely under the control of libxdp, we can propagate the frags flag between the component programs and the dispatcher as we see fit, and the same program can be attached to both an XDP program that sets the frags flag, and one that doesn't.

For the dev_bound flag, the kernel explicitly handles freplace programs. It does this, by forcing the freplace program to inherit the dev_bound settings of the XDP program it is being attached to at load time. Meaning that it's irreversible for the lifetime of the program. So for a program to be dev_bound, it has to be attached to a dev_bound dispatcher the first time it is loaded, we can't add it later. And the flag itself has no meaning when set at freplace program load time, only the setting when loading the dispatcher matters.

This means we don't really have to track the flag setting of each program after load (so no need to store the flag in the dispatcher data structure for every program), since it will never be unset again. I think we have a couple of options for how to handle enabling dev_bound functionality:

1. Don't track it, just propagate the dev_bound flag to the dispatcher. Any subsequent program loaded without the flag will then be automatically dev_bound because of the inheritance in the kernel.
2. As above, but add tracking to the dispatcher so we can check if there's a program loaded already that is dev_bound, reject loading another one that isn't.

If there's an existing program loaded that is not dev_bound, the operation will fail in any case. But we also disallow mixing programs with and without the flag set for the frags case, so that's consistent.

If we go with 1., that will prevent a non-devbound program from being attached to more than one interface if the first one it attaches to has another devbound program, which will probably be surprising. I think 2. is most consistent with how we handle frags anyway.

The last question is whether this should be an attach time argument (flag to xdp_program__attach()), or a "program flag" the way you're doing it currently. The latter is more consistent with frags, but kinda has an impedence mismatch with how the kernel treats things because of the inheritance. So maybe having the difference reflected in the API is actually better? In any case, if we go with the set_dev_bound() type function, supplying the ifindex when setting the flag doesn't make sense, this will be overridden by the attach ifindex anyway. Which also means it may make more sense to make it an attach-time flag. WDYT?

[jalalmostafa]

Hi @tohojo ,
I got busy with my PhD thesis lately.

I agree with you: the better approach is to have an attach flag since it will be done once in the lifetime of the program and to reject to load any programs that mismatch with the loaded dispatcher.

I made a new draft of this PR that you can please review. The draft will propagate the device binding flag of the dispatcher from the first attached program, any subsequent program attachments will be rejected if they mismatch with the running dispatcher.