Change default default qdisc from fq_codel to sch_fq

[roland-bless]

systemd version the issue has been seen with

e6c253e

Used distribution

Ubuntu 18.04

Expected behaviour you didn't see

Better performance than sch_fq

Unexpected behaviour you saw

Packet loss at the sender (ok, not really unexpected)

Steps to reproduce the problem
The main problem is that fq_codel is nice for routers or forwarding devices, but not for end-systems or servers (and I assume that this is rather the majority of linux installations): it doesn't provide an advantage there, but a serious drawback: packet loss at the sending end-system (note that this will also increase the overall latency for the affected TCP connections!). Normally, this should be avoided by using back-pressure mechanisms locally. However, CoDel drops packets instead of propagating back-pressure.
The steps to reproduce the problems are described here:
https://lists.bufferbloat.net/pipermail/bloat/2018-June/008318.html
(BTW: there is no technical argument in this thread for using fq_codel as default).
So if you're using Linux for a software-based router (openwrt), you'll need a special configuration anyway (so it's easy to set fq_codel as default there), but for servers and normal end-systems/hosts the default should be sch_fq, not fq_codel!

[poettering]

@michich opinion on this?

[roland-bless]

Please note that the Bufferbloat wiki also recommends sch_fq for servers:
https://www.bufferbloat.net/projects/codel/wiki/
(see heading "Binary code and kernels for Linux based operating systems").
However, as I wrote: the default shipping Linux code should be for end-systems and CoDel doesn't make any sense inside an end-system (flow queueing is useful though!).

[dtaht]

@michich @poettering

um. fq_codel is the most general purpose qdisc there is handling udp and tcp traffic rationally and equally, while managing queue length, and working with routers, and virtual network substrates also.

sch_fq is for tcp-serving heavy workloads, primarily from the datacenter, and does nothing sane for other forms of traffic including acks in the reverse direction. IF all you are doing is tcp serving at 10gigE+, sch_fq is totally the right thing. Otherwise it can actually be worse than pfifo_fast! At 1gigE, sch_fq's defaults are set too high (for example) for even your typical nas or laptop.

One of the big benefits to sch_fq (pacing) arrived for all tcp connections in recent kernels so now fq_codel takes advantage of pacing also.

My vote is you keep fq_codel as the default, and I'll try to clarify the referenced bufferbloat page, and show some benchmarks as to why.

I'd also made some lengthy comments on your ecn enablement side on another bug report.

[dtaht]

#9748 on that front. @roland-bless I have no idea how you are turning bufferbloat.net's general recomendations around. The best all-round general purpose default for linux remains fq_codel. tcp serving from the dc is not what you want to optimize your typical linux distro user for.

[dtaht]

ah, I see the thread: https://lists.bufferbloat.net/pipermail/bloat/2018-June/thread.html which is all over the place, and I'd had pithy comments.

one of the problems with your 100 flow test is in that case, current linux tcps cap cwnd reductions so they never get the right rate anymore at gigE speeds, lots of flows, and short rtts. In fact, fractional cwnds would be nice... But you'll find the achieved bandwidth is exactly the same (despite the retransmits) but sch_fq .caps will have an ever-expanding rtt.

You want things like fq_codel to regulate that behavior, not just for tcp, but things like webrtc, which doesn't run over tcp. All traffic, not just tcp serving workloads.

[cherusk]

Technically, are there cues that there has to be the ONE. Why not converging to the compromise that every distro could isolate its default choice? Most distros are even churned out in explicit flavours for e.g. back-end or desktop use cases ...

[roland-bless]

@dtaht

um. fq_codel is the most general purpose qdisc there is handling udp and tcp traffic rationally and equally, while managing queue length, and working with routers, and virtual network substrates also.

fq_codel is great for forwarding devices that can easily drop on dequeue, but that's not the point. The point is that it's not reasonable to loose the backpressure signal in sending end-systems by using fq_codel.

sch_fq is for tcp-serving heavy workloads, primarily from the datacenter, and does nothing sane for other forms of traffic including acks in the reverse direction.

It also favors new/small flows like fq in fq_codel, so the benefit for those packets is the same as in fq_codel.

With respect to the test case: 100 concurrent flows from a 1GigE web server isn't unrealistic . It shows that fq_codel drops outgoing packets, because TCP CC doesn't work here so well either since the load in number of flows is the problem. sch_fq doesn't have to drop packets, because of propagating backpressure to applications locally. Throughput and latency are comparable, and sch_fq works quite well even in this scenario (we didn't adjust any sch_fq parameters).

You want things like fq_codel to regulate that behavior, not just for tcp, but things like webrtc, which doesn't run over tcp. All traffic, not just tcp serving workloads.

That's not the point, because you want to use backpressure in the sending end-system. sch_fq works also for UDP or other traffic. One point is that in the mentioned scenario, fq_codel cannot even control the TCP flows so well, so the same would happen with UDP flows (which should also be congestion controlled).

[dtaht]

Roland, you are wrong on multiple fronts here. I don't really want to take the time to exaustively write this - it's that your viewpoint is "I'm the web server in the datacenter" that sticks in my craw. What can I do to convince you my points are valid? What concrete set of experiments?

Systemd runs on everything from iot to laptops to virtual hosts to servers. Having a good default that works across the widest range is what we want. "I'm the laptop on the edge of the network."

"sch_fq works also for UDP or other traffic. "

No, it doesn't manage queue length in that case. Next question. That really should be the end of this debate!!!

"One point is that in the mentioned scenario, fq_codel cannot even control the TCP flows so well"

Look at the rtts and throughput in both cases. Try using the flent tool to sample rtts. Things are being controlled, the fact the local tcp cannot in this case match the rate is more the fault of today's tcps. [1] I miss reno sometimes.

" so the same would happen with UDP flows (which should also be congestion controlled)."

"can" happen. When we are pushing more data out than we can match rates on, what do you want to happen? grow the buffer? Have backpressure for udp? You don't have backpressure through the whole system , you have drop, that's it.

0. fq_codel, so far as I know, has been the systemd default for years now. I'm not aware of many complaints. It's way better than pfifo_fast at rates from 1mbit to 40gigE+.

1. Take caps of sch_fq. Look at your rtts.

2. sch_fq does not apply backpressure to anything besides local tcp traffic and not enough of it at low rates either. Switching to sch_fq would bring back the era of 10,000 packet buffers for all other kinds of traffic, no regulation of queue size at all. It would be bufferbloat^10. It won't do ecn. It won't handle quic. nor vms. nor encapsulated traffic. Is that what you want?

fq_codel applies global regulation when each application only has it's own narrow viewpoint.

3. TSQ only applies to a limited number of flows before self congesting. Last I recall it always stacked up 4 packets per flow. It works really well for low numbers of flows and/or longer paths. TSQ + sch_fq works spectacularly for lots of tcp flows, 10+gigE, and long paths.

sch_fq is great for serving tcps in the datacenter. It may even be a good choice for your local web server example. Although that example is flawed, using only 100 greedy, continuously running flows. A real webserver handles a variety of flows from 1 packet up to (last I checked) about 2Mbytes in size. A much better benchmark of web-server - nfs server - etc is in transactions per sec across a workload, where most of the workload lives in slow start. The vast majority of flows, being rtt-bound in their ramp, or size limited, will never exceed their fair share and get hit by codel.

I'd like to clearly establish where we are disagreeing or not. You've made several assertions that are provably false. Others, like the amount of backpressure needed, or your distaste for retransmits on a short path where it does not matter... are something of a matter of taste. My "taste" leans towards never having much more than 5ms local buffering, except for what is needed to handle bursts. There's buffering in the socket (see tcp_sent_lowwat), in the app, in the qdisc, and in the network.

[1] You can make an argument that fq_codel should interact with the local tcp stack better than it currently does, under conditions of extreme load. Or you can argue the local tcp stack should shrink it's demands better. You can also argue that bql tends towards overbuffering also.

But I think first up, would be looking into how tsq and sch_fq currently interact at 1gbit rates. Because of this whole thread I did go looking at 100+flow tests on short paths, at bbr's current behaviors, and at ecn, while looking at cake, sch_fq, and q_codel, but I didn't get iproute2-next to let me sample buffer sizes again until a commit arrived for fixing, ironically, it's overbuffered output buffer!

[roland-bless]

Dave: my point is not about web-servers in a data center, it's about using fq_codel in the end-system, which can cause harm. So we have the case where the sender is the only bottleneck. Why should I abuse congestion control signals and add RTT latency when I can manage this locally? Congestion control should prevent overloading the network, but it's not adequate to manage overload in the end-system. In our test case you can see that congestion control doesn't work, while local backpressure mechanisms do work.

What can I do to convince you my points are valid? What concrete set of experiments?

Show me a case where fq_codel performs significantly better than sch_fq in an end-system.

...is more the fault of today's tcps

Basically, we have AQMs to deal with the fault of today's TCPs... :-)

No, it doesn't manage queue length in that case.

Hm, so flow isolation still works, but UDP flows may suffer from self-inflicted queuing delay
(due to lack of TSQ)?

Look at the rtts and throughput in both cases.

Both were fine, as described.

your distaste for retransmits on a short path where it does not matter

This could also be retransmits on a long path, where they do matter. So, yes, my main point is that we should use local backpressure instead of congestion signals in order to avoid these retransmits.

[dtaht]

No, it doesn't manage queue length in that case.

Hm, so flow isolation still works, but UDP flows may suffer from self-inflicted queuing delay
(due to lack of TSQ)?

Flow isolation still works, but it's a second class citizen in sch_fq. Queue length management reverts to tail drop on a 10,000 packet (and GSO enabled, thus 64k possible per packet) queue.

All flows not directly managed by the local tcp stack do not get backpressure. This includes tcp flows from vms, stuff flowing through hypervisors, encapsulated traffic from vpns and containers and network name spaces, udp from any source including quic, webrtc, voip, dns, gaming, and attack traffic, or any other protocol. So any of these flows can self inflict queuing delay, and by being present still inflict some delay on other flows.

To me this ends the debate over sch_fq as a good default! Does it work on you yet?

What can I do to convince you my points are valid? What concrete set of experiments?
Show me a case where fq_codel performs significantly better than sch_fq in an end-system.

OK, well, that depends on what you consider as a valid test. Would a MOS score of voip flows taken against an also tcp_loaded server work? Or a measurement of self inflicted latency from webrtc? or locally vpn'd traffic competing with local traffic? Bittorrent?

"end-system" partially depends on whether you are a client or server, on wifi or ethernet. Certainly the fq_codel queue management we did for wifi routers also applies to clients and we should probably do a followup on that paper running the same algo on both sides, ( https://arxiv.org/pdf/1703.00064.pdf )

Our viewpoint as to "performs" might differ. I'm all about low latency and filling the pipe not the queue. Forcing tcp to back off and retransmit "does no harm" so long as utilization is 100%.

your distaste for retransmits on a short path where it does not matter

This could also be retransmits on a long path, where they do matter. So, yes, my main point is that we >should use local backpressure instead of congestion signals in order to avoid these retransmits.

Just to clearly establish things for those in the audience, retransmits "fill in the hole". If your rtt is short, the signal gets through faster. retransmits do cause additional work on behalf of both sides of the stack.

"where they do matter"

It only matters if the remaining duration of your transaction is less than the RTT, or you've dropped the last packet in flight, forcing an RTO.

as the rtt grows, the need to do congestion control via dropping declines dramatically. One drop means a lot. So we could repeat your 100 flow test over a 100ms or 1000ms rtt, to show that (or we could dig up a paper on relationship between drop rates and rtt)

Yes, I totally agree we should use local backpressure to avoid retransmits whenever possible. pfifo_fast does tail drop, which is not actually dropped, but pushes a cwnd reduction into it and reposts the packet. sch_fq + TSQ applies local backpressure and cwnd (and even better, pacing) (but does grow the self inflicted apparent rtt). fq_codel does head drop, which means you generally won't see an RTO, as the flow's next packet immediately behind it is delivered, thus the receiving tcp notices the hole and asks that it gets filled in.

we did produce a tail dropping version of fq_codel at one point. yep, local tcp stack backpressure. But it hurt all other applications (as noted above), that can't observe that backpressure, to not get the earliest congestion signal possible. It matters to voip/video/dns to drop the stalest packet, in particular, and tcps in general not synchronized bulk drop (as what happens when drop tail is highly congested) Also, often, when fq_codel is in a dropping state the local total queue is so full in the first place that by the time it empties it's already got the ack from the other side, indicating please reduce your window and fill in this hole - there's often (on short paths) waaay more than a bdp in there when overloaded in the first place.

I think we are both in agreement that having min/max fair fq (sch_fq, fq_codel, and now sch_cake) is better than a fifo queue?

...is more the fault of today's tcps
Basically, we have AQMs to deal with the fault of today's TCPs... :-)

Even if I could step back to 1986 and mandate a delay, rather than drop based tcp, it wouldn't have worked in todays highly variable rtt environment. The need for aqm was understood, and if only red had worked - or drr/sfq applied more universally, we'd have had a better internet. I got involved in all this because I ran an ISP in 1993-96 and still have the scars on my back...

If I could step back to 2000 and reserve 3 bits for QoS and 5 for ecn, instead of the diffserv mess, that would have helped.

I'm pretty sure, if I could go back to 1989 and the first ietf meeting, and stood shoulder to shoulder with John Nagle about the need for fair queuing everywhere, that would have made a difference! Most congestion would thus be self inflicted and applications could just do themselves in.

Dave: my point is not about web-servers in a data center, it's about using fq_codel in the end-system, >which can cause harm.

What harm? Retransmits and congestion control are totally normal highly optimized aspects of tcps behavior.

No harm, and a general benefit. A light tap here and there to reduce self inflicted congestion in the general case.

So we have the case where the sender is the only bottleneck. Why should I abuse congestion control >signals and add RTT latency when I can manage this locally? Congestion control should prevent >overloading the network, but it's not adequate to manage overload in the end-system. In our test >case you can see that congestion control doesn't work, while local backpressure mechanisms do >work.

Our definition of working is different here. You want no retransmits, and effective backpressure. I'm saying that effective backpressure is impossible in the general case.

Utilization is 100% in both cases. the same amount of data is transferred. Locally observed RTT with fq_codel is lower (probably - I'd have to go look as I mentioned earlier, it's a per-flow curve with tsq in place, and pacing now helps a lot and I have only last week got iproute2 fixed ). As I noted on your test case, it's not an example of a web workload, either.

It's "working".

I too want effective backpressure. I'd like it if modern tcps were IW4, not IW10, that cubic backed off .5 rather than .7, that tcps still reduced to minimal cwnd AND that since fractional cwnd is impossible, used pacing instead, and that tcps reduced their mss size when under congestion also.

I would not mind at all if the head dropping of packets in fq_codel immediately forced a cwnd or pacing reduction locally. Hmm... that might be feasible... similarly a local response to seeing ecn excerted... this is not stuff the main folk working at the big g care about.

and add RTT latency when I can manage this locally

so far as I know - and like I said, I have to go look as it has been a while, sch_fq + TSQ add RTT latency at a minimum of 4 packets outstanding locally per flow. (I will revise this statement after doing the work, but I did ask you to take packet caps of sch_fq in your benchmark). I would not mind if it dropped to 1 packet, nor would I mind if it then started reducing packet size - but the DC guys simply don't see problems at 1gigE and below - they would generally love to be able to self congest but are otherwise out of cpu.

so you are making an assertion that sch_fq is not contributing to rtt latency that I currently doubt is true. I observe HUGE buffer sizes in sch_fq when I look at it by eyeball as I add flows.

[dtaht]

Not quite relevant but I had to get this off my chest:

IMHO, the ideal self inflicted delay is 1 packet. Ironically, as we go higher rates, latency suffers further on the host, as we need more buffering because we can't handle interrupts fast enough. So you'll find - in BQL - that self inflicted delays at 100mbit are in the 2 packet, a few dozen usec range, and often a few msec at 1gigE. in other words going at higher rates, even with all this newfangled fq/aqm stuff in place, fairly fat fifo queues grow at the device driver, and 100mbit networks can actually have less latency than gig+ ones because of interrupt latency and fq being above it.

BQL is a godsend, because 100mbit networks had gone completely to hell prior to it's development and the addition of GSO and big ring buffers to linux.

I wish we could do better.

[dtaht]

OK, I setup a brief test using network namespaces to make the point. I did not go to any great extent to make it terribly scientific, I'd much rather you repeat my tests to make the point to yourself.
A tarball of the setup script, flent tests and captures for 100 flows and the square wave test is now up at: http://flent-newark.bufferbloat.net/~d/netns_sch_fq_sch_fq_codel.tgz

(if you've pulled this in the last 20 minutes I updated it with newer and more data )

This is that sch_fq result - 150-250ms delays on the netns'd cubic flow - (probably more regulated by all the other tcps competing and their RTTs than drop).

The observed RTTs in this test (it's in the cap and xplot data) for fq_codel as the ending qdisc rather than sch_fq:

This lines up with the flent measurement as well with fq_codel'd ~10ms RTTs, is here:

drop counts were much higher for fq_codel (100x?) (I still haven't fixed my sampler) - but throughput, identical. RTTs 1/15 or better that of the alternative.

Just for giggles, I did a couple tests with flent's tcp_square_wave test (4 flows, two cubic, two bbr). the cubic result - even for only two cubic flows going through sch_fq was painful.

Given the limited number of flows on this test the difference in drops was much better:
fq_codel drops ~260 packets over the duration of that test, 120 or so for sch_fq

bandwidth identical:

but which level of latency do you want for your tcp flow?

(You can certainly see a compelling advantage to bbr over cubic in this test also). Either FQ system gives it a fair share to start with, and then BBR probes for the right rate (those drops in throughput every 10sec), and gets it. If it were competing with an overlarge fifo, and not self congested, it would be uglier still. I'd rather like it if fq made it across the edges of the internet, and then all sorts of congestion controls would work way better.

[dtaht]

in terms of the local stack only, (no netns) TSQ works pretty good in both the sch_fq and sch_fq_codel cases. Over 60 seconds at gigE, 8 full rate flows going through either sch_fq or fq_codel never drop a packet. 16 flows drop 5 "packets" with fq_codel, none with sch_fq. you can't count conventional "packets" anymore as most of these are TSO and greater than 1514 bytes - but that was out of 7341269449 bytes sent.

at 15 it drops 3 packets. In both cases no difference in throughput same rtt.