Memory leak in sysread when reading a reset TCP connection on macOS

[drclaw1394]

Description
I have perl based HTTP server code which I've been testing and benchmarking persistent (keep alive) connections with the ab and wrk benchmarking programs.

• When testing with ab the memory usage of my server stabilises after a couple of tests
• When test with the wrk program, memory usage continues to grow.
• Memory increases quickly at the end of a test, when the wrk persistent connection is closed.

Putting many debug print statements in my code, the difference (other than basic headers) was wrk was causing a 'Connection reset by peer' error in my server when attempting to do a sysread.

Installing the server on a Linux virtual machine and running the same benchmark tests DID NOT exhibit the same behaviour. Memory usage was stable when using either ab or wrk.

For further investigation, I made a boiled down a client and server to replicate the issue. Please see the code listing below. After connecting, the client code abruptly closes the connection (using a linger value of 0). This results in the same 'Connection reset' error in the server and increasing memory usage on macOS

Running this test client/sever on my Linux virtual machine, DID NOT exhibit the memory growth.

Steps to Reproduce

Run the server script below in one terminal:
server.pl

In another terminal run the client in 'clean' mode:
client.pl

The server terminal should start reporting the memory usage or the server. It should stabilise quickly.

Rerun the client again, this time with an argument of 1 to reset the TCP connection:
client.pl 1

The server terminal will again report the memory usage. This time the memory usage will increase when running on macOS.

Repeating the above on a Linux virtual machine does not cause the memory to increase.

Server

use strict;
use warnings;
use feature "say";
use feature "switch";

use Socket;
use Fcntl qw(F_GETFL F_SETFL O_NONBLOCK);
use Errno qw(EAGAIN EINTR);
use Time::HiRes qw<sleep>;

my $port=8080;

socket(my $server, PF_INET, SOCK_STREAM,0) or die "Could not create socket: $!";
setsockopt ($server, SOL_SOCKET, SO_REUSEADDR, pack ("l",1)) or die "Setsockopt fail: $!";


my $name=sockaddr_in($port, INADDR_ANY);
bind $server,$name;
listen $server, 1024;

while (1){
	my $peer_address=accept(my $client,$server);
	say "";
	#set to non blocking
	fcntl $client, F_SETFL, O_NONBLOCK;	#this nukes other flags... read first?
	my $buf="";
	while (1){
		#$!=undef;
		#syswrite $client,"asdf";
		my $ret=sysread $client,$buf,1024;#, length $buf;
		given($ret){
			when(undef){
				#check errors
				say $!;
				next unless ( $!==EAGAIN or $!== EINTR ); 
			}
			when($ret>0){
				#say "BUFFER: $buf";
				$buf="";
			}
			when(0){
				say "EOF";
				last;
			}
			default {
			}
		}
		for(grep /$$/, split /\n/s, `ps aux`){
			my @f=split / +/;
			say "Memory Usage (KB): $f[5]";
		}
		#sleep 0.1;
	}
}

Client

use strict;
use warnings;
use feature "say";
use feature "switch";

use Socket;
use Time::HiRes qw<sleep>;

my $port=8080;
my $name=sockaddr_in($port,inet_aton("127.0.0.1"));

my $do_reset=$ARGV[0];
say "Forcing TCP Reset: ", $do_reset?"true":"false";
while(1){
	socket(my $socket, PF_INET, SOCK_STREAM,0) or die "Could not create socket: $!";
	connect($socket,$name) or die "connect error: $!";
	if($do_reset){
		setsockopt($socket, SOL_SOCKET, SO_LINGER, pack ("ll",1,0)) or die "Setsockopt fail: $!";
	}

	syswrite($socket, "Hello");
	close $socket;
	sleep 0.2;
}

Expected behavior

Expected behaviour is memory usage not to increase unbounded when using sysread on a reset tcp connection.

Perl configuration

Summary of my perl5 (revision 5 version 34 subversion 0) configuration:
   
  Platform:
    osname=darwin
    osvers=20.5.0
    archname=darwin-thread-multi-2level
    uname='darwin bigsurx.internal.macports.net 20.5.0 darwin kernel version 20.5.0: sat may 8 05:10:33 pdt 2021; root:xnu-7195.121.3~9release_x86_64 x86_64 '
    config_args='-des -Dprefix=/opt/local -Dscriptdir=/opt/local/bin -Dvendorprefix=/opt/local -Dusemultiplicity=y -Dusethreads -Duseshrplib -Dcc=/usr/bin/cc -Dman1ext=1pm -Dman3ext=3pm -Dinstallstyle=lib/perl5 -Dman1dir=/opt/local/share/man/man1p -Dman3dir=/opt/local/share/man/man3p -Dsitebin=/opt/local/libexec/perl5.34/sitebin -Dsiteman1dir=/opt/local/share/perl5.34/siteman/man1 -Dsiteman3dir=/opt/local/share/perl5.34/siteman/man3 -Dvendorbin=/opt/local/libexec/perl5.34 -Dvendorman1dir=/opt/local/share/perl5.34/man/man1 -Dvendorman3dir=/opt/local/share/perl5.34/man/man3 -Dpager=/usr/bin/less -sR -Dperlpath=/opt/local/bin/perl5.34 -Dstartperl=#!/opt/local/bin/perl5.34 -Acppflags=-I/opt/local/include  -Accflags=-pipe -Os   -Alddlflags=-L/opt/local/lib -Wl,-headerpad_max_install_names   -Aldflags=-L/opt/local/lib -Wl,-headerpad_max_install_names  '
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
  Compiler:
    cc='/usr/bin/cc'
    ccflags ='-fno-common -DPERL_DARWIN  -pipe -Os   -fno-strict-aliasing -fstack-protector-strong -I/opt/local/include -DPERL_USE_SAFE_PUTENV'
    optimize='-O3'
    cppflags='-I/opt/local/include  -fno-common -DPERL_DARWIN  -pipe -Os   -fno-strict-aliasing -fstack-protector-strong -I/opt/local/include'
    ccversion=''
    gccversion='Apple LLVM 12.0.5 (clang-1205.0.22.9)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='/usr/bin/cc'
    ldflags ='  -L/opt/local/lib -Wl,-headerpad_max_install_names   -fstack-protector-strong'
    libpth=/opt/local/lib /Library/Developer/CommandLineTools/usr/lib/clang/12.0.5/lib /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/lib /Library/Developer/CommandLineTools/usr/lib /usr/lib
    libs=-lgdbm
    perllibs=
    libc=
    so=dylib
    useshrplib=true
    libperl=libperl.dylib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=bundle
    d_dlsymun=undef
    ccdlflags=' '
    cccdlflags=' '
    lddlflags='  -bundle -undefined dynamic_lookup -L/opt/local/lib -Wl,-headerpad_max_install_names   -fstack-protector-strong'


Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_IMPLICIT_CONTEXT
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    PERL_USE_SAFE_PUTENV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
    USE_THREAD_SAFE_LOCALE
  Built under darwin
  Compiled at Jul  5 2021 10:31:17
  @INC:
    /opt/local/lib/perl5/site_perl/5.34/darwin-thread-multi-2level
    /opt/local/lib/perl5/site_perl/5.34
    /opt/local/lib/perl5/vendor_perl/5.34/darwin-thread-multi-2level
    /opt/local/lib/perl5/vendor_perl/5.34
    /opt/local/lib/perl5/5.34/darwin-thread-multi-2level
    /opt/local/lib/perl5/5.34

[tonycoz]

perl doesn't do anything strange for sysread (POSIX read()) on darwin that I can see.

I don't have darwin available to test, but when testing this on FreeBSD after a few connections the accept() would fail due to the client closing the connection and get stuck in the read loop.