DNS resolution not working on macOS with "bind: operation not permitted"

[thom-nic]

Puma dev's DNS resolution doesn't appear to work at all anymore on MacOS Catalina.

I've done puma-dev -cleanup and even uninstalled/reinstalled from homebrew, then sudo puma-dev -setup; puma-dev -install again.

dig @127.0.0.1 -p 9253 blah.test

; <<>> DiG 9.10.6 <<>> @127.0.0.1 -p 9253 blah.test
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

However everything looks good otherwise. Puma is running, resolver file in place, responds to HTTP but not DNS on port

$ ls -l /etc/resolver 
total 8
-rw-r--r--  1 thom  staff    55B Mar 24 10:22 test

$ cat /etc/resolver/test 
# Generated by puma-dev
nameserver 127.0.0.1
port 9253

$ ps ax|grep puma
39752   ??  S      0:00.04 /usr/local/Cellar/puma-dev/0.13/bin/puma-dev -launchd -dir ~/.puma-dev -d test -timeout 15m0s

$ cat ~/.puma-dev/blah
2999

$ curl -vH "Host: blah.test" localhost
*   Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 80 failed: Connection refused
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: blah.test
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 25 Mar 2020 13:18:25 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Closing connection 0

$ tail -n20 ~/Library/Logs/puma-dev.log
2020/03/24 10:22:09 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/thom/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: inherited from launchd
* HTTPS Server port: inherited from launchd
! Puma dev listening on http and https
* Generated proxy connection for 'blah' to http://127.0.0.1:2999
2020/03/25 09:18:21 http: proxy error: dial tcp 127.0.0.1:2999: connect: connection refused

Summary: HTTP proxy works, but DNS does not.

[nonrational]

I wrote a quick test to ensure the DNS server works when it can listen properly, but it seems like 9253 isn't reachable at all on your machine.

Would you mind running sw_vers; nc -zv localhost 9253 and providing the result?

[thom-nic]

$ sw_vers; nc -zv localhost 9253
ProductName:	Mac OS X
ProductVersion:	10.15.3
BuildVersion:	19D76
nc: connectx to localhost port 9253 (tcp) failed: Connection refused
nc: connectx to localhost port 9253 (tcp) failed: Connection refused

ETA: Shouldn't 9253 be a UDP socket?
Edit 2: sudo lsof -i :9253 also returns nothing

[nonrational]

ETA: Shouldn't 9253 be a UDP socket?

puma-dev's DNS is intended to serve both TCP and UDP. https://github.com/puma/puma-dev/blob/master/dev/dns.go#L75-L85

Running two copies of puma-dev in the foreground, puma-dev bails on the http/s port, but it doesn't bail on the DNS port.

So, I expect puma-dev is failing to bind to 9253 and failing silently.

# keep this one running 
$ puma-dev
2020/03/26 15:41:02 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https

# attempt no. 1
$ puma-dev
2020/03/26 19:42:16 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https
2020/03/26 19:42:16 Error listening: listen tcp 127.0.0.1:9280: bind: address already in use

# attempt no. 2
$ puma-dev -http-port 9000 -https-port 90001
2020/03/26 19:42:48 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9000
* HTTPS Server port: 90001
! Puma dev listening on http and https

Also, this "works," which is disconcerting.

$ puma-dev -dns-port -1
2020/03/26 20:09:00 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: -1
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https

[nonrational]

@thom-nic will you try running the following?

puma-dev -dns-port 32000
sudo lsof -i :32000

edit: I realized I'm using GNU nc. Not sure what the equivalent BSD nc flags are. One sec. Your lsof approach is better for checking port binding.

[nonrational]

Confirmed that DNS port bind will fail silently. I have a WIP branch that will expose the error messages and bail.

[thom-nic]

For some reason when I try to run on OSX, puma-dev can't seem to bind to any HTTP/S port...?

puma-dev -dns-port 32000 -http-port 34999 -https-port 34998
2020/03/27 09:57:38 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/thom/.puma-dev
* Domains: test
* DNS Server port: 32000
* HTTP Server port: 34999
* HTTPS Server port: 34998
! Puma dev listening on http and https
2020/03/27 09:57:38 Error listening: listen tcp 127.0.0.1:34999: bind: operation not permitted

This may be part of the root cause of the issue I'm facing. Although the HTTP bind doesn't fail when running as a launch daemon, but then again in that case HTTP/S are being forwarded via launchd socket.

[nonrational]

Can you bind to those ports with other user processes e.g. rackup ?

[thom-nic]

I tried:

$ python3 -m http.server 34999
Serving HTTP on 0.0.0.0 port 34999 (http://0.0.0.0:34999/) ...

This prompt popped up:

(I clicked yes)
EDIT Firewall panel:

Oddly enough, nodejs isn't listed here and I do most of my development on node. Unsure why I'm not prompted for that. Or why I'm not prompted when I attempt to run puma-dev in the foreground.

EDIT 2: I manually added puma-dev in the firewall list and it still fails with the same error.

[dangerous]

I have the same issue, the firewall is stopping puma-dev from binding to the DNS port, however thought it might be worth mentioning that I am still using Mojave (10.14.6).

[dangerous]

I suspect it's because puma-dev is not signed

[nonrational]

I can't replicate the blocked behavior on my 10.14 machine. Haven't tried on 10.15 yet.

Can you run the following and report what you see?

$ /usr/libexec/ApplicationFirewall/socketfilterfw --getappblocked $(which puma-dev)
The application is not part of the firewall

And maybe try

$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add $(which puma-dev)
Application at path ( /path/to/puma/dev/puma-dev ) added to firewall
$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp $(which puma-dev)
Incoming connection to the application is permitted

[swiknaba]

Just for the record: it is also broken on my machine running macOS 10.14.6 (Mojave) after installing the following update from Apple:

XProtectPlistConfigData:

  Version:	2118
  Source:	Apple
  Install Date:	11.04.20, 18:23

the launchd still spawns, but routing traffic from .test domains does not resolve to localhost, but does a request to my regular DNS provider.

Didn't find much about that update from Apple, but it seems to be about blocking possibly malicious processes: https://support.apple.com/en-gb/HT207005