Ensure `vector` sources/sink works with AWS ALBs

[jszwedko]

We should setup a demo in https://github.com/timberio/vector-demos using an AWS ALB. If I remember right, last time we tried this, we ran into a gRPC protocol error that we haven't resolved yet.

[spencergilbert]

I think it was something like https://stackoverflow.com/questions/60044046/how-to-setup-grpc-ingress-on-gke-w-nginx-ingress which I'm also running into while testing against nginx-ingress

[gbregar]

The error I'm getting on the agent vector instance while trying to send data to an aggregator which is behind AWS ALB is
"http2 error: connection error detected: frame with invalid size".
The only relevant result I got with this error message is: hyperium/hyper#1574

[tesharp]

Was running into same problem with nginx-ingress. Trying to do TLS termination on ingress.

Like the post @spencergilbert linked to it has a problem with the initial "PRI * HTTP/2.0" package. Nginx just returns 400. Strangely enough nginx also says schema for this package is http and not https, even if tls is enables.

After lots of testing I did manage to get it to work based on the post linked by @gbregar (thanks!)

Setting the connection to use only h2 protocol in sinks\vector\v2\config.rs -> new_client:

https.set_callback(move |c, _uri| {
    if let Some(settings) = &settings {
        c.set_alpn_protos(b"\x02h2")?;
        settings.apply_connect_configuration(c);
    }

    Ok(())
});

it works! now, dont know if this is the right fix for this.. but at least fixed problem with nginx ingress

[jszwedko]

Interesting, thank you for that note @tesharp ! We'll test this and integrate it.

[mr-karan]

Is there any more progress on this? I'd really appreciate some help in using ALBs for Vector sources. Since I am using version=2 (which is the gRPC version), I've set the protocol_version for target groups to be gRPC. However, I am unsure what should be the correct healthcheck config here:

Port 6000 is where vector source listens on.

This is what I have and it's failing:

[gopiio]

Healthcheck path is incorrect. If you set the backend-protocol to GRPC then path should be /grpc.health.v1.Health/Check

[mr-karan]

@gopiio Thanks for the help! The following config worked for healthcheck.

However, I am still unable to connect it for vector sink. Anything that I am missing here?

Verbose logging:

2022-10-15T06:38:36.072903Z DEBUG hyper::client::connect::http: connecting to 172.31.yyy.xxx:443
2022-10-15T06:38:36.112687Z DEBUG hyper::client::connect::http: connected to 172.31.yyy.xxx:443
2022-10-15T06:38:36.199539Z DEBUG h2::client: binding client connection
2022-10-15T06:38:36.199585Z DEBUG h2::client: client connection bound
2022-10-15T06:38:36.200495Z DEBUG h2::codec::framed_write: send frame=Settings { flags: (0x0), enable_push: 0, initial_window_size: 2097152, max_frame_size: 16384 }
2022-10-15T06:38:36.201429Z DEBUG Connection{peer=Client}: h2::codec::framed_write: send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 5177345 }
2022-10-15T06:38:36.201483Z DEBUG hyper::client::pool: pooling idle connection for ("https", alb-address:443)
2022-10-15T06:38:36.202630Z DEBUG Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
2022-10-15T06:38:36.202683Z DEBUG Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(1) }
2022-10-15T06:38:36.202699Z DEBUG Connection{peer=Client}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(1), flags: (0x1: END_STREAM) }
2022-10-15T06:38:36.240701Z DEBUG Connection{peer=Client}: h2::proto::connection: Connection::poll; connection error error=GoAway(b"", FRAME_SIZE_ERROR, Library)
2022-10-15T06:38:36.240739Z DEBUG Connection{peer=Client}: h2::codec::framed_write: send frame=GoAway { error_code: FRAME_SIZE_ERROR, last_stream_id: StreamId(0) }
2022-10-15T06:38:36.240748Z DEBUG Connection{peer=Client}: h2::proto::connection: Connection::poll; connection error error=GoAway(b"", FRAME_SIZE_ERROR, Library)
2022-10-15T06:38:36.240785Z DEBUG hyper::client::client: client connection error: http2 error: connection error detected: frame with invalid size
2022-10-15T06:38:36.241060Z DEBUG hyper::proto::h2::client: connection error: connection error detected: frame with invalid size
2022-10-15T06:38:36.241367Z DEBUG hyper::proto::h2::client: client response error: connection error detected: frame with invalid size
2022-10-15T06:38:36.241426Z ERROR vector::topology::builder: msg="Healthcheck: Failed Reason." error=Vector source unhealthy component_kind="sink" component_type="vector" component_id=vector_out component_name=vector_out

Sink config:

[sinks.vector_out]
type = "vector"
inputs = ["tag_events"]
address = "alb-address:443"
version = "2"

Source config:

[sources.vector_agents]
type = "vector"
address = "0.0.0.0:6000"
version = "2"

When cURLing the ALB, I see this:

curl -X POST --verbose -I --http2 https://alb-address

*   Trying 172.31.xxx.yyy:443...
* Connected to alb-address (172.31.xxx.yyy) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.domain.tld
*  start date: Dec 24 00:00:00 2021 GMT
*  expire date: Jan  7 23:59:59 2023 GMT
*  subjectAltName: host "alb-address" matched cert's "*.domain.tld"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5642781d2e80)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> POST / HTTP/2
> Host: alb-address
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 502 
HTTP/2 502 
< server: awselb/2.0
server: awselb/2.0
< date: Sat, 15 Oct 2022 07:05:06 GMT
date: Sat, 15 Oct 2022 07:05:06 GMT
< content-type: text/html
content-type: text/html
< content-length: 122
content-length: 122

< 
* Excess found: excess = 122 url = / (zero-length body)
* Connection #0 to host alb-address left intact

[jszwedko]

Hi @gopiio !

I think you need to set: tls.apln_protocols = "h2" on the vector sink. Or, at least, one other user had success with that and AWS ALBs. Hopefully this works for you too! Assuming it does, we should add this mention to the documentation.

[dizlv]

@jszwedko just tried to glue two Vector topologies with vector sink and source in-between, and it failed.

1. Topolgy 1 -> AWS ALB -> Topology 2
2. AWS ALB has HTTPS listener, HTTP with gRPC in Target Group

Component has following configurations:

tls.apln_protocols = "h2"
tls.enabled = true

And it fails with

{"timestamp":"2022-11-22T15:50:50.975966Z","level":"WARN","message":"Retrying after error.","error":"Request failed: status: Unknown, message: \"http2 error: connection error detected: frame with invalid size\", details: [], metadata: MetadataMap { headers: {} }","internal_log_rate_limit":true,"target":"vector::sinks::util::retries","span":{"component_id":"sink_name","component_kind":"sink","component_name":"sink_name","component_type":"vector","name":"sink"},"spans":[{"component_id":"sink_name","component_kind":"sink","component_name":"sink_name","component_type":"vector","name":"sink"}]}

Do you have any working examples or tips on make this working?

Thank you!