Connection resets / timout with Varnish 6.0 and HTTP/2

Discussion:

Winkelmann, Thomas (RADIO TELE FFH - Online)

2018-07-04 08:43:44 UTC

Hello everbody,

finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So far everything works fine, also HTTP/2 Support.
But as soon as we are receiving some more requests (approx. > 500req/s) varnish does not deliver all requests anymore. The syslog is flooded with messages like:

Jul 4 08:04:05 cache1-vm hitch[5480]: 20180704T080405.289397 [ 5502] {backend-connect}: Connection refused
Jul 4 08:04:05 cache1-vm hitch[5480]: 20180704T080405.290213 [ 5502] {backend-connect}: Connection refused
...

Jul 4 07:57:29 cache1-vm hitch[5480]: 20180704T075729.837457 [ 5504] xxx.xxx.xxx.xxx:5835 :0 1469:1470 backend connect timeout
Jul 4 07:57:29 cache1-vm hitch[5480]: 20180704T075729.851809 [ 5510] xxx.xxx.xxx.xxx:54396 :0 7190:7191 backend connect timeout
...

Jul 4 07:57:31 cache1-vm hitch[5480]: 20180704T075731.488096 [ 5510] {backend} Socket error: Connection reset by peer

We had some similar problems in the past with HTTPS. We could solve them by adding:

net.ipv4.ip_local_port_range = 4096 64999
net.ipv4.tcp_tw_reuse = 1

to /etc/sysctl.conf But this seems to be not the problem here.

As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf everything is working normally.

Are there any limitations regarding HTTP/2 within varnish?

Our config:

[Service]
Type=simple
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :80 -a '[::1]:6086,PROXY' -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p thread_pools=2 -p thread_pool_min=200 -p thread_pool_max=5000 -p timeout_idle=100 -p send_timeout=3600 -p feature=+http2 -s malloc,2g
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true

Also DefaultLimitNOFILE is set to 250000 in /etc/systemd/system.conf.

Hitch config:

# Listening
frontend = "[*]:443"
ciphers = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"
# TLS 1.0 wird gebraucht, damit alte Server (intranet-srv) eine Verbindung aufbauen können
tls-protos = TLSv1.1 TLSv1.2 TLSv1.0

# Send traffic to the Varnish backend using the PROXY protocol
backend = "[::1]:6086"
write-proxy-v2 = on
alpn-protos = "h2,http/1.1"

# Number of processes
workers = 8

We already searched on the varnish github account for similar problem, but did not found anything...

Thanks,
Thomas

________________________________
RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth

Dridi Boukelmoune

2018-07-06 12:34:17 UTC

Permalink

Hello Thomas,

On Wed, Jul 4, 2018 at 10:43 AM, Winkelmann, Thomas (RADIO TELE FFH -

Post by Winkelmann, Thomas (RADIO TELE FFH - Online)
Hello everbody,
finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So far
everything works fine, also HTTP/2 Support.

<snip>

Post by Winkelmann, Thomas (RADIO TELE FFH - Online)
net.ipv4.ip_local_port_range = 4096 64999
net.ipv4.tcp_tw_reuse = 1
to /etc/sysctl.conf But this seems to be not the problem here.

Thanks again for reporting both your problem and solution.

Post by Winkelmann, Thomas (RADIO TELE FFH - Online)
As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf everything
is working normally.
Are there any limitations regarding HTTP/2 within varnish?

<snip>

Post by Winkelmann, Thomas (RADIO TELE FFH - Online)
We already searched on the varnish github account for similar problem, but
did not found anything...

You may have run into a known worker thread leak [1] that could be
caused by either misbehaving browsers or bugs in our h2 stack. Leaking
too many threads may put your varnish in a deadlock [2] situation that
we have yet to fix (but much less likely in the absence of the
aforementioned leak).

Could you please try building from source my 6.0 branch [3] that is
work in progress towards a 6.0.1 release? You may still run into a
crash but I'm waiting for a test case to be written before resuming
the back-porting effort.

You will likely need to rebuild your modules too, because unless I'm
confusing you with someone else I'm pretty sure you were referring
to our varnish-modules [5] collection of VMODs.

Thanks,
Dridi

[1] https://github.com/varnishcache/varnish-cache/issues/2623
[2] https://github.com/varnishcache/varnish-cache/issues/2418
[3] https://github.com/dridi/varnish-cache/tree/6.0
[4] https://github.com/varnishcache/varnish-cache/issues/2572#issuecomment-402075064
[5] https://github.com/varnish/varnish-modules

Winkelmann, Thomas (RADIO TELE FFH - Online)

2018-07-12 11:00:10 UTC

Permalink

Hello Dridi,

we just managed to clone your git tree and successfully build varnish from source. But the vmods are not found...

Could not load VMOD vsthrottle
File name: libvmod_vsthrottle.so
dlerror: libvarnishapi.so.1: cannot open shared object file: No such file or directory

They were compiled to /usr/local/lib/varnish/vmods/, but varnish seems to search in another directory?

As soon as I have running vsthrottle I can put some traffic to the server...

Thanks,
Thomas

-----Ursprüngliche Nachricht-----
Von: Dridi Boukelmoune [mailto:***@varni.sh]
Gesendet: Freitag, 6. Juli 2018 14:34
An: Winkelmann, Thomas (RADIO TELE FFH - Online)
Cc: varnish-***@varnish-cache.org
Betreff: Re: Connection resets / timout with Varnish 6.0 and HTTP/2

Hello Thomas,

On Wed, Jul 4, 2018 at 10:43 AM, Winkelmann, Thomas (RADIO TELE FFH -

Hello everbody,
finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So
far everything works fine, also HTTP/2 Support.

<snip>

We had some similar problems in the past with HTTPS. We could solve
them by
net.ipv4.ip_local_port_range = 4096 64999 net.ipv4.tcp_tw_reuse = 1
to /etc/sysctl.conf But this seems to be not the problem here.

Thanks again for reporting both your problem and solution.

As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf
everything is working normally.
Are there any limitations regarding HTTP/2 within varnish?

<snip>

We already searched on the varnish github account for similar problem,
but did not found anything...

You may have run into a known worker thread leak [1] that could be caused by either misbehaving browsers or bugs in our h2 stack. Leaking too many threads may put your varnish in a deadlock [2] situation that we have yet to fix (but much less likely in the absence of the aforementioned leak).

Could you please try building from source my 6.0 branch [3] that is work in progress towards a 6.0.1 release? You may still run into a crash but I'm waiting for a test case to be written before resuming the back-porting effort.

You will likely need to rebuild your modules too, because unless I'm confusing you with someone else I'm pretty sure you were referring to our varnish-modules [5] collection of VMODs.

Thanks,
Dridi

[1] https://github.com/varnishcache/varnish-cache/issues/2623
[2] https://github.com/varnishcache/varnish-cache/issues/2418
[3] https://github.com/dridi/varnish-cache/tree/6.0
[4] https://github.com/varnishcache/varnish-cache/issues/2572#issuecomment-402075064
[5] https://github.com/varnish/varnish-modules

RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth

Winkelmann, Thomas (RADIO TELE FFH - Online)

2018-07-12 11:24:56 UTC

Permalink

We could fix the problem by running ldconfig.

See: https://ma.ttias.be/varnish-varnishhistvarnishtop-error-while-loading-shared-libraries-libvarnishapi-so-1-cannot-open-shared-object-file/

In which way do you prefer to reveice the crash reports? Or which log files are helpful for you to get the h2 bug fixed?

-----Ursprüngliche Nachricht-----
Von: Winkelmann, Thomas (RADIO TELE FFH - Online)
Gesendet: Donnerstag, 12. Juli 2018 13:00
An: 'Dridi Boukelmoune'
Cc: 'varnish-***@varnish-cache.org'
Betreff: AW: Connection resets / timout with Varnish 6.0 and HTTP/2

Hello Dridi,

we just managed to clone your git tree and successfully build varnish from source. But the vmods are not found...

Could not load VMOD vsthrottle
File name: libvmod_vsthrottle.so
dlerror: libvarnishapi.so.1: cannot open shared object file: No
such file or directory

Hello everbody,
finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So
far everything works fine, also HTTP/2 Support.

<snip>

Thanks again for reporting both your problem and solution.

As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf
everything is working normally.
Are there any limitations regarding HTTP/2 within varnish?

<snip>

We already searched on the varnish github account for similar problem,
but did not found anything...

You may have run into a known worker thread leak [1] that could be caused by either misbehaving browsers or bugs in our h2 stack. Leaking too many threads may put your varnish in a deadlock [2] situation that we have yet to fix (but much less likely in the absence of the aforementioned leak).

Could you please try building from source my 6.0 branch [3] that is work in progress towards a 6.0.1 release? You may still run into a crash but I'm waiting for a test case to be written before resuming the back-porting effort.

You will likely need to rebuild your modules too, because unless I'm confusing you with someone else I'm pretty sure you were referring to our varnish-modules [5] collection of VMODs.

Thanks,
Dridi

[1] https://github.com/varnishcache/varnish-cache/issues/2623
[2] https://github.com/varnishcache/varnish-cache/issues/2418
[3] https://github.com/dridi/varnish-cache/tree/6.0
[4] https://github.com/varnishcache/varnish-cache/issues/2572#issuecomment-402075064
[5] https://github.com/varnish/varnish-modules

RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth

Guillaume Quintard

2018-07-12 12:21:38 UTC

Permalink

Hi Thomas,

You can use the vcl_path parameter to adjust where varnish will look for
the vmod files.

Cheers,
--
Guillaume Quintard

On Thu, Jul 12, 2018 at 4:24 AM, Winkelmann, Thomas (RADIO TELE FFH -

Post by Winkelmann, Thomas (RADIO TELE FFH - Online)
We could fix the problem by running ldconfig.
See: https://ma.ttias.be/varnish-varnishhistvarnishtop-error-
while-loading-shared-libraries-libvarnishapi-so-1-
cannot-open-shared-object-file/
In which way do you prefer to reveice the crash reports? Or which log
files are helpful for you to get the h2 bug fixed?
-----UrsprÃŒngliche Nachricht-----
Von: Winkelmann, Thomas (RADIO TELE FFH - Online)
Gesendet: Donnerstag, 12. Juli 2018 13:00
An: 'Dridi Boukelmoune'
Betreff: AW: Connection resets / timout with Varnish 6.0 and HTTP/2
Hello Dridi,
we just managed to clone your git tree and successfully build varnish from
source. But the vmods are not found...

Could not load VMOD vsthrottle
File name: libvmod_vsthrottle.so
dlerror: libvarnishapi.so.1: cannot open shared object file: No
such file or directory

They were compiled to /usr/local/lib/varnish/vmods/, but varnish seems to
search in another directory?
As soon as I have running vsthrottle I can put some traffic to the server...
Thanks,
Thomas
-----UrsprÃŒngliche Nachricht-----
Gesendet: Freitag, 6. Juli 2018 14:34
An: Winkelmann, Thomas (RADIO TELE FFH - Online)
Betreff: Re: Connection resets / timout with Varnish 6.0 and HTTP/2
Hello Thomas,
On Wed, Jul 4, 2018 at 10:43 AM, Winkelmann, Thomas (RADIO TELE FFH -

Hello everbody,
finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So
far everything works fine, also HTTP/2 Support.

<snip>

Thanks again for reporting both your problem and solution.

As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf
everything is working normally.
Are there any limitations regarding HTTP/2 within varnish?

<snip>

We already searched on the varnish github account for similar problem,
but did not found anything...

You may have run into a known worker thread leak [1] that could be caused
by either misbehaving browsers or bugs in our h2 stack. Leaking too many
threads may put your varnish in a deadlock [2] situation that we have yet
to fix (but much less likely in the absence of the aforementioned leak).
Could you please try building from source my 6.0 branch [3] that is work
in progress towards a 6.0.1 release? You may still run into a crash but I'm
waiting for a test case to be written before resuming the back-porting
effort.
You will likely need to rebuild your modules too, because unless I'm
confusing you with someone else I'm pretty sure you were referring to our
varnish-modules [5] collection of VMODs.
Thanks,
Dridi
[1] https://github.com/varnishcache/varnish-cache/issues/2623
[2] https://github.com/varnishcache/varnish-cache/issues/2418
[3] https://github.com/dridi/varnish-cache/tree/6.0
[4] https://github.com/varnishcache/varnish-cache/
issues/2572#issuecomment-402075064
[5] https://github.com/varnish/varnish-modules
RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
GeschÃ€ftsfÃŒhrer / Programmdirektor: Hans-Dieter Hillmoth
_______________________________________________
varnish-misc mailing list
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc