Debugging: fatal: unable to access ‚https://github.com/cloudfoundry/java-buildpack.git/‘: gnutls_handshake() failed: A TLS packet with unexpected length was received.

Johannes Hiemer

31. October 2015

Reading time: 4 min

Recently we have upgraded our infrastructure to CF v222. During this upgrade we also moved our infrastructure below from Openstack Kilo to Openstack Liberty. When running the upgrade of CF, there were no issues. Trying to deploy an application, we received an error after a couple of minutes:

-----> Downloaded app package (24M)
Cloning into '/tmp/buildpacks/java-buildpack'...
fatal: unable to access 'https://github.com/cloudfoundry/java-buildpack.git/': gnutls_handshake() failed: A TLS packet with unexpected length was received.
Cloning into '/tmp/buildpacks/java-buildpack'...
fatal: unable to access 'https://github.com/cloudfoundry/java-buildpack.git/': gnutls_handshake() failed: A TLS packet with unexpected length was received.
/var/vcap/packages/dea_next/buildpacks/lib/git.rb:23:in `clone': Git clone failed: git clone --recursive https://github.com/cloudfoundry/java-buildpack.git /tmp/buildpacks/java-buildpack (RuntimeError)
from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:154:in `clone_buildpack'
from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:120:in `build_pack'
from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:83:in `block in compile_with_timeout'
from /usr/lib/ruby/1.9.1/timeout.rb:69:in `timeout'
from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:82:in `compile_with_timeout'
from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:58:in `block in stage_application'
from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:53:in `chdir'
from /var/vcap/packages/dea_next/buildpacks/lib/buildpack.rb:53:in `stage_application'
from /var/vcap/packages/dea_next/buildpacks/bin/run:10:in `'

Googling for the error this could have several issues:

A misconfigured proxy
A misconfigured firewall

The interesting thing indeed was, that cloning from the buildpack repository on the DEA VM showed the following:

/var/vcap/data/packages/rootfs_cflinuxfs2/cbcda034c2bc785743c64ad4bbf689e5c96f09ed.1-3ddba574f5c44e217415333e2c040fdb1d03196c/rootfs/usr/lib/git-core/git clone --recursive https://github.com/cloudfoundry/java-buildpack.git
Cloning into 'java-buildpack'...
warning: templates not found /usr/share/git-core/templates
remote: Counting objects: 8817, done.
remote: Total 8817 (delta 0), reused 0 (delta 0), pack-reused 8817
Receiving objects: 100% (8817/8817), 2.64 MiB | 1.93 MiB/s, done.
Resolving deltas: 100% (4891/4891), done.
Checking connectivity... done.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

Connecting into the container via (https://docs.cloudfoundry.org/running/troubleshooting/troubleshooting-apps.html) and running:

vcap@192r55pd6vg:~$ git clone http://github.com/cloudfoundry/java-buildpack.git  
Cloning into 'java-buildpack'...
fatal: unable to access 'https://github.com/cloudfoundry/java-buildpack.git/': gnutls_handshake() failed: A TLS packet with unexpected length was received.
vcap@192r55pd6vg:~$ curl https://github.com
curl: (35) Unknown SSL protocol error in connection to github.com:443 
vcap@192r55pd6vg:~$ curl http://github.com
vcap@192r55pd6vg:~$ curl http://google.com
302 Moved

That stated, that curl HTTP was not a problem while curl HTTPS lead to the end of the communication. Now the question was, why did that happen? There were several different options:

Security Groups
IP Tables Configuration
….or

something I did not have in mind and was brought to my attention through my colleagues Paulo and Mathias: MTU (https://en.wikipedia.org/wiki/Maximum_transmission_unit). According to the installation specs the MTU should have been set to 1450 on OS. The question here was, why did alls HTTPS communication work on the VM but not on the container. The answer to that, are: tunnels. The network connectivity between the outside world and the Warden container is contained in a tunnel between the host OS (my Ubuntu VM) and the container itself.

You can verify which MTUs between 1400 and 1500 work on Ubuntu with:

ping github.com -M dont -s 1426
PING github.com (192.30.252.131) 1426(1454) bytes of data.
1434 bytes from github.com (192.30.252.131): icmp_seq=1 ttl=51 time=101 ms
1434 bytes from github.com (192.30.252.131): icmp_seq=2 ttl=51 time=101 ms
1434 bytes from github.com (192.30.252.131): icmp_seq=3 ttl=51 time=102 ms

--- github.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 101.231/101.758/102.252/0.556 ms

So to find out, which was the appropriate MTU we started iterating through and we found out, that 1426 was the correct and best fit for the MTU size. Luckily you are able to configure the MTU size via in your deployment stub for the DEA: