Troubleshooting

• Decrypt and Dump TLS Traffic
  • TLS key log feature
    • Envoy Key Log configuration
      • extensions.transport_sockets.tls.v3.TlsKeyLog
  • Decryption Tools
  • Key Log Format
  • Ref
• TCP Proxy half-closed connection leak for 1 hour in some scenarios
  • Base knowledge
    • Conntrack base knowledge
    • TCP
    • Conntrack table
  • Environment
    • Testing POD
    • TCP service pod
    • enable conntrack’s invalid packet log on Testing POD
    • skills
      • nsenter network namespace
  • socket leak & occupy on FIN_WAIT2
    • 1. TCPProxy: connections ESTABLISHED
    • 2. TCPProxy: upstream service active close connection
    • 3. after 60s, conntrack table CLOSE_WAIT entry timeout
    • 4. app close connection - FIN not reach peer
• App outbound connecting timed out because App selected a ephemeral port which collisions with the existing socket on 15001(outbound) listener
  • Base knowledge
    • TCP
      • TCP Challenge ACK
    • Conntrack table
  • Environment
    • Testing POD
    • TCP service pod
    • enable conntrack’s invalid packet log on Testing POD
  • New connection timeout
    • App build connection on the same ephemeral port
      • Case 1: connect timed out by collision ephemeral port
      • Case 2: connect by collision ephemeral port but seq-no happens to be in the TCP window of the old connection
  • Skills
• Preserving HTTP/1 header case across Envoy accidentally disabled HTTP/2 on a Istio mesh mixing HTTP1 and HTTP2
  • Motivation
  • Symptom
  • Glossary
  • Background knowledge
  • Investigate
    • Debug log of outbound istio-proxy
    • Evnoy Listener
    • Envoy upstream cluster meta-data declare
  • Root cause
  • Summary