Skip to content

API Timeout

This runbook details the steps to fault find timeout errors when connecting to the api server. i.e. the following errors:

  • proxy io timeout

  • curl -k https://kubernetes.default.svc/api/v1/namespaces timeout


  1. determine which node has the timeout:

    1. run command curl -k on all nodes, where is the ip address for kubernetes.default.svc.

      the node that doesn't return anything within a few seconds is a good guess as the node with the issue

  2. Stop the k3s service on the affected node service k3s stop for master node and service k3s-agent stop on a slave node

  3. manually run the k3s binary which will output logs to stdout

    • Master Node: k3s server

    • Slave node: k3s agent

  4. as the log scrolls through look for [ERROR] which will be colour coded red:

    • This error denotes a connection problem, likely firewall.

      ERRO[0141] Failed to connect to proxy. Empty dialer response  error="dial tcp <master node ip>:6443: connect: connection timed out"
      ERRO[0141] Remotedialer proxy error                      error="dial tcp <master node ip>:6443: connect: connection timed out"
      1. Fault find the firewall with the following, and from the affected node, which within a second or two should return a json unauthorized document:

        1. curl -k This command may fail

        2. curl -k https://<internal IP>:6443/api/v1/namespaces

        3. curl -k https://<external IP>:6443/api/v1/namespaces


        It may also be helpful to run these commands on the node that the affected node was trying to connect to. This will aid in the fault finding, specifically the connection path.

      2. if any of the above http requests work, that should tell you where the connectivity problem is.

      3. review the firewall(s) between the affected nod and the master the node was trying to connect to and ensure that there is sufficient rules to allow the node to connect.


This page is part of our Project ITIL Runbooks.

Page Metadata
Version: ToDo: place files short git commit here
Date Created: 2024-02-01
Date Edited: 2024-02-08


Would You like to contribute to our ITIL Runbooks project? You can assist in the following ways:


ToDo: Add the page list of contributors