API Timeout

This runbook details the steps to fault find timeout errors when connecting to the api server. i.e. the following errors:

proxy io timeout
curl -k https://kubernetes.default.svc/api/v1/namespaces timeout

Workflow

determine which node has the timeout:
1. run command curl -k https://172.16.100.1/api/v1/namespaces on all nodes, where 172.16.100.1 is the ip address for kubernetes.default.svc.
  
  the node that doesn't return anything within a few seconds is a good guess as the node with the issue
Stop the k3s service on the affected node service k3s stop for master node and service k3s-agent stop on a slave node
manually run the k3s binary which will output logs to stdout
- Master Node: k3s server
- Slave node: k3s agent
as the log scrolls through look for [ERROR] which will be colour coded red:
- This error denotes a connection problem, likely firewall.
```
ERRO[0141] Failed to connect to proxy. Empty dialer response  error="dial tcp <master node ip>:6443: connect: connection timed out"
ERRO[0141] Remotedialer proxy error                      error="dial tcp <master node ip>:6443: connect: connection timed out"
```
  1. Fault find the firewall with the following, and from the affected node, which within a second or two should return a json unauthorized document:
    1. curl -k https://127.0.0.1:6443/api/v1/namespaces This command may fail
    2. curl -k https://<internal IP>:6443/api/v1/namespaces
    3. curl -k https://<external IP>:6443/api/v1/namespaces
    Tip
    
    It may also be helpful to run these commands on the node that the affected node was trying to connect to. This will aid in the fault finding, specifically the connection path.
  2. if any of the above http requests work, that should tell you where the connectivity problem is.
  3. review the firewall(s) between the affected nod and the master the node was trying to connect to and ensure that there is sufficient rules to allow the node to connect.

About:

This page is part of our Project ITIL Runbooks.

Page Metadata

Version: ToDo: place files short git commit here
Date Created: 2024-02-01
Date Edited: 2024-02-08

Contribution:

Would You like to contribute to our ITIL Runbooks project? You can assist in the following ways:

Edit This Page If there is a mistake or a way you can improve it.
Add a Page to the Manual if you would like to add an item to our manual
Raise an Issue if there is something about this page you would like to improve, and git is unfamiliar to you.

ToDo: Add the page list of contributors