Introduction

Recently I helped to debug an issue where connections into a gRPC server seemingly stopped responding for around 15 minutes. We figured out a solution but I wanted to try and really understand why it was a problem, which took me down a rather large rabbit hole and I decided to document everything for posterity.

tcp_retries2

Before diving into the issue, it's important to first explain tcp_retries2. The linux man-pages provides the following definition:

tcp_retries2 (integer; default: 15; since Linux 2.2)
The maximum number of times a TCP packet is retransmitted in established state before giving up. The default value is 15, which corresponds to a duration of approximately between 13 to 30 minutes, depending on the retransmission timeout. The RFC 1122 specified minimum limit of 100 seconds is typically deemed too short.

The TCP protocol is a connection-oriented stateful network protocol. For each packet sent, the TCP stack considers it successfully delivered once it gets an ACK back for that specific packet.

If the package is unacknowledged, TCP will retransmit it up to the number of times defined in the tcp_retries2 setting (defaults to 15) using an exponential backoff timeout for which each retransmission timeout is between TCP_RTO_MIN and TCP_RTO_MAX. Once the last retries expires, the TCP stack will notify the application of a broken connection.

We can reference the table from Marco Pracucci for a good idea of how changing this number changes the time before a timeout is communicated:

Retransmission #	RTO (ms)	Time before timeout (secs)	Time before timeout (mins)
1	200	0.2	0.0
2	400	0.6	0.0
3	800	1.4	0.0
4	1600	3.0	0.1
5	3200	6.2	0.1
6	6400	12.6	0.2
7	12800	25.4	0.4
8	25600	51.0	0.9
9	51200	102.2	1.7
10	102400	204.6	3.4
11	120000	324.6	5.4
12	120000	444.6	7.4
13	120000	564.6	9.4
14	120000	684.6	11.4
15	120000	804.6	13.4
16	120000	924.6	15.4

The current value of tcp_retries2 can be retrieved from sysctl (sysctl net/ipv4/tcp_retries2), and can be set temporarily using sysctl (e.g. sysctl -w net.ipv4.tcp_retries2=5) or permanently by setting it in /etc/sysctl.conf. Note: This applies to Ubuntu, other distributions may differ.

gRPC

So now we have at least a basic understanding of what tcp_retries2 is, how does it translate to gRPC? gRPC uses HTTP/2 over a single, long-lived TCP connection for all RPCs between a client and server.

This means, if once a connection has been established between the client and server, subsequent packages go unacknowledged TCP will continue to try and retransmit until it hits the tcp_retries2 limit before notifying the application and creating a new TCP connection. If the tcp_retries2 setting is unaltered, this could cause a gRPC client to get "stuck" for 15 minutes if something other than the server/client has killed the connection (e.g. firewall, load balancer etc).

Reproducing the issue

To reproduce the issue we're going to create a simple gRPC server/client, and then we're going to break the TCP connection by blocking it with a firewall (iptables) and observe the behaviour.

First, so we don't have to wait for ~15 minutes to observe the whole process, I'm going to set tcp_retries2 to a value of 10 (3.4 minutes before timeout) temporarily by running sysctl -w net.ipv4.tcp_retries2=10:

$ sudo sysctl -w net.ipv4.tcp_retries2=10
net.ipv4.tcp_retries2 = 10

$ sysctl net.ipv4.tcp_retries2
net.ipv4.tcp_retries2 = 10

Server

For our server we're going to use the following code. The full code (including protobuf definitions) can be found on my GitHub if needed.

The server will start on port 5000 by default, and exposes one method (Heartbeat).

package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"net"
	"time"

	keepalivev1alpha1 "github.com/grpckeepalive/server/proto/v1alpha1"
	"google.golang.org/grpc"
)

var port = flag.Int("name", 5000, "port to run the server on")

func main() {
	flag.Parse()

	s := &Server{}

	g := grpc.NewServer()

	keepalivev1alpha1.RegisterExampleServer(g, s)

	l, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
	if err != nil {
		log.Fatalf("failed to listen: %+v", err)
	}

	fmt.Printf("Listening on :%d\n", *port)

	if err := g.Serve(l); err != nil {
		log.Fatalf("failed to serve: %+v", err)
	}

}

type Server struct {
	keepalivev1alpha1.UnimplementedExampleServer
}

func (s *Server) Heartbeat(ctx context.Context, r *keepalivev1alpha1.HeartbeatRequest) (*keepalivev1alpha1.HeartbeatReply, error) {
	time.Sleep(10 * time.Second)

	return &keepalivev1alpha1.HeartbeatReply{Message: "pong"}, nil

}

Client

The client is equally as simple. It connects to the gRPC server on port 5000 by default, sends an initial Heartbeat request and then continues to send one every 20 seconds for the rest of time.

package main

import (
	"context"
	"flag"
	"log"
	"time"

	keepalivev1alpha1 "github.com/grpckeepalive/server/proto/v1alpha1"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
)

var addr = flag.String("addr", "localhost:5000", "the address to connect to")

func main() {
	flag.Parse()

	conn, err := grpc.NewClient(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()))

	if err != nil {
		log.Fatalf("did not connect: %v", err)
	}
	defer conn.Close()

	c := keepalivev1alpha1.NewExampleClient(conn)

	sendHeartbeat(c)

	ticker := time.NewTicker(20 * time.Second)

	for range ticker.C {
		sendHeartbeat(c)
	}

}

func sendHeartbeat(c keepalivev1alpha1.ExampleClient) {
	ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
	defer cancel()

	log.Println("Sending request")

	r, err := c.Heartbeat(ctx, &keepalivev1alpha1.HeartbeatRequest{})
	if err != nil {
		log.Printf("Heartbeat failed: %+v", err)
		return
	}

	log.Printf("Response: %s\n", r.GetMessage())

}

Reproducing the problem

First, we'll run the server (go run server/main.go) and client (go run server/main.go). We'll see the server prints out a Listening on :5000 log and the client will print out Sending request and Response: pong logs every 20 seconds.

# Server
$ go run server/main.go
Listening on :5000

# Client
$ go run client/main.go
2025/07/03 13:57:52 Sending request
2025/07/03 13:58:02 Response: pong
2025/07/03 13:58:22 Sending request
2025/07/03 13:58:32 Response: pong
2025/07/03 13:58:42 Sending request
2025/07/03 13:58:52 Response: pong
2025/07/03 13:59:02 Sending request
2025/07/03 13:59:12 Response: pong

Great - everything is working. Now, let's break it. We can use netstat to find the TCP port that gRPC is using to communicate between the client and server by running sudo netstat -anp | grep main. This will give a similar output to the below:

$ sudo netstat -anp | grep main
tcp        0      0 127.0.0.1:37320         127.0.0.1:5000          ESTABLISHED 1929656/main
tcp6       0      0 :::5000                 :::*                    LISTEN      1929105/main
tcp6       0      0 127.0.0.1:5000          127.0.0.1:37320         ESTABLISHED 1929105/main
Active UNIX domain sockets (servers and established)

The line we're most interested in is the first line which tells us the TCP port gRPC is using is 37320 (127.0.0.1:37320). To simulate the TCP connection being dropped by an outside source, we're going to use iptables to deny packets on that port.

$ sudo iptables -I INPUT -p tcp --dport 37320 -j DROP

$ sudo iptables -L INPUT --line-numbers
Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    DROP       tcp  --  anywhere             anywhere             tcp dpt:37320
2    ufw-before-logging-input  all  --  anywhere             anywhere
3    ufw-before-input  all  --  anywhere             anywhere
4    ufw-after-input  all  --  anywhere             anywhere
5    ufw-after-logging-input  all  --  anywhere             anywhere
6    ufw-reject-input  all  --  anywhere             anywhere
7    ufw-track-input  all  --  anywhere             anywhere

Immediately we can see the requests starting to fail on the client side:

2025/07/03 14:04:22 Sending request
2025/07/03 14:04:32 Response: pong
2025/07/03 14:04:42 Sending request
2025/07/03 14:04:57 Heartbeat failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2025/07/03 14:05:02 Sending request
2025/07/03 14:05:17 Heartbeat failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded

But gRPC continues to use the same port because we haven't yet exhausted the attempts set in net.ipv4.tcp_retries2:

$ sudo netstat -anp | grep main
tcp        0    288 127.0.0.1:37320         127.0.0.1:5000          ESTABLISHED 1929656/main
tcp6       0      0 :::5000                 :::*                    LISTEN      1929105/main
Active UNIX domain sockets (servers and established)

Once we have exhausted the number of attempts set in net.ipv4.tcp_retries2 (~4 minutes with our value of 10), we can see that the requests start to succeed again because the TCP stack has informed the application that the TCP connection is dead and to create a new one.

2025/07/03 14:09:42 Sending request
2025/07/03 14:09:57 Heartbeat failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2025/07/03 14:10:02 Sending request
2025/07/03 14:10:17 Heartbeat failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2025/07/03 14:10:22 Sending request
2025/07/03 14:10:27 Heartbeat failed: rpc error: code = Unavailable desc = error reading from server: read tcp 127.0.0.1:37320->127.0.0.1:5000: read: connection timed out
2025/07/03 14:10:42 Sending request
2025/07/03 14:10:52 Response: pong

$ sudo netstat -anp | grep main
tcp        0      0 127.0.0.1:44676         127.0.0.1:5000          ESTABLISHED 1929656/main
tcp6       0      0 :::5000                 :::*                    LISTEN      1929105/main
tcp6       0      0 127.0.0.1:5000          127.0.0.1:44676         ESTABLISHED 1929105/main
Active UNIX domain sockets (servers and established)

And once the new connection is created, everything starts to work again even though neither the client or server have been restarted. So it's hopefully easy to now see how a gRPC client can get "stuck" for 15 minutes.

2025/07/03 14:10:27 Heartbeat failed: rpc error: code = Unavailable desc = error reading from server: read tcp 127.0.0.1:37320->127.0.0.1:5000: read: connection timed out
2025/07/03 14:10:42 Sending request
2025/07/03 14:10:52 Response: pong
2025/07/03 14:11:02 Sending request
2025/07/03 14:11:12 Response: pong
2025/07/03 14:11:22 Sending request
2025/07/03 14:11:32 Response: pong
2025/07/03 14:11:42 Sending request
2025/07/03 14:11:52 Response: pong
2025/07/03 14:12:02 Sending request
2025/07/03 14:12:12 Response: pong
2025/07/03 14:12:22 Sending request
2025/07/03 14:12:32 Response: pong

Note: If you're following along, don't forget to remove the iptables rule:

$ sudo iptables -L INPUT --line-numbers
Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    DROP       tcp  --  anywhere             anywhere             tcp dpt:37320
2    ufw-before-logging-input  all  --  anywhere             anywhere
3    ufw-before-input  all  --  anywhere             anywhere
4    ufw-after-input  all  --  anywhere             anywhere
5    ufw-after-logging-input  all  --  anywhere             anywhere
6    ufw-reject-input  all  --  anywhere             anywhere
7    ufw-track-input  all  --  anywhere             anywhere

$ sudo iptables -D INPUT 1

Using keepalives

If you don't want to manually configure tcp_retries2, another way of speeding up the time to identify a closed TCP connection is to use gRPC keepalives.

Keepalives are a way to keep the connection alive, even when there is no data being transferred. This is done by periodically sending a PING frame to the other end of the connection. Keepalives can improve performance and reliability of connections, but it is important to configure the interval carefully.

Configuring a gRPC keep alive can do more harm that good, particularly if the client configures parameters at a more frequent interval than the server allows as it can cause GOAWAY responses disabling further keep alive requests from being processed until actual traffic is received by the server again.

For this reason, it's important for clients and servers to discuss the right settings. A safe default would be to set the keepalive requests to 5 minutes, which matches a gRPC servers default EnforcementPolicy.MinTime. If you're not sure what settings the server supports, it's best to ask to avoid issues.

To implement this on our demonstration project, we'll replace this line in the client code (Note: no changes are required on the server side):

conn, err := grpc.NewClient(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()))

With these lines:

kap := grpc.WithKeepaliveParams(keepalive.ClientParameters{
	// Sets the frequency of keep alive requests to 5 minutes, which matches the servers default EnforcementPolicy.MinTime of 5 minutes
	// Setting this to be more frequent could result in GOAWAY responses.
	Time: 5 * time.Minute,
	// Allows the client to send keepalive pings even with no active RPCs
	PermitWithoutStream: true,
})
conn, err := grpc.NewClient(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()), kap)

We'll change net.ipv4.tcp_retries2 back to 15

$ sudo sysctl -w net.ipv4.tcp_retries2=15
net.ipv4.tcp_retries2 = 15

$ sysctl net.ipv4.tcp_retries2
net.ipv4.tcp_retries2 = 15

And run the demonstration again (blocking the port via iptables etc), and we should see the application recover much quicker than 15 minutes because of the keepalive:

$ sudo netstat -anp | grep main
tcp        0      0 127.0.0.1:48490         127.0.0.1:5000          ESTABLISHED 1980874/main
tcp6       0      0 :::5000                 :::*                    LISTEN      1980214/main
tcp6       0      0 127.0.0.1:5000          127.0.0.1:48490         ESTABLISHED 1980214/main
Active UNIX domain sockets (servers and established)

$ sudo iptables -I INPUT -p tcp --dport 48490 -j DROP

$ sudo iptables -L INPUT --line-numbers
Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    DROP       tcp  --  anywhere             anywhere             tcp dpt:48490

We can see from the client logs that we starting seeing failures at 14:26:07 and less than a minute later (14:26:32) we had a new TCP connection.

$ go run client/main.go
2025/07/03 14:25:21 Sending request
2025/07/03 14:25:32 Response: pong
2025/07/03 14:25:52 Sending request
2025/07/03 14:26:07 Heartbeat failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2025/07/03 14:26:12 Sending request
2025/07/03 14:26:13 Heartbeat failed: rpc error: code = Unavailable desc = error reading from server: read tcp 127.0.0.1:48490->127.0.0.1:5000: read: connection timed out
2025/07/03 14:26:32 Sending request
2025/07/03 14:26:42 Response: pong
2025/07/03 14:26:52 Sending request
2025/07/03 14:27:02 Response: pong

$ sudo netstat -anp | grep main
tcp        0      0 127.0.0.1:59088         127.0.0.1:5000          ESTABLISHED 1980874/main
tcp6       0      0 :::5000                 :::*                    LISTEN      1980214/main
tcp6       0      0 127.0.0.1:5000          127.0.0.1:59088         ESTABLISHED 1980214/main
Active UNIX domain sockets (servers and established)

If we focus on the first six logs from the client, we can see:

The first request was made at 14:25:21 and the response received at 14:25:32
The second request was made at 14:25:52 and failed with a DeadlineExceeded error at 14:26:07
The first request was made at 14:26:12 and failed with a Unavailable error at 14:26:13, which promps a new connection to be made.

But if our keepalive frequency is set to 5 Minutes, how did this recover so quickly and why did the first failure not result in a new connection?

We'll start with the latter question, because it's easier to answer. Because we applied the iptables rule in the middle of the request, the ACK had already been received and we ended up hitting the context timeout which we set in the client code below:

ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()

To answer the former question, we're going to need to change some settings in both the client and server. First let's increase the allowed frequency of keepalive requests by setting the EnforcementPolicy.MinTime to 5 seconds from the default 5 minutes:

// server/main.go

// Change this
g := grpc.NewServer()

// To this
var kep = keepalive.EnforcementPolicy{
	MinTime:             5 * time.Second,
	PermitWithoutStream: true,
}

g := grpc.NewServer(grpc.KeepaliveEnforcementPolicy(kep))

Then we'll change the client to send a keepalive every 6 seconds instead of every 5 minutes:

// client/main.go

// Change this
kap := grpc.WithKeepaliveParams(keepalive.ClientParameters{
	Time: 5 * time.Minute,
	PermitWithoutStream: true,
})
conn, err := grpc.NewClient(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()), kap)

// To this
kap := grpc.WithKeepaliveParams(keepalive.ClientParameters{
	Time: 6 * time.Second,
	PermitWithoutStream: true,
})
conn, err := grpc.NewClient(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()), kap)

I also changed the frequency of requests from every 20 seconds, to every 20 minutes, so it's easy to see just the keepalive traffic. This isn't strictly required, so long as the keepalive Time is less than the request frequency, but it was easier for me to figure out what was going on:

// client/main.go

// Change this
ticker := time.NewTicker(20 * time.Second)

// To this
ticker := time.NewTicker(20 * time.Minute)

Now if we run the server with HTTP debug logs enabled (GODEBUG=http2debug=2 go run server/main.go), we can pull out and focus on the keepalive traffic. The following represents two gRPC keepalive requests, 6 seconds apart, and each comprises of sending a PING frame and receiving and ACK.

2025/07/04 11:55:45 http2: Framer 0xc0000020e0: read PING len=8 ping="\x00\x00\x00\x00\x00\x00\x00\x00"
2025/07/04 11:55:45 http2: Framer 0xc0000020e0: wrote PING flags=ACK len=8 ping="\x00\x00\x00\x00\x00\x00\x00\x00"
2025/07/04 11:55:55 http2: Framer 0xc0000020e0: read PING len=8 ping="\x00\x00\x00\x00\x00\x00\x00\x00"
2025/07/04 11:55:55 http2: Framer 0xc0000020e0: wrote PING flags=ACK len=8 ping="\x00\x00\x00\x00\x00\x00\x00\x00"

If the client doesn't receive an ACK within the specified Timeout (default 20 seconds), gRPC will consider the connection dead/closed and will generate a new one.

Now if we reset everything back to the way it was on the client side (keepalive set to every 5 minutes, requests sent every 20 seconds), and run the server with HTTP debug logs enabled again we can inspect the request traffic:

2025/07/04 12:06:09 http2: Framer 0xc0000020e0: read HEADERS flags=END_HEADERS stream=3 len=16
2025/07/04 12:06:09 http2: decoded hpack field header field ":method" = "POST"
2025/07/04 12:06:09 http2: decoded hpack field header field ":scheme" = "http"
2025/07/04 12:06:09 http2: decoded hpack field header field ":path" = "/keepalive.v1alpha1.Example/Heartbeat"
2025/07/04 12:06:09 http2: decoded hpack field header field ":authority" = "localhost:5000"
2025/07/04 12:06:09 http2: decoded hpack field header field "content-type" = "application/grpc"
2025/07/04 12:06:09 http2: decoded hpack field header field "user-agent" = "grpc-go/1.73.0"
2025/07/04 12:06:09 http2: decoded hpack field header field "te" = "trailers"
2025/07/04 12:06:09 http2: decoded hpack field header field "grpc-timeout" = "14999842u"
2025/07/04 12:06:09 http2: Framer 0xc0000020e0: read DATA flags=END_STREAM stream=3 len=5 data="\x00\x00\x00\x00\x00"
2025/07/04 12:06:09 http2: Framer 0xc0000020e0: wrote WINDOW_UPDATE len=4 (conn) incr=5
2025/07/04 12:06:09 http2: Framer 0xc0000020e0: wrote PING len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"
2025/07/04 12:06:09 http2: Framer 0xc0000020e0: read PING flags=ACK len=8 ping="\x02\x04\x10\x10\t\x0e\a\a"

The last two lines show that, on a normal gRPC request, the client and server are still doing the PING/ACK dance. So I believe we saw a new connection created much quicker than the 5 minute interval set for keepalives because a normal request was made, a PING frame was sent, but no ACK was received in the required timeframe so gRPC considered the connection dead/closed and created a new one.

gRPC keepalive vs tcp_retries2

If you've got to this point, you may be wondering why setting the gRPC keepalive works even though the tcp_retries2 limit hasn't been exhausted. Firstly, it's important to understand that they both operate at different layers; gRPC keepalive operates at the application layer (HTTP/2) and tcp_retries2 operates at the transport layer (TCP).

As we learnt in the earlier tcp_retries2 section, this is a system level setting that handles situations where a TCP packet is sent but not acknowledged; with the value of tcp_retries2 deciding the number of retries with exponential backoff.

The gRPC keepalive, on the other hand, is an application level setting that is used to send HTTP/2 pings to detect if the connection is down. If the ping is not acknowledged by the other side within a certain time period, the connection is closed.

That is to say if we're relying on tcp_retries2 only, then the application has to wait for a signal from the TCP layer that the connection is closed and to start a new one, but by using gRPC keepalives the application can configure it's own settings to determine whether the connection is closed.

Using the below example the gRPC keepalive will issue a HTTP/2 ping every 5 minutes and, if a ping ACK is not received within 20 seconds, the connection is deemed closed and a new one is created.

keepalive.ClientParameters{
	Time:                5 * time.Minute,  // send pings every 5 minutes if there is no activity
	Timeout:             20 * time.Second, // wait 20 seconds for ping ack before considering the connection dead
	PermitWithoutStream: true,             // send pings even without active streams
}

gRPC connections stuck for 15 minutes