Oreoft

Turning My Windows Mini PC Loaded with Random Chores into PVE

Sat, 03 Oct 2026 00:00:00 +0800

Introduction

I have a Beelink SER mini PC at home equipped with an AMD Ryzen 7 5800H CPU (8 cores, 16 threads), two 16GB DDR4 3200 RAM sticks, and a 1TB NVMe SSD. Since my home broadband comes with a public IP, it has been running 24/7 as a home server.

It had always been running Windows. Over time, I piled more and more tasks onto it, but the stability was never quite where I wanted it to be. This post documents why I eventually decided to reinstall everything with PVE (Proxmox VE), how I set it up, and how the virtual machines are partitioned now. Along the way, I also tore it down for some dust cleaning, so there are a few pictures included.

p.s. This is more of an experience and mindset log rather than a step-by-step tutorial.

1. More and More Work Piled on Windows

Initially, it was just an ordinary Windows machine. Later, I assigned more and more jobs to it:

Various Services: A few self-written services (like an AI service), ddns-go (which automatically updates my dynamic public IP to my domain), an Nginx gateway, and proxy utilities like Hysteria 2 (hy2).
NAS & Smart Home: fnOS (Feiniu OS) and HAOS (Home Assistant OS).
Windows-Exclusive Applications: A few apps that only exist on Windows and must run in a Windows environment.

Among these, only the third category genuinely required Windows. The first two were just piggybacking on it.

2. Stability Issues Never Went Away

Using Windows as the host OS came with two major problems:

Windows itself is a resource hog. The OS disk easily takes up dozens of gigabytes, a bunch of useless background services run constantly, and it eats up a large chunk of RAM.
It restarts on its own. No matter how you tweak the Windows Update policies, it always finds an excuse to reboot every once in a while. When the host reboots, every single service goes down, and fnOS disconnects along with them.

3. Three Iterations of Service Deployment

To make those services run more stably, I cycled through three different approaches.

1. The Hyper-V Era

Initially, I spun up Linux VMs inside Hyper-V (Windows’ built-in hypervisor) to host services, and fnOS ran there too. That setup suffered from three issues:

Heavy resource overhead. Each VM had to run a full Linux OS. As services grew, each VM added its own OS-level overhead.
Unreliable auto-start after power loss. Sometimes after a power outage and recovery, the VMs would fail to start up properly.
Too bloated overall. Later, I thought WSL would be lighter, so I switched to WSL.

2. The WSL2 Era

WSL2 is the subsystem for running Linux inside Windows. It is much lighter than traditional VMs, but brought a new set of headaches:

Networking pain. Although WSL supports mirrored networking (sharing network adapters and IPs with the Windows host), configuring it reliably is a real pain.
File sharing with the host was prone to errors. Service logs and miscellaneous files resided on the Windows side. WSL’s default file sharing threw errors when files were modified concurrently from both sides. It only improved after switching over to SMB shares.
Randomly exiting. Even with active background services, WSL would terminate itself out of nowhere. I had to write a keepalive script on Windows just to keep it alive.

3. All In Docker based on Windows

Next, I installed Docker directly on Windows and shoved all services into containers. But Docker Desktop for Windows essentially runs on top of WSL anyway—it is just a dedicated WSL distro running the Linux version of Docker. So that extra layer remained, and all the quirks with it.

The most memorable issue: UDP port forwarding in Docker on Windows goes through an extra layer, leading to cases where the container is clearly alive, but the port stops accepting traffic. To work around this, I had to deploy a “sidecar” container specifically for hy2: every 30 seconds it performed a real handshake, and if it failed, it fired an alert and restarted the service. It worked, but needing such a hack proved the foundation was flawed. Along with various other messy glitches, it was simply exhausting.

After cycling through three different solutions, the root cause remained the host layer: as long as the host was Windows, its reboots, resource hogging, and translation layer for the Linux ecosystem were unavoidable.

4. Why Switch to PVE

This machine actually packs plenty of power; there is no need to bottle it all up under a single Windows instance.

Proxmox VE (PVE) is a Debian-based virtualization platform installed bare-metal, allowing you to manage VMs and containers via a web UI. After migrating to it, the strategy became clear:

The host does nothing but virtualization. It runs no user workloads and will not reboot out of the blue.
When Windows is needed, spin up a lightweight Windows VM solely to run Windows-exclusive apps.
Offload other services to a stable Linux VM, running all Docker applications natively without translation layers.
Give systems like fnOS and HAOS their own dedicated VMs.

This way, CPU cores and RAM can be allocated on demand. If something breaks, only that specific VM needs a reboot without affecting others. Backups and snapshots are handled natively by PVE. It is a clean slate, and expanding in the future becomes seamless.

5. Cleaning Out the Dust First

Before getting to work, I took the machine apart for a quick dust cleanup.

Two 16GB RAM sticks, with the 1TB SSD on the right

Inside are the two 16GB RAM modules (bought back before RAM prices surged) and next to them is the 1TB SK Hynix NVMe SSD. Overall, the hardware specs are quite solid.

When cleaning the bottom cover, I facepalmed: turns out I had never peeled off the protective film on the SSD’s thermal pad.

Thermal pad on the bottom plate; that blue tab is the unpeeled protective film

While cleaning, I noticed a SATA connector on the bottom bracket, which reminded me of an old 2.5-inch mechanical HDD sitting around.

An old mechanical hard drive removed from an external enclosure

This was salvaged from an ancient Mac over a decade ago. Since then, it barely saw any use except as a spare, and I had bought an external enclosure for it back then. Now that I use NVMe external enclosures, this drive was totally obsolete. Might as well slap it into the mini PC as a backup/storage drive.

Installing the mechanical HDD into the bottom bracket slot

Being an older drive, it is 9.5mm thick. Even though it is a 2.5-inch drive, it was a tight squeeze and made the bottom cover bulge slightly. But after tightening down the four corner screws, it did not stick out too much.

The bottom cover bulges a bit after installation

It looks a bit funny with its potbelly, but it works without issues. I later formatted it specifically to store PVE VM backups.

6. Backing Up the Data First

My Mac has an 8TB drive, which is more than enough for the migration, so I dumped everything onto the Mac first. The data I actually needed to move was pretty minimal:

fnOS Virtual Disk: Originally 250GB. I copied out the large files first, reclaimed unused space inside fnOS, shrunk the virtual disk, and got it down to just 33GB.
Configurations and data for each service: Totaled around 2GB. Docker images and Git repos can always be pulled again, so no need to back those up.

After copying and verifying checksums, I started working on the machine.

Copying large files from fnOS over to the Mac first, totaling over 200GB

Running `fstrim` inside fnOS to reclaim 257GB of free space before shrinking the virtual disk

7. BIOS Configuration and PVE Installation

First, I flashed the PVE ISO onto a USB drive on my Mac.

Using `dd` to write the PVE 9.2 ISO to the USB drive

Then, I went into the BIOS and adjusted a few settings:

Enable virtualization: SVM (AMD CPU virtualization) and IOMMU (required for device passthrough).
AC Power Loss recovery: Automatically powers on after an outage without needing someone to press the power button.
Reduce iGPU VRAM to 1GB (previously 3GB when Windows was the host; now I can lower it to free up more RAM).
Set System Mode to Performance Mode and enable CPPC (enabling finer-grained CPU frequency scaling, which the PVE governor relies on).

BIOS main screen: R7 5800H with 32GB RAM

Enable SVM

Enable IOMMU

Auto power-on upon AC recovery

System Mode set to Performance Mode

Enable CPPC

After applying settings, I booted from the USB drive to install PVE 9.

PVE installer screen

Installing the OS on the 1TB SSD with ext4 filesystem

Assigning a static LAN IP to the host

After installation, I made a small tweak: set the CPU scaling governor straight to performance mode. This machine runs 24/7 plugged in—I don’t care about power consumption, I only care about performance.

8. Allocating Virtual Machines

There are currently four VMs running on PVE:

Virtual Machine	OS	Specs	Purpose
ser-docker	Debian 13	4 Cores / 4G	All Docker services
ser-fnos	fnOS (Feiniu OS)	4 Cores / 4G	NAS, existing virtual disk imported directly
ser-win	Windows Server 2025	4 Cores / 6G	Runs only Windows-exclusive apps
ser-mac	macOS 15 (Hackintosh)	8 Cores / 8G	Dedicated to AI Agent use

After migrating to the Docker VM, I immediately deleted the “sidecar” watchdog container previously created for hy2. On Linux, UDP packets are routed directly by the kernel, so the old bug simply vanished.

This time around, I installed Windows Server instead of Windows 11. The Server edition cuts out the bloatware and offers granular control over system updates, making it much better suited for long-running apps. There was one minor hiccup during installation: PVE assigns VirtIO virtual disks, which the Windows installer doesn’t recognize out of the box—no disks showed up until I manually loaded the VirtIO drivers.

Windows Server installer cannot find any drive; VirtIO drivers must be loaded first

Using Sysinternals Autologon to configure automatic logon on startup

Finally, I also set up a Hackintosh VM. Since the iGPU cannot be passed through or properly accelerated, GUI rendering falls back completely on CPU software rendering, making it a bit sluggish. However, this CPU is decent enough that without graphic-intensive workloads, pure CPU brute-forcing handles most compilation and web browsing tasks fine. It is more than enough for an AI Agent.

Hackintosh running on PVE

My daily driver Mac is plenty powerful, so the division of labor is straightforward: compilation, iOS development, and graphical tasks happen on my local Mac; miscellaneous chores like web browsing are delegated to the Agent running on the virtual Hackintosh. Neither interferes with the other, and I do not have to surrender my own workstation to the Agent.

Conclusion

After years of tinkering across Hyper-V, WSL2, and Windows Docker, I finally realized the problem wasn’t the individual solutions, but the host OS itself. Once the host switched to PVE, Windows retreated to what it does best—just a dedicated VM for Windows apps—while everything else runs natively on Linux.

Installing Tailscale on Alibaba Cloud Blocked All of Alibaba Cloud’s Own Internal Services

Sun, 20 Sep 2026 00:00:00 +0800

Introduction

I’ve had an Alibaba Cloud machine sitting around for a long time. I bought it during a promo, and later renewed it for 20 years. The specs and bandwidth aren’t great (2 vCPU, 2 GB RAM, 3 Mbps), and since I usually manage more machines on GCP and use GCP more often, this one has mostly been idle.

Recently I wanted to give it something useful to do. I have a few servers at home running various services, and they go down from time to time—sometimes because the machine rebooted, sometimes because memory blew up. When a service dies, I usually don’t notice until something in production starts breaking. I’ve never had a unified monitoring setup. Using the home machines to monitor themselves doesn’t feel very reliable if they’re the ones going down; this Alibaba Cloud box may be underpowered, but it’s stable, which makes it a good fit for health checks.

My plan was to connect everything with Tailscale. Tailscale is a WireGuard-based networking tool that pulls machines scattered across different places into the same virtual private network. Each machine gets a 100.x.x.x address, and they can talk to each other like they’re on the same LAN. I’ve installed it plenty of times on AWS and GCP machines and basically never had issues. But this time on Alibaba Cloud, right after installing it, DNS resolution, apt package installs, and pulling images from registries all started failing one after another.

The reason can be summed up in one sentence: Alibaba Cloud puts its internal services in the 100.64.0.0/10 range, and Tailscale uses that same range. On Linux, Tailscale installs an anti-spoofing firewall rule that drops any packet that “claims to come from that range but didn’t arrive through the Tailscale interface.” Replies from Alibaba Cloud’s internal services got caught by that rule.

This post is a record of how I traced it down step by step, a few workarounds I tried along the way, and the official Tailscale solution I ended up using.

p.s. The system is Ubuntu 22.04, and Tailscale is 1.102.

1. What this machine is supposed to do

First, here’s how I planned to use it, because a lot of the decisions later are tied to this.

Network access: connect all services that need monitoring to the Tailscale private network. This machine will hit each service’s health check endpoint over the private network, then expose a dashboard.
Access management: normally I’ll manage it directly over the Tailscale network; on the public internet, I’ll only expose SSH on port 22, so I can still get in when I’m not on the private network.
Dashboard access: access the dashboard directly from inside the private network; when I’m outside it, use ssh -L to forward the dashboard port locally.

Overall architecture: health checks and management go through the Tailscale private network, and only port 22 is exposed publicly

Health checks, management, and dashboard access all go through Tailscale. So whether Tailscale can work properly on this machine is the prerequisite for the whole setup.

2. After installing Tailscale, DNS resolution completely broke

After installing Tailscale and joining the private network, domain resolution on this machine stopped working. getent hosts returned nothing:

$ getent hosts github.com
$

My first suspicion was upstream DNS or outbound connectivity. If that were the case, querying a public DNS server directly should also fail. So I checked:

$ dig +short github.com @8.8.8.8
20.205.243.166
$ dig +short github.com
;; communications error to 127.0.0.53#53: timed out

Querying 8.8.8.8 worked, but using the default resolver timed out. Outbound networking was fine, so the problem was in the local resolution path.

Ubuntu 22.04 uses systemd-resolved for local DNS by default: /etc/resolv.conf points to 127.0.0.53, which is a local forwarding stub, while the real upstream resolvers are configured elsewhere. So I checked which upstreams it was using:

$ resolvectl status
Link 2 (eth0)
    DNS Servers: 100.100.2.136 100.100.2.138
Link 3 (tailscale0)
    DNS Servers: 100.100.100.100

The two addresses on eth0 are Alibaba Cloud internal DNS servers provided by DHCP. The 100.100.100.100 on tailscale0 is Tailscale’s own DNS, used to resolve machine names inside the tailnet. Tailscale calls this MagicDNS.

The address 100.100.2.136 is the important one here. Tailscale assigns machine addresses from 100.64.0.0/10, which is the RFC 6598 reserved range for carrier-grade NAT (CGNAT), covering 100.64.0.0 through 100.127.255.255. Alibaba Cloud’s internal DNS also falls inside that range.

If Tailscale applies restrictions to that range, Alibaba Cloud DNS would get caught too. So I looked at the iptables rules Tailscale had installed:

$ iptables -S ts-input
-A ts-input -i tailscale0 -j ACCEPT
-A ts-input -p udp -m udp --dport 41641 -j ACCEPT
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP

The last line means: drop any packet whose source address is in 100.64.0.0/10 unless it came in through tailscale0.

The intent of this rule is anti-spoofing. Every machine in the Tailscale network uses an address from this range, and legitimate Tailscale traffic should only arrive through the virtual tailscale0 interface. If a packet comes in from a physical NIC but claims to be from 100.x, that looks like someone pretending to be a Tailscale machine, so dropping it makes sense.

But Alibaba Cloud’s internal DNS also lives in that range. The machine sends a query to 100.100.2.136, which goes out through eth0 just fine; the reply comes back with source address 100.100.2.136, enters through eth0, and gets hit by that rule.

How the same rule treats three kinds of packets: spoofed packets and Alibaba Cloud replies look identical to it

I verified it directly:

$ dig +short +tries=1 github.com @100.100.2.136
;; communications error to 100.100.2.136#53: timed out
$ dig +short +tries=1 github.com @223.5.5.5
20.205.243.166

Alibaba Cloud internal DNS timed out, while Alibaba Cloud’s public DNS 223.5.5.5 (outside that range) worked. That matched the theory.

One extra note: this rule has nothing to do with Tailscale’s exit node feature (using one machine as the internet gateway for other devices). As long as tailscaled is running, it installs this rule.

Actually, before reinstalling the OS, when I was doing some initial checks on this machine, I had already noticed that I couldn’t reach Alibaba Cloud’s instance metadata service at 100.100.100.200 (used for things like instance ID and security group info). It was blocked by the same rule. Back then it only affected metadata. This time it was DNS, which made the impact global—anything that needed domain resolution stopped working.

That also explains why I’d never seen this on AWS or GCP: their metadata service and DNS live in the 169.254.x.x link-local range, not in the CGNAT range, so they never collide with this rule.

3. First workaround: switch DNS to public DNS

Once I knew the cause, there were a few possible directions:

Approach	Problem
Add an allow rule in the `ts-input` chain	`ts-input` is managed by `tailscaled` itself and gets rebuilt on every start, so manually added rules get wiped
`tailscale up --accept-dns=false`	Prevents Tailscale from taking over DNS, but then I lose the ability to access other machines in the tailnet by hostname
Switch DNS to public resolvers outside this range	Works, but means no longer using Alibaba Cloud internal DNS

At the time, I chose the third option. Alibaba Cloud’s DNS is provided via DHCP, so replacing it required changing two things together: first tell netplan not to accept DNS from DHCP, then explicitly set upstream resolvers for systemd-resolved.

# /etc/netplan/99-dns-override.yaml
network:
  version: 2
  ethernets:
    eth0:
      match:
        macaddress: 00:16:3e:xx:xx:xx
      set-name: eth0
      dhcp4-overrides:
        use-dns: false

# /etc/systemd/resolved.conf.d/99-public-dns.conf
[Resolve]
DNS=223.5.5.5 119.29.29.29

Resolution came back, and MagicDNS was unaffected because it goes through the tailscale0 interface.

4. The same problem showed up again with `apt`

After fixing DNS, I went to install Docker, and then apt update failed too:

W: Failed to fetch http://mirrors.cloud.aliyuncs.com/ubuntu/dists/jammy-security/InRelease
   Unable to connect to mirrors.cloud.aliyuncs.com:http:

On Alibaba Cloud Ubuntu images, the default package source is mirrors.cloud.aliyuncs.com, which is an internal Alibaba Cloud mirror. I checked what it resolved to:

$ getent hosts mirrors.cloud.aliyuncs.com
100.100.2.148   mirrors.cloud.aliyuncs.com

Still 100.100.x.x, so it was the same root cause again.

Following the same logic as DNS, I switched the source to the public mirror mirrors.aliyun.com. This time it worked, but it was slow:

	Time
`apt update`	366 seconds
Install Docker	still not finished after 25 minutes

The reason was bandwidth. Traffic over Alibaba Cloud’s internal network is free and not rate-limited; public internet traffic has to squeeze through this machine’s 3 Mbps bandwidth. A full Ubuntu apt update downloads over 100 MB of index files, which takes several minutes at 3 Mbps. Previously, using the internal mirror didn’t consume public bandwidth at all. Once that internal path was blocked, all traffic got pushed onto the public network.

At this point, I had hit the same root cause for the third time: metadata, DNS, and apt mirrors. Alibaba Cloud’s internal services mostly live in 100.100.x.x. Replacing them one by one with public endpoints was just whack-a-mole, and every replacement consumed more public bandwidth. I needed to make the whole range work again.

5. Put an allow rule at the very top of the INPUT chain

As mentioned above, I couldn’t add a rule directly to ts-input, because that chain gets rebuilt. But iptables matches in order, and the INPUT chain only jumps to ts-input on its first line. If I allow Alibaba Cloud packets in INPUT before that jump, they’ll never reach the DROP rule:

iptables -I INPUT 1 -i eth0 -s 100.100.0.0/16 -j ACCEPT

The -i eth0 restriction is the key: Tailscale’s own traffic comes through tailscale0, so this rule won’t accidentally allow spoofed Tailscale traffic.

After adding it, internal DNS, apt mirrors, and the metadata service all started working again.

Before making it persistent, I needed to confirm one thing: when tailscaled restarts, does it reinsert its own jump back at the top? If it does, my rule would get pushed down to line 2 and become useless. So I tested:

# Before restart
1    ACCEPT     all  --  100.100.0.0/16
2    ts-input   all  --  0.0.0.0/0

# After systemctl restart tailscaled
1    ts-input   all  --  0.0.0.0/0
2    ACCEPT     all  --  100.100.0.0/16

It does. Every time tailscaled starts, it uses -I to put its jump back at line 1, and my manual rule immediately stops being effective.

So it wasn’t enough to just save an iptables rule and restore it at boot. I needed to move the rule back to line 1 every time tailscaled started. I wrote a systemd unit that follows tailscaled:

[Unit]
After=tailscaled.service
PartOf=tailscaled.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/aliyun-internal-allow.sh

[Install]
WantedBy=tailscaled.service

PartOf makes it restart when tailscaled restarts, and WantedBy makes it get pulled in when tailscaled starts.

There was also a timing issue in the script: when the tailscaled service becomes active, its iptables rules may not have been installed yet. So the script first waits for the ts-input jump to appear, then moves my rule back to line 1:

for i in $(seq 1 30); do
  iptables -C INPUT -j ts-input 2>/dev/null && break
  sleep 1
done
iptables -D INPUT -i eth0 -s 100.100.0.0/16 -j ACCEPT 2>/dev/null
iptables -I INPUT 1 -i eth0 -s 100.100.0.0/16 -j ACCEPT

After restarting tailscaled again, the rule stayed at line 1. Then I switched both apt and DNS back to Alibaba Cloud internal endpoints:

	Public mirror	Internal mirror
`apt update`	366 seconds	18 seconds
Install Docker	not finished after 25 minutes	19 seconds

6. The internal address for the image registry wasn’t in that range

After Docker was installed, I needed to pull images. Accessing Docker Hub from mainland China is basically unreliable, so I planned to use Alibaba Cloud Container Registry (ACR) as a relay. I checked the ACR internal endpoint:

$ getent hosts registry-vpc.cn-hangzhou.aliyuncs.com
100.103.7.180   registry-vpc.cn-hangzhou.aliyuncs.com

100.103.x.x—not inside the 100.100.0.0/16 range I had just allowed, so it would still be blocked.

Alibaba Cloud internal services are not all in 100.100.x.x. To cover everything, I’d have to widen the allow rule to the full 100.64.0.0/10 range. Before doing that, I went to see how other people were handling this problem.

7. Community workarounds, and Tailscale’s official solution

There’s an issue on Tailscale’s GitHub describing this exact problem: after installing Tailscale on Alibaba Cloud ECS, internal DNS and apt both stop working. It’s still open. There are also quite a few Chinese blog posts about it. Summarizing the approaches I found:

Approach	Notes
Move DNS and mirrors to public endpoints	Works, but consumes public bandwidth and requires changing services one by one
`--accept-dns=false`	Only fixes DNS; `apt` and image registries still fail
Manually insert allow rules	This is what I did above; you also have to deal with reordering when `tailscaled` restarts
`--netfilter-mode=nodivert`	Tailscale creates rule chains but does not attach the jump; you manage the jump yourself
`disable-linux-cgnat-drop-rule`	A node attribute provided officially by Tailscale

Tailscale’s docs mention that nodivert was the recommended approach before disable-linux-cgnat-drop-rule existed, but now the latter is the recommended one.

disable-linux-cgnat-drop-rule is a node attribute. If you assign it to a specific machine in the Tailscale policy file, tailscaled changes that DROP rule in ts-input to RETURN: instead of dropping the packet, it hands it off to the following rules. Since this rule is generated by tailscaled itself, there’s no issue with it being overwritten on restart, which means I no longer needed that systemd unit babysitting my custom rule.

8. If anti-spoofing is disabled, what replaces it?

The official docs also mention the tradeoff: removing that rule removes the anti-spoofing protection, so machines on the local network could send packets pretending to come from Tailscale addresses. The recommended compensation is to enable reverse path filtering (rp_filter).

rp_filter is a kernel check: when a packet arrives, the kernel asks, “if I were to send a reply to this source address, which interface would I use?” If that doesn’t match the interface the packet actually came in on, the packet gets dropped. In strict mode (rp_filter=1), it has to be the exact same interface.

There was one thing I needed to confirm first: if Tailscale had installed a route for the entire 100.64.0.0/10 range via tailscale0, then replies from Alibaba Cloud DNS would also be considered “supposed to come in through tailscale0”, and strict mode would drop them too, bringing me right back to square one.

So I checked Tailscale’s routing table. It uses table 52:

$ ip route show table 52
101.102.103 dev tailscale0
101.102.104 dev tailscale0
100.100.100 dev tailscale0

These are per-host routes, not a route for the whole range. Then I checked the return path for a few addresses:

$ ip route get 100.100.2.136      # Alibaba Cloud DNS
100.100.2.136 via 172.16.x.x dev eth0
$ ip route get 100.103.7.180      # ACR internal
100.103.7.180 via 172.16.x.x dev eth0
$ ip route get 100.101.102.103    # A machine in Tailscale
100.101.102.103 dev tailscale0

Alibaba Cloud internal services route back through eth0, and they also arrive through eth0, so they pass. If someone on eth0 spoofs a packet pretending to be a Tailscale machine, the return path should go through tailscale0, which doesn’t match, so it gets dropped.

Strict mode turns out to be exactly the anti-spoofing protection I needed, and it’s more precise than the old blanket DROP rule: it only protects real Tailscale addresses and doesn’t accidentally hit Alibaba Cloud.

After changing DROP to RETURN and enabling rp_filter, how the same three kinds of packets are handled

9. What I actually changed

1. Add a tag and a node attribute in the policy file

"tagOwners": {
  "tag:aliyun": ["autogroup:admin"],
},
"nodeAttrs": [
  {
    "target": ["tag:aliyun"],
    "attr":   ["disable-linux-cgnat-drop-rule"],
  },
],

I used a tag instead of applying it directly to my own account because this attribute affects all Linux machines, and I also have some Linux machines outside Alibaba Cloud joining the tailnet. There’s no reason to disable anti-spoofing on those. Using a tag lets me target only the Alibaba Cloud machines.

Before changing it, I backed up the original policy file. After editing it, I first called Tailscale’s validation API to make sure it was fine, and only then applied it for real.

2. Tag this machine with `tag:aliyun`

You can do this from the machine list in the admin console, or via the API. As soon as I applied the tag, the rule on the machine changed immediately:

-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j RETURN

3. Enable `rp_filter`

# /etc/sysctl.d/999-ali-hz-tuning.conf
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

The filename needs to start with 999-—this is a weird Alibaba Cloud-specific pitfall, which I’ll explain in the next section.

4. Remove the old systemd unit

The previous workaround was no longer needed. After removing it, the INPUT chain only had the ts-input jump left.

Verification

Internal DNS   100.100.2.136     OK
apt mirror     100.100.2.148     HTTP 200
Metadata       100.100.100.200   OK
ACR internal   100.103.7.180     HTTP 401      ← previously it timed out

The 401 from ACR is expected—an image registry returns 401 if you haven’t logged in. The important part is that it responded; before, it just timed out. Docker container networking, SSH over the Tailscale network, and public SSH all worked normally too. I restarted tailscaled one more time, and the RETURN rule was still there.

10. Alibaba Cloud’s built-in sysctl config can override yours

Files under /etc/sysctl.d/ are loaded in filename order, and later-loaded files override earlier ones. Alibaba Cloud images ship with a 99-apsara-sysctl.conf file that contains these lines:

vm.swappiness = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.*.rp_filter = 0

It disables rp_filter entirely. If your config file is named something like 99-xxx.conf, it may sort before 99-apsara, and then get overridden back to 0 at boot.

I actually discovered this pitfall through swappiness first. In an earlier tuning pass, I had written vm.swappiness = 10 in 99-ali-hz-tuning.conf. If I loaded just that file with sysctl -p, the runtime value was indeed 10. But when I used sysctl --system to simulate the full boot-time load order, the result was 0—overridden by 99-apsara. So every reboot silently undid it.

Renaming the file to start with 999- fixed it, because it sorts after all 99-* files (9 has ASCII code 0x39, which is greater than - at 0x2d).

There’s another detail here: if you just use ls, 999- may appear before 99-apsara, because UTF-8 collation can ignore punctuation. But systemd-sysctl, which loads sysctl settings at boot, uses byte order. To see the real order, use LC_ALL=C ls:

$ LC_ALL=C ls /etc/sysctl.d/
...
99-apsara-sysctl.conf
99-sysctl.conf
99-tailscale.conf
999-ali-hz-tuning.conf

To verify sysctl config, use sysctl --system and check the final value. Don’t rely only on sysctl -p for a single file. The latter only tells you your file is syntactically fine; it doesn’t tell you whether it wins in the end.

11. Retrospective

1. If you install Tailscale on Alibaba Cloud, do these steps first

In the policy file, give Alibaba Cloud machines a tag and add disable-linux-cgnat-drop-rule
Enable rp_filter=1, and make the config filename start with 999-
Keep Alibaba Cloud’s default internal DNS and apt mirror config; no need to change them

If you do this before installation, you won’t run into the whole chain of problems above.

2. If the same root cause shows up a second time, it’s time to look for the real cause

This time I hit the same issue four times in a row: metadata, DNS, apt, and the image registry. DNS and apt were each “fixed” by switching to public endpoints, and each fix worked in isolation—but together they were just whack-a-mole, and every change consumed more public bandwidth.

3. If you manually insert firewall rules, check whether some other program will rewrite them

The rule I manually inserted into INPUT got pushed down to line 2 after tailscaled restarted. If I hadn’t verified that, the rule would have silently stopped working after some future restart.

4. Check the official docs first

Most community solutions involve writing your own script to insert rules, or using nodivert, but Tailscale already has an official node attribute for exactly this case. If I had gone through the docs earlier, I could have skipped that whole detour in the middle.

Summary

The root cause this time was simple: Alibaba Cloud puts internal services in 100.64.0.0/10, and Tailscale uses the same range. Tailscale’s anti-spoofing rule dropped all replies from Alibaba Cloud internal services. AWS and GCP put their internal services in 169.254.x.x, which is why I’d never hit this before.

In the end, I used Tailscale’s official disable-linux-cgnat-drop-rule, combined with rp_filter=1 to restore anti-spoofing protection. DNS, apt, and the image registry all went back to using Alibaba Cloud’s internal network, with no public bandwidth cost and no custom firewall rules to maintain. Along the way I tried public DNS, public mirrors, and manually inserted rules—each one worked at the time, but each only solved the immediate symptom for one service. The actual problem was a subnet conflict, and the final fix was also applied at the subnet level.

References

After Upgrading the Client, Redis Started Returning Tons of MOVED Errors Again: A Local Config That Had Been Lurking for Seven Months

Sat, 05 Sep 2026 00:00:00 +0800

Introduction

At the end of the previous post (We added a shard to Redis and broke production writes for over an hour), upgrading redis-py from 5.0.0 to 8.x stopped the client hang issue.

What it stopped was only the hanging.

The next day, when I looked at that cluster again, INFO errorstats was sitting on tens of millions of MOVED responses, and the rejection rate for read commands was hovering around 40%:

zrevrange         42.5% MOVED
zrangebyscore     35.6%
hgetall           44.9%

In CloudWatch, each shard was creating 440 new connections per second, but only 280 were connected at any given time. In other words, each connection lived for about half a second on average before being dropped and recreated.

There was business impact too. In the recommendation service, one provider is responsible for reading users’ like, play, and favorite history. If the Redis read fails, the whole provider throws an exception. An outer except Exception catches it, logs one critical line, and then continues. The API still returns 200 as usual. So the user profile data was silently dropped, and the request itself showed no obvious error. Worse, that critical log line never made it into Datadog, so this incident left almost no visible trace.

(Image placeholder: Redis error curve in Datadog, or CloudWatch NewConnections curve staying high)

This post records the next day and a half of investigation. The final cause wasn’t complicated, but I think the way those intermediate hypotheses were proposed and then ruled out is more useful than the conclusion itself.

1. First, figure out which machines are connecting

This time I didn’t start by guessing the cause. I started by figuring out who was connecting to this cluster.

CLIENT LIST showed 21 client IPs. I mapped them back through ECS, and got this:

Service	Machine Count	redis-py	Connection Lifetime
Recommendation service	8	8.1.0	1–3 seconds
API service	8	8.0.0	1.8–5 hours
Event consumer service	1	7.1.0	68 minutes

All 8 recommendation service machines were constantly rebuilding connections. Right next to it, another backend API service (I’ll call it the API service below) was connecting to the same cluster, also using redis-py 8.x, and its connections lived for hours. Same cluster, same major library version, but one was on a seconds scale and the other on an hours scale. From this point on, the API service became the best control group.

Then I looked at error attribution. The server-side errorstats is a global counter and doesn’t distinguish clients, but the three services happened to use completely non-overlapping command sets: the recommendation service used zrevrange and zrangebyscore, the API service only used zrevrangebyscore, and the event consumer service only used zremrangebyscore. Matching that against the rejection rates:

zrevrange          42.5% MOVED   ← exclusive to recommendation service
zrangebyscore      35.6%         ← exclusive to recommendation service
zrevrangebyscore    0.004%       ← exclusive to API service
zremrangebyscore    0.0%         ← exclusive to event consumer service

Same cluster, same moment in time: commands used only by the recommendation service were getting 40% MOVED, while the other two services were at 0. All MOVEDs were coming from the recommendation service.

The problem was now compressed into one sentence: same library, same cluster — why does the recommendation service break while the API service doesn’t?

2. Hypothesis 1: the removed `socket_timeout`

The first thing that came to mind was socket timeout. There were two reasons.

First, the only client config difference between the recommendation service and the API service was exactly this: the API service had socket_timeout=3, socket_connect_timeout=2, while the recommendation service had removed those two parameters the day before. Second, the deleted comment was very explicit:

-  # Bounded connect/read/write: a hung socket times out and closes itself,
-  #   instead of exposing a blocked await to external cancellation
-  #   (which is exactly the window where pool slot leaks happen)

Without timeouts, a blocked await can only end through external cancellation, and cancellation is exactly the window where connection pool leaks can happen. It looked like a complete causal chain.

But when I checked it against reality, two things didn’t line up. First, timeout is a fallback mechanism. The service itself had no visible errors, and a healthy service should not have 440 hung sockets per second. Missing a fallback doesn’t create a failure; it just fails to catch one once it happens. Second, if this were really a leak, the number of live connections should keep climbing until it hit the limit of 300 and then start erroring. But it stayed stable at 280. Stable live connections plus hundreds of new connections per second means the connection objects themselves are not leaking — it’s the same set of objects whose sockets are repeatedly disconnecting and reconnecting.

So this hypothesis was ruled out.

3. Hypothesis 2: MOVED and connection rebuilds are triggering each other

The second idea came from reading the library source. redis-py handles MOVED like this:

except MovedError as e:
    self.reinitialize_counter += 1
    if self.reinitialize_counter % self.reinitialize_steps == 0:   # default 5
        await self.aclose()       # close the whole client
    else:
        await self.nodes_manager.move_slot(e)   # only fix this one slot

Every 5 MOVEDs, the client decides its topology is stale, closes and rebuilds the entire client, and invalidates all connections. And during each rebuild, set_nodes() marks all connections on all nodes as needing reconnect — the library author’s comment says it very directly: “reconnect is lazy and cheap”, assuming topology changes are rare.

So I got a chain that looked self-consistent: MOVED triggers rebuild, rebuild invalidates all connections, routing changes during rebuild, commands go to the wrong node, which creates more MOVEDs. The numbers even lined up: 208 MOVEDs per second, divided by 5, is about 41 rebuilds; measured CLUSTER SLOTS was about 30 times per second, same order of magnitude.

But one part of this chain didn’t make sense. If the connection is broken, the command should fail to send — why would it go to the wrong place? Connection health and routing correctness are two different things. So I went back to check the “routing table changes during rebuild” step: if redis-py can’t find a slot, it throws an exception; it doesn’t send randomly. And aclose() doesn’t clear the routing table either. That step was just an assumption.

This chain explains why connections were being rebuilt over and over, but it doesn’t explain where the MOVEDs came from. MOVED is the cause, not the effect. I had to look elsewhere.

4. Couldn’t reproduce it on the bastion host

So I moved to reproduction. The bastion host was in the same VPC and could connect directly to the cluster. I only did read operations.

I created a client with exactly the same constructor parameters as the recommendation service and sent a few hundred read commands: 0 MOVED. Increased concurrency to 256: still 0. Tried pipeline: 0. Added dependencies one by one, like hiredis and ddtrace: still 0.

Then I pointed all 16,384 slots in the routing table to a single shard, simulating the old pre-expansion topology — after about 20 commands, the client repaired itself automatically. That was actually useful information: a stale routing table could not be the cause, because the client can fix that on its own, while production had been stuck at 42% for two days.

5. A reproduction that didn’t hold up

After the bastion path stalled, I tried a different angle: force-refresh topology 33 times per second in the background while sending commands, and measure the increment using the server-side MOVED counter.

A  no refresh                  MOVED per command = 0.00
B  topology refresh 33/s       MOVED per command = 0.22

That looked like a signal. At the time I was ready to go change config in that direction.

But before changing anything, I hooked the client method that handles MOVED, because I wanted to capture what the client thought the slot owner was versus what the server said it was. Result: across 3000 commands, that method was never called once. My client had not produced a single MOVED.

So where did the 0.22 in group B come from? What I had measured was the server’s global counter, and the production recommendation service itself already had a background rate of 390 MOVEDs per second. In my measurement window, the background value was over nine thousand, and my observed “increment” was only one tenth of that, while production’s own fluctuation was already on that scale. That 0.22 was completely within the noise floor.

I reran it using client-side counting instead, and both groups were 0.

If you’re measuring your tiny signal in a place that already has huge background noise, the result is unreliable. And sometimes the noise gives you a number that looks very convincing.

6. I could reproduce the symptom, but not the production cause

Digging further into the source, I found that the routing strategy is decided before client initialization completes: if the client doesn’t yet have a default_node, the command is treated as a “no-key command”, doesn’t consult the routing table, and is sent to a randomly chosen node. And the first line of aclose() is to set default_node to None.

So I artificially forced default_node to stay None, ran at concurrency 300, and counted on the client side:

normal                      0.00 / 0.00 / 0.00
default_node = None         0.91 / 0.89 / 0.89     ← persistent, no decay

This time I really did reproduce it, and it was persistent. The mechanism itself was valid.

The problem is that this only proves that “if you break X, you get this symptom” — it does not prove that X was actually broken in production. So I checked the startup logs from the production containers:

22:39:05 | app.core.database_manager:322
  Redis Cluster connection initialized - host=clustercfg.xxx...

Initialization had succeeded, and default_node is assigned at exactly that step. So production did have a valid default_node; the state I had constructed did not exist online.

7. One INFO line in the startup logs

But reading the startup logs wasn’t wasted effort. Right before that initialized line was this:

22:39:05 | app.core.database_manager:383
  Redis Cluster address_remap enabled for local development

All 8 containers logged this line.

address_remap is a redis-py parameter. Under normal circumstances, the cluster client first asks for CLUSTER SLOTS, the cluster tells it which slots belong to which nodes and what those node addresses are, and then the client sends commands according to that table. address_remap lets you rewrite the node address after receiving it. The implementation in the recommendation service was:

# Address remapping: map cluster-returned private addresses to local port-forwarded addresses (for local development only)
def address_remap(address):
    return (host, port)      # host = clustercfg

No matter which node address the cluster returned, it rewrote all of them to the configured clustercfg address. In essence, it hardcoded all node addresses to the same one.

Its intended use was local development: a laptop connects to the cluster inside the VPC through SSM port forwarding, the cluster returns private IPs, the laptop can’t reach them, so everything has to be rewritten back to the tunnel address. The comment said “for local development only”, and the code default was false. All of that was fine.

So how did it get enabled in production? There was no REDIS_USER_ADDRESS_REMAP in the task definition environment variables. Then I checked the repo:

.env (committed in git):  REDIS_USER_ADDRESS_REMAP=true
no .dockerignore in repo
Dockerfile:               COPY . .           ← .env gets baked into the image
app/main.py:16            load_dotenv()      ← fills in values from .env for variables that are not set

load_dotenv() does not override existing environment variables, so the ones explicitly set in ECS were all correct. But ECS did not set REDIS_USER_ADDRESS_REMAP, so the local-development true from .env took effect.

Once enabled, when the cluster answered “slots 0-8191 belong to node A, 8192-16383 belong to node B”, remap rewrote both addresses to clustercfg. The routing table became:

normal:  slot 0-8191 → node A,         slot 8192-16383 → node B
actual:  slot 0-8191 → clustercfg,     slot 8192-16383 → clustercfg

The slot calculation was correct, but both results pointed to the same name. When a real connection was created, clustercfg DNS round-robined between the two nodes, so the effect was equivalent to randomly picking a shard — half the requests would go to the wrong one.

This time I started with production evidence and then reproduced it:

address_remap off    1500 commands   MOVED = 0        routing table = [node A, node B]
address_remap on     1500 commands   MOVED = 1459     routing table = [clustercfg]

The two real nodes in the routing table had been collapsed into one.

8. Why nothing happened for seven months, and why the API service was fine

I checked with git log -S, and both the address_remap code and the true in .env came in through the same commit on 2026-01-28. The incident happened seven months later.

Nothing happened for seven months because before the scale-out there was only one shard. clustercfg only resolved to that single node, so “rewrite all addresses to clustercfg” was effectively the same as not rewriting anything. On 09-02, when the second shard was added, this config changed from harmless into random routing.

Then it got masked by a more obvious problem — in the previous post, the redis-py 5.0.0 client hang caused all writes to fail, which was much more visible than MOVED. Only after upgrading to 8.x on 09-03 and fixing the hang did MOVED become the remaining curve.

Why was the API service fine? Its initialization code makes it obvious:

if _is_cluster_host(host):              # whether it starts with clustercfg
    _client = RedisCluster(**common)        # prod/stage: cluster client
else:
    _client = aioredis.Redis(**common)      # dev: regular single-node client

In the API service, the dev environment connects to a single-node instance with cluster mode disabled, and the client treats it as plain Redis. It doesn’t do topology discovery, so the “can’t reach private node addresses” problem doesn’t exist there. It never needed the remap config in the first place. The recommendation service didn’t have this branch, and used the cluster client locally too, so it needed remap to deal with private addresses.

One easy point of confusion is worth clarifying here. In the AWS console, all ElastiCache instances are called clusters, but Cluster mode: Enabled and Disabled are two different things: Disabled is a primary-replica replication group, while Enabled is the actual Redis Cluster protocol. A colleague mentioned that “other services have always connected to cluster through SSM tunnel just fine” — those instances were all Disabled.

9. The fix

The fix was the most conservative possible: leave .env untouched, and explicitly add REDIS_USER_ADDRESS_REMAP=false to the prod and stage task definitions. ECS environment variables have higher priority than .env, so this was a four-line change.

I didn’t remove .env because I checked and found that 14 other variables from it were currently taking effect in production. Removing it could have side effects I couldn’t confidently predict; that needed to be handled separately.

After deployment:

	Before Fix	After Fix
Read command MOVED ratio	35–45%	0
New connections per second	~880	0.1
Provider crashes	11/minute	0

(Image placeholder: error curves before and after the fix, dropping to 0 after deployment)

One note on the last line. The direct cause of the provider crashes was a bug in redis-py on the reconnect path (#4028, fix already merged but not yet released): connections were being repeatedly disconnected and rebuilt, and during the reconnect handshake, a concurrent disconnect could race with it and throw AttributeError. It wasn’t an independent issue. Once connections stopped being rebuilt over and over, that race condition no longer had a chance to trigger, so it also dropped to zero after the fix.

10. Retrospective

1. Read the logs first, then run experiments

The most effective step in the whole investigation was reading the startup logs from the prod containers. Once the line address_remap enabled showed up, the problem was basically settled. Before that, I had tried more than a dozen combinations on the bastion host, and every one of them was just guessing.

The logs had actually been telling me the answer the whole time. I just hadn’t looked first.

2. Being able to produce the symptom does not mean you’ve found the cause

This time, two different hypotheses could both stably reproduce the same MOVED rate as production on the bastion host, and neither one was what was actually happening online. The difference is the direction of evidence: did you first see evidence in production and then reproduce it, or did you first construct a state and then try to match the symptom? There can be many different paths to the same symptom.

3. In a noisy environment, first ask whether the signal is even separable

That 0.22 “reproduction” was just noise in the server-side counter. Switching to client-side counting made it 0. In the future, when running experiments in a live-traffic environment, the first question should be: how big is the signal, how big is the noise, and can I separate them? If not, measure somewhere else.

4. How do you keep local config from leaking into production

A .env committed into git, COPY . . baking it into the image, and load_dotenv filling in missing values — each of those is common on its own. Together, they mean: any local-only switch will go to production unless ECS explicitly sets it. This time it was address_remap; there are still 14 other variables in .env in the same state.

The practical fixes are straightforward: exclude .env with .dockerignore; explicitly list switches in production task definitions instead of relying on defaults; and add environment guards in code for “local only” switches.

Summary

The root cause was actually very simple: a local-development config hardcoded all node addresses to the same one, and it got carried into production through .env. With only one shard, it had no effect. After scaling out to two shards, it turned into random routing, and half the reads went to the wrong node. redis-py rebuilds the entire client every 5 MOVEDs, so connections couldn’t live for even 1 second, and the rebuild process then triggered another race condition, causing user profile data to be dropped too.

Four lines of config fixed everything.

But between seeing those tens of millions of MOVEDs and changing those four lines, there was a day and a half, seven or eight hypotheses, and a lot of dead ends. Every hypothesis looked internally consistent at the time, and each could produce some kind of symptom on the bastion host, but none of them matched production once checked against real evidence. The step that finally settled it was simply reading through the container startup logs.

That’s probably the most practical lesson from this incident: when debugging a production issue, finish looking at what production is already telling you before you start running experiments.

Adding Redis Sharding Took Production Writes Down for Over an Hour

Thu, 03 Sep 2026 00:00:00 +0800

Introduction

This started because the Redis instance used by our recommendation system ran out of memory again. used_memory hit 79.34G / 79.36G (99.98%), and Evictions also started climbing, which meant it had already begun throwing data out.

We first tried cleaning things up. The biggest chunk in this DB was exposure history (device_imp:* and the like, which records what content each device has seen for recommendation deduplication), taking up more than half the memory, so we started there and deleted keys that hadn’t been accessed for over 30 days based on LRU idle time.

The result was pretty underwhelming. The reason was simple: keys untouched for over 30 days mostly belonged to churned users, and those users barely used our product anyway, so their exposure history was tiny to begin with. After all that work, we deleted 880k keys and only freed about 5 GB of memory. Basically a lot of effort for almost nothing.

So my teammate proposed two action items: shorten the TTL, and also just add another shard.

Adding a shard in ElastiCache is an online operation. AWS advertises it as not affecting traffic, and we were also connecting through the cluster-mode configuration endpoint (clustercfg.xxx), so in theory the client should automatically detect topology changes.

And that “in theory” was exactly what caused our online exposure writes to break for over an hour. So this post is about what happened at the application layer after adding the shard, and whether cluster move behaved normally. Even if the process is supposed to be transparent, production still deserves respect. Sure enough, things broke right after the scale-out.

1. Errors started 12 seconds after the shard was added

First, here’s the ElastiCache event timeline (all times are UTC):

11:29  Scaling out replication group from 1 node groups to 2 node groups
15:39  Modified replication group to add 1 new node groups - 0002
17:18  Migrating slots from node groups 0001 to 0002 to rebalance slots   ← slot migration starts
21:24  Moved a total of 8192 slots out of 8192 slots from shard 0001 to shard 0002

The first application-side error appeared at 05:17:30, exactly 12 seconds after slot migration began.

And during the 12 minutes before that—from the scale-out starting at 05:11 to the new shard being ready at 05:15—not a single error showed up. Adding nodes and creating the shard were both completely fine. It only blew up the moment slots actually started moving.

The error looked like this:

ERROR | app.service:215 - info_collect failed,
exception_type=RedisClusterException,
exception=Redis Cluster cannot be connected. Please provide at least one reachable node: None

The Datadog graph made it especially obvious: it had been flat at 0, then suddenly shot straight up:

The `step.exception` metric had been at 0 the whole time, then jumped straight to 2k–3k / 2min once slot migration started, and stayed there

At that rate, it meant 17–25 requests per second were failing to write exposure records.

Quick side note: info_collect is one step in our recommendation pipeline. It does three things, and the order matters:

ctx.response.items = rsp_items        # 1. First assemble the recommendation results to return to the user
self.stats(ctx)                       # 2. Emit metrics
await PosterMgr.run_all(ctx, ...)     # 3. Finally write exposure records to Redis  ← this is where it blew up

The response is already assembled in step 1, so step 3 failing doesn’t affect the user getting recommendations. So throughout this entire incident, the API kept returning 200s, with no timeouts, no 5xxs, and no user complaints at all. It was completely silent. If I hadn’t happened to be looking at this Redis key distribution at the time, it probably would have stayed broken even longer.

2. I thought a restart would fix it. It didn’t.

Seeing “cannot be connected”, my first reaction was very natural: when the service started, there was only one shard. Now there were two, and maybe it didn’t know that. The old process was holding stale topology info, so restarting it to fetch topology again should fix it, right?

So I restarted it.

After the restart, it was still failing.

That made things more interesting. And even more confusing: the startup logs clearly showed initialization succeeding:

06:01:41 | app.core.database_manager:299 - Redis Cluster connection initialized -
          host=clustercfg.xxx-redis.xxx.cache.amazonaws.com, port=6379,
          max_connections=300, use_tls=True

There wasn’t a single initialization failed log. In other words, the process connected just fine at startup, then somehow broke while running.

Later I compared timestamps: initialization succeeded at 06:02, and errors started again at 06:09. So it relapsed after about 7 minutes. Restarting only bought us a few more minutes of life.

At that point, the hypothesis that “the service didn’t detect the new node” no longer held up. The new process clearly did detect it. It just forgot again after running for a while.

3. The `None` at the end of the error was the real clue

Later I pulled the full traceback, and only then noticed an important detail:

File "/usr/local/lib/python3.11/site-packages/redis/asyncio/cluster.py", line 685, in execute_command
    await self.initialize()
File "/usr/local/lib/python3.11/site-packages/redis/asyncio/cluster.py", line 392, in initialize
    await self.nodes_manager.initialize()
File "/usr/local/lib/python3.11/site-packages/redis/asyncio/cluster.py", line 1300, in initialize
    raise RedisClusterException(
redis.exceptions.RedisClusterException: Redis Cluster cannot be connected.
Please provide at least one reachable node: None

At first I thought this error message wasn’t very informative. “Please provide at least one reachable node” sounded like a plain network issue. But the network was clearly fine—I could connect to the cluster manually from the bastion host with no problem.

The key was that final None. I checked the redis-py source, and this is how that exception gets built:

exception = None
for startup_node in self.startup_nodes.values():
    try:
        cluster_slots = await startup_node.execute_command("CLUSTER SLOTS")
        startup_nodes_reachable = True
    except Exception as e:
        exception = e          # ← if connection fails, the error is stored here
        continue

if not startup_nodes_reachable:
    raise RedisClusterException(
        f"...Please provide at least one reachable node: {str(exception)}"
    )

If this were really a connectivity issue, exception should have contained a ConnectionError or TimeoutError. But it was None, which means the for loop never ran even once—in other words, self.startup_nodes was empty.

It wasn’t that the client couldn’t connect to any node. It was that the client no longer had any nodes left to connect to.

4. How the client lost all its nodes

Following that lead, I kept digging through redis-py and found this in the exception handling inside execute_command (we were using redis==5.0.0 in production):

except (ConnectionError, TimeoutError):
    # Connection retries are being handled in the node's Retry object.
    # Remove the failed node from the startup nodes before we try
    # to reinitialize the cluster
    self.nodes_manager.startup_nodes.pop(target_node.name, None)   # ← this is the line
    await self.close()
    raise

As soon as a node times out, it gets removed from startup_nodes. The design assumption here is: “there are still other nodes available for rediscovering topology.”

But there was a second trap too. I verified this on the bastion host:

When constructing the client: startup_nodes = [clustercfg.xxx...]
After first initialization:   startup_nodes = [0001-001, 0002-001]   ← replaced by discovered real nodes

The clustercfg configuration endpoint you provide gets discarded after the first successful initialization, and replaced with the concrete nodes discovered at that time.

Those two traps together were what made this fatal:

Before scale-out: single shard → after first initialization, the list only contains {0001-001}
                               (the clustercfg endpoint has already been overwritten and lost)
    ↓
05:17:18 slot migration starts, connection jitter happens, ConnectionError/TimeoutError is raised
    ↓
startup_nodes.pop("0001-001")   → the list becomes empty
    ↓
await self.close()              → marks the client as needing reinitialization
    ↓
Every command afterward → initialize() → iterates over an empty list → exception stays None
                       → raise "...reachable node: None"
    ↓
Permanently wedged; it never recovers unless the process restarts

So this is especially easy to hit when scaling from a single shard, because the list only has one node. One pop, and it’s empty. If the cluster had already had multiple shards, deleting one node would still leave others, and it probably could have recovered on its own. We just happened to be in the worst possible scenario.

One thing that confused me at the time was that there were no timeout errors anywhere in the logs. Later it made sense: that ConnectionError was swallowed inside redis-py’s internal retry loop. After the pop, it does raise, the outer retry logic catches it and calls initialize() again, and this time it hits the empty node list, so what finally bubbles up is RedisClusterException. That’s all the application sees. No timeout in the logs does not mean no timeout actually happened.

5. This isn’t new—people had already hit it on GitHub

I searched around and found an issue in the redis-py repo describing exactly the same problem:

RedisCluster becomes unrecoverable if all nodes timeout · Issue #3221

The title was literally our symptom. It explicitly points to that startup_nodes.pop line as the root cause, and also mentions that it’s especially bad in single-node cluster setups, which matched our case perfectly.

There was another related issue too:

async redis cluster should use initial startup nodes during reinitialization in case of failover · Issue #2472

This one describes the second trap: the async version overwrites the startup nodes you originally configured during first initialization.

Put those two issues together, and you get the full picture of our incident. The frustrating part is that #3221 is marked closed as not planned + stale. Upstream basically didn’t care and let it auto-close.

6. Why another service was unaffected: the version was three major releases newer

There was one thing during the investigation that kept bothering me: another backend service of ours was connected to the same cluster, went through the exact same scale-out, and had zero issues.

At first I thought it was due to different initialization styles:

# The other service: pass host/port directly
RedisCluster(host=host, port=port, ssl=..., decode_responses=True, ...)

# Recommendation service: pass startup_nodes
startup_nodes = [ClusterNode(host=host, port=port)]
RedisCluster(startup_nodes=startup_nodes, ssl=..., ...)

I ran a comparison test using the same redis-py version, and it turned out both styles behaved exactly the same: both got overwritten, both could get stuck. Inside redis-py, host=/port= just gets converted into a ClusterNode and stuffed into startup_nodes anyway, so they end up on the same code path.

That left only one variable: the version.

Other service         : redis[hiredis]~=8.0.0      → uv.lock pins 8.0.0
Recommendation service: redis==5.0.0               → confirmed by the traceback path

That’s a full three major versions apart. I checked the same code path in 8.0.0, and that pop line is gone:

# 5.0.0 —— directly removes the node from the list
self.nodes_manager.startup_nodes.pop(target_node.name, None)

# 8.0.0 —— no longer removes it; just moves the failed node to the end and retries later
target_node.update_active_connections_for_reconnect()
await target_node.disconnect_free_connections()
self.nodes_manager.move_node_to_end_of_cached_nodes(target_node.name)
e.last_failed_node_name = target_node.name

The list never becomes empty, so reachable node: None can never happen.

Same cluster, same slot migration, same scale-out, two services: one got wedged, one didn’t. The only variable was the client version. Honestly, the variable control was cleaner than most lab experiments.

A quick side note on something that’s easy to mix up: the client version number does not map 1:1 to the server engine version. The 8 in redis-py 8.x has no direct relationship to the 8 in Valkey 8.2. They don’t need to match, and things still work fine. Our server was Valkey 8.2.0 (INFO still reports redis_version as 7.2.4, because Valkey intentionally pretends to be old Redis for compatibility), while our clients were 5.0.0 and 8.0.0, and both could connect and read/write normally.

But “it works” doesn’t mean “it’s fine to leave it like that forever.” If the server engine version moves forward, the client should ideally move forward too, and not lag too far behind. I think that’s the real takeaway here. There are two reasons:

First, new engine features are simply unavailable to old clients. RESP3 support, newer commands, client-side caching—those all require client support too. Upgrading only the server is basically wasting the upgrade.

Second, old clients accumulate bugs that were fixed long ago, but you still haven’t benefited from those fixes. This incident was exactly that second category. Version 5.0.0 was from August 2023, and we missed every fix across three major releases, including the removal of that pop line.

So the reason to upgrade needs to be framed correctly: it’s not “the client version must match the server version,” it’s “the client shouldn’t stay frozen forever.” That distinction matters. Otherwise next time someone might draw a ridiculous conclusion like “then should we downgrade the server instead?” (No. The bug is in the client. It has nothing to do with the server version.)

7. Temporary mitigation: add a watchdog

At the time, we understood the root cause, but upgrading a client library across three major versions in the middle of the night felt too risky, so we first added a watchdog as a safety net.

The idea was simple: if the client can get itself into a permanently unrecoverable state, then something outside it should monitor it and rebuild it when that happens:

async def _watchdog(self):
    while True:
        await asyncio.sleep(self.WATCHDOG_INTERVAL_SECONDS)   # 10 seconds
        ping_task = asyncio.ensure_future(self.client.ping())
        done, _ = await asyncio.wait({ping_task}, timeout=self.WATCHDOG_PING_TIMEOUT_SECONDS)
        if ping_task not in done:
            ping_task.cancel()      # treat slow ping as a transient issue; don't rebuild
            continue
        error = ping_task.exception()
        if error is None or not isinstance(error, RedisClusterException):
            # Only RedisClusterException means the client entered a non-self-healing state;
            # transient errors like timeouts can recover on their own, and rebuilding would just add noise
            continue
        # rebuild using the original configuration endpoint
        new_client = await self._build_client()
        old_client, self.client = self.client, new_client
        await old_client.close()

There were two details I paid special attention to:

Rebuilding must use the original configuration endpoint (clustercfg), not whatever nodes the client currently has in memory, because its current node list is already corrupted.
Only rebuild on RedisClusterException. My first instinct was “rebuild on any error,” but that would be bad—if the network hiccups, rebuilding can easily create a rebuild storm, and in those cases the client can recover by itself anyway.

We set the detection interval to 10 seconds, which means in the worst case we’d lose 10 seconds of exposure data. Compared to “permanently wedged until someone manually restarts it,” that was acceptable.

During the day, we also cleaned things up on the Redis side and ran verification. Only then was the issue truly under control. The before/after comparison was very clear (20-second window):

	During incident	After recovery
MOVED on shard 0001	+5,567	+1
ZADD rejected on shard 0001	+5,560	+0
ZADD succeeded on shard 0002	+0	+5,656
Total commands on shard 0002	+28	+16,902

The new shard finally started receiving traffic, and writes were now being split normally across the two halves of the slot range.

After the watchdog was deployed, `step.exception` dropped back to a normal level

There was also a small side story here. The curve dropped, but not all the way to zero—it was still around 14%. After checking, we found one old-version instance hadn’t been replaced. Out of 7 instances, 1 was still old, and 1/7 = 14.3%, which matched the residual error ratio exactly. Once we stopped that instance, everything was clean.

8. Postmortem: a few things more worth remembering than the bug itself

1. The most troublesome failures aren’t the ones that error—they’re the ones that don’t

Throughout this incident, the API kept returning 200s. No timeouts, no 5xxs, no user complaints. Exposure writes were silently lost for over an hour, and luckily I happened to notice.

The root cause was this code:

try:
    await step_ins.do_process(ctx)
except Exception as e:
    logger.error("{} failed, ...", step_name, ...)
    datadog_agent.increment("step.exception", tags=["step:" + step_name])
    continue      # ← swallow it and continue to the next step

This except + continue pattern isn’t wrong by itself. If exposure writing fails, it really shouldn’t take down the whole recommendation API. The problem is that this degradation path had no alerting attached to it. The metric had been emitted the whole time, but nobody had set a threshold on it. A failure that could swallow one-third of all writes just sat there in metrics for over an hour with nobody looking. That’s the real hole we need to patch.

2. “Try restarting it” can be misleading

After the restart, the error count did go down for a while, which makes it very easy to think, “looks fixed, let’s keep watching.” In reality, the new process just started from a clean state and then fell into the exact same trap 7 minutes later. To tell whether something is really fixed, you can’t just look at whether errors dropped—you need to look at how far they dropped, whether they hit zero, and whether they bounce back.

3. When a cloud vendor says “online scale-out doesn’t affect traffic,” they mean the server side

And to be fair, the server side really did its job. Throughout the whole process, cluster_state: ok, all 16384 slots remained assigned, and no data was lost. But whether the client can keep up with topology changes is a separate matter, and that’s your responsibility.

Next time we do this kind of topology change, the runbook should include one extra step: after the change, check INFO commandstats on every shard and confirm the new shard is actually receiving business traffic. This time, the new shard held half the slots and 4.33 million keys, yet its business command count was 0. That signal was already pretty obvious—we just didn’t think to look until after the fact.

Summary

My biggest takeaway from this incident is that every link in the failure chain looked “reasonable” when viewed in isolation.

The cloud vendor’s online scale-out worked fine; the server stayed healthy the whole time. We used a cluster client and connected through the configuration endpoint, which was also correct. Exceptions were caught and degraded gracefully, preventing the whole API from going down, which even sounds like a good practice. And yet all of those individually reasonable pieces, combined with one line in the client library that says “remove the node if it times out,” turned into a completely silent production incident that lasted over an hour.

Another takeaway was about debugging method. Along the way I disproved four of my own hypotheses: “stale topology caused retries to exhaust,” “redis 5.0.0 is fundamentally broken for cluster,” “reinitialization clears the node list,” and “close clears the node list.” Every one of them sounded plausible at the time, and every one was disproven by experiments. In the end, the thing that really nailed the problem was that lonely little None at the end of the error message.

But after going in such a big circle, the final conclusion was actually pretty simple: this was a version problem. That pop line in 5.0.0 is a real bug. It’s already gone in 8.0.0. Same cluster, same scale-out, and the service running 8.0.0 was unaffected.

So the most practical lesson is still the same one from earlier: if the server engine version moves forward, don’t let the client lag too far behind. Missing out on new features is one thing. What’s really dangerous is missing fixes you should have gotten for free just by upgrading. Most of the time you won’t notice the difference when everything is calm. But once you do topology changes like scale-out, scale-in, or failover, that technical debt gets collected all at once. We let ours sit untouched across three major versions, and this time we paid the interest too.

A Synchronous Blocking Call Froze the Entire Event Loop

Sat, 22 Aug 2026 00:00:00 +0800

Introduction

There have been quite a few issues in the service lately, so I’ve been watching the dashboards much more closely than usual. One day, while looking through the memory metrics, I noticed a very strange segment: Min, Average, and even Max—which normally stayed steadily above 70%—all dropped together. All three lines fell all the way down to somewhere near the cold-start baseline, then spent more than half an hour slowly climbing back to normal.

I checked the event history in the container orchestration platform, and during that period, almost all running containers for this service were marked as “unhealthy” and removed/rebuilt in bulk. This wasn’t one or two containers flapping a bit—it was a real batch-scale traffic removal event, which is why even Max got dragged down instead of only Min dropping.

(Image placeholder: memory usage Min/Max/Average curves, with all three lines dropping together near the cold-start baseline, then slowly climbing back to normal)

I checked the deployment history too. During that time, nobody deployed anything, and there wasn’t a single release. The containers were not restarted as part of a “normal new version rollout.” If you don’t rule that out completely, nothing you find afterward really stands—because a batch restart could very well have just been caused by an ordinary deployment, and there’d be no need to look elsewhere. Only after confirming this was not deployment-triggered did it make sense to keep digging.

The most confusing part was: the database looked perfectly normal, and the application itself had no OOMs and no worker heartbeat timeout records. Nothing seemed broken, yet the service was still being judged unhealthy and removed in batches.

In the end, the culprit turned out to be a very easy-to-miss piece of code: a third-party SDK whose interface was written as async def, but whose actual internal call was pure synchronous blocking. That one call froze the entire service’s event loop for several seconds, and like knocking over the first domino, it cascaded into the database connection pool, health checks, and eventually large-scale traffic removal and rebuilds. I’m writing down the full investigation process and the underlying mechanics here.

1. First, rule out the two most common suspects

The service uses gunicorn + uvicorn workers. When something like this happens, the usual first checks are these two:

The system OOM Killer force-killed the process: if the kernel kills a worker, the gunicorn master process logs a very specific message when it notices the worker disappeared (something like Worker was sent SIGKILL! Perhaps out of memory?).
The worker itself missed heartbeats and got declared dead: uvicorn workers periodically “check in” with the gunicorn master process. If the master doesn’t receive heartbeats for a while, it assumes the worker is hung and force-kills/restarts it. The logs would contain WORKER TIMEOUT.

I checked how many times those two keywords appeared around the incident window—not once. The two most common explanations for “why was the container replaced” both didn’t match.

2. Change the angle: who removed the containers?

If the application logs weren’t getting me anywhere, then I needed a different angle—look at the service events maintained by the container orchestration platform itself (this information is not in the application logs; it’s a separate operational event stream recorded by the orchestrator).

Once I pulled that up, things started to make sense. In those few minutes, the orchestration platform marked almost all running containers for this service as “unhealthy,” and the reason was identical across the board:

(task xxx) is unhealthy in (target-group ...)
due to (reason Request timed out).

In other words, the load balancer’s active health checks timed out. The process hadn’t actually died—it was just responding too slowly to the health check probe, slow enough to exceed the load balancer’s threshold, so it got removed and rebuilt. And it happened wave after wave: once a few containers were removed, the remaining ones had to absorb more traffic, got even slower, and then more containers were removed. It snowballed.

(Image placeholder: orchestration platform event stream screenshot, showing many tasks marked unhealthy and rebuilt due to health check timeouts in a short period)

At this point the nature of the issue was basically clear: the process itself most likely never died; it was just so slow to respond to health checks that an external system misjudged it as dead. That lines up perfectly with “no OOM, no worker timeout,” because those are two completely separate detection mechanisms.

3. Trace backward: what exactly was happening during those few seconds?

I looked through the business logs from the tens of seconds before this batch of containers got removed, and found a large number of database errors. The gist was: “all connections in the pool were occupied, waited in queue for 30 seconds and still didn’t get one, so gave up” (QueuePool limit ... connection timed out, the standard SQLAlchemy connection pool error).

By itself, that error isn’t unusual. Under high concurrency, a saturated connection pool is nothing new. But the strange part was that the timestamps of these errors were almost all squeezed into the same sub-second window, with dozens of them. Under normal circumstances, if the connection pool is exhausted and everyone waits and times out independently, the errors should be spread out—each request enters the queue at a different time, so their 30-second countdowns should expire at different moments. This kind of “all at once” pattern looked wrong. It also matched the earlier observation that “the database itself looked stable”: if the database really lacked capacity, the errors should have been more evenly distributed, not clustered.

(Image placeholder: timestamp distribution of connection pool timeout errors in logs, with dozens of entries tightly clustered into a very short window)

It looked much more like this: “these requests’ countdowns should actually have expired earlier, but the thing responsible for triggering those timeout callbacks stopped functioning normally for a while; then the moment it resumed, all the accumulated callbacks fired at once.” In other words: it wasn’t that the database was insufficient—it was that something froze the event loop that this whole timeout mechanism depended on.

4. Find the real culprit: a call that “looked async but was actually sync”

With the hypothesis that “the event loop got frozen,” I went back to check what this service was doing at the time, focusing specifically on places where code was written inside async def, but the actual internal call was synchronous blocking (this kind of code can fool the eye, but not the event loop—async/await does not automatically turn an ordinary synchronous function call into a non-blocking one).

I found one. The project had integrated the official SDK of a third-party AB testing / feature flag platform a long time ago. Judging from the commit history, this wrapper code was written pretty early, back when traffic was probably much lower than it is now, so this kind of synchronous blocking call likely went unnoticed for a long time. The codebase came from multiple contributors, and the overall quality was uneven. A pitfall like this—”looks async, actually sync”—wasn’t something anyone intentionally left behind; it was more like historical baggage that nobody paid much attention to, until recent traffic growth amplified it into a real avalanche. The wrapper looked roughly like this:

class ExperimentManager:
    _client = None

    @classmethod
    async def fetch_variants(cls, *, device_id, user_id) -> dict:
        """docstring 里写着"用线程避免阻塞事件循环"……"""
        if cls._client is None:
            return {}
        user = User(device_id=device_id, user_properties={...})
        # 这一行是纯同步调用，SDK 内部走的是同步 HTTP 客户端
        variants = cls._client.fetch_v2(user)
        return {k: v.value for k, v in (variants or {}).items()}

The docstring sounded great—”use a thread pool to avoid blocking the event loop”—but in the actual code there was no asyncio.to_thread, nor any kind of thread scheduling at all. cls._client.fetch_v2(user) was just a plain synchronous method call executed in place. The SDK itself was configured with “3-second timeout, retry once on failure, backoff 0.3–2 seconds before retry, then another 3 seconds for the retry,” so a single call could theoretically take several seconds.

I checked the SDK’s own failure logs. Under normal conditions they were almost zero, but during the incident minute, failures shot up into the hundreds. And the timing of that spike happened to land before the clustered database connection pool errors.

(Image placeholder: bar chart of this third-party SDK’s failure logs aggregated by minute, spiking to hundreds in one minute while adjacent minutes stay in single digits)

The timing across the three evidence chains lined up: this synchronous call blocked the event loop first → during that period, everything relying on event-loop scheduling (waiting for DB connections, responding to health checks) got delayed → the moment the event loop resumed, a batch of timeouts exploded all at once → health checks couldn’t be answered → the load balancer judged them timed out → the orchestration platform started removing containers.

5. One remaining question: why did the downstream storm last longer than the trigger itself?

When I put the two timelines side by side, one thing looked asymmetric: that SDK’s failure logs only exploded within a single minute, then disappeared completely after that; but the database connection pool timeout errors only really got started after that minute ended, and kept going for nearly ten minutes, eventually totaling more than 70,000 errors.

One thing needs to be made clear first: the database itself did not have a problem during this period—core metrics like connection count and CPU stayed stable the whole time. These errors were entirely from the application-side connection pool failing to get a slot and giving up on its own (QueuePool is maintained inside the SQLAlchemy application process itself; it is not the database’s physical connection count being exhausted). This had nothing to do with database capacity. It was an application-layer problem.

So the question becomes: if the trigger only caused trouble for one minute, why did the downstream application-layer errors last ten times longer?

Looking closely at the distribution of connection pool errors during that period, it wasn’t one single explosion and then done. There were 5 distinct peaks over roughly ten minutes, each separated by about 2 minutes. The overall shape was “small initial rise, then two or three waves reaching the highest point, then gradual decay.” It was rhythmic oscillation, not random noise. There were probably two mechanisms layered together behind it:

First layer: avalanche feedback caused by traffic removal. After Amplitude froze the event loop that one time, the first batch of containers got removed and rebuilt because they couldn’t answer health checks. The traffic they had been carrying shifted onto the remaining containers. Those remaining containers suddenly had to handle more requests, so competition for the connection pool naturally became more intense. Then more containers got judged unhealthy because of slower responses and were removed as well. Once this feedback loop starts spinning, it can sustain oscillation for a while even without Amplitude continuing to add fuel, until the number of removed/rebuilt nodes reaches some equilibrium. This also explains why the errors came in waves: each wave roughly corresponds to one cycle of “removal → impact → more removal,” matching the cadence of health checks plus the orchestrator’s node rebuild cycle.

Second layer: restarts themselves were creating new resource pressure. When a container gets removed and rebuilt, the first thing the new container has to do is rebuild the entire connection pool from scratch. Those connections don’t magically exist just because a number is written in the config—they have to establish real TCP connections one by one and go through database authentication handshakes. If a batch of containers restarts in a short period, those new containers all compete to establish a large number of new connections at nearly the same time. That “creating connections” action itself takes time and can queue up, which is effectively another round of contention piled on top of already stressed resources, further extending recovery time.

In other words, that one synchronous Amplitude call merely knocked over the first domino. After it stopped, the chain reaction already had its own inertia. The two mechanisms above had to burn themselves out before things could truly settle down. That’s what makes this kind of avalanche failure so nasty: the root cause may only act up for one minute, but cleaning up the aftermath can take ten times longer.

6. Break down the mechanics: why can one synchronous call drag so many things down together?

There are a few details here that I repeatedly verified during the investigation, and they’re worth calling out separately.

1. “Written inside an async function” does not mean “this line of code is asynchronous”

async/await only suspends a coroutine and yields control at an actual await expression. If you directly call an ordinary synchronous blocking function inside the function body (with no await, because it isn’t awaitable at all), Python will not automatically insert a suspension point just because the outer function is async def—it just executes normally, waits normally, and returns normally.

The key point is: the asyncio event loop runs on a single thread the whole time, and all coroutines take turns executing on that one thread. A real asynchronous wait gives control back to that thread so it can schedule other coroutines; a naked synchronous call, on the other hand, occupies that one and only thread in place until it returns. During that time, the thread can do nothing else, including scheduling other coroutines or accepting new connections.

2. If we rewrote the health check endpoint as synchronous, could that avoid frequent task removal?

Based on the analysis above, this is really a two-layer issue: first, the event loop really was blocked for those few seconds—that’s just a fact; there’s no way around it. Second, tasks were judged unhealthy because they couldn’t answer health checks, and that traffic removal in turn amplified the subsequent congestion—that’s the nearly ten-minute avalanche described in the previous section.

So the question needs to be phrased more precisely: if the health check endpoint were implemented synchronously, could it at least still respond during those few seconds when the event loop was blocked, thereby avoiding the task being judged unhealthy, avoiding traffic removal, and cutting off the avalanche chain at the source?

The answer is still no, for the same reason as before: regardless of whether an endpoint is ultimately implemented synchronously or asynchronously, the very first step when a request arrives always has to go through the event loop. The event loop is what listens on the port, accepts the new connection, parses the HTTP request, and only then can we even talk about “should this endpoint run directly, or be handed off to a thread pool?” If the event loop can’t get through that first step, it never even reaches the point of “hand it to the thread pool.” So even if the health check endpoint were rewritten as synchronous, during those few seconds when the event loop was truly frozen, it still wouldn’t be able to answer, would still be judged unhealthy, and would still be removed from traffic. This avalanche chain cannot be cut off at the level of “is the endpoint sync or async”—that path simply doesn’t work.

3. Why did the timeouts explode in a cluster?

Database connection pool queue timeouts are generally implemented underneath using the event loop’s timer mechanism (“invoke a callback after N seconds to check whether the resource became available”), so they also depend on the event loop continuing to run in order to fire on time. While the event loop is frozen, all the timers that should have expired one after another just sit there. The moment the event loop resumes, they all fire in a batch—that’s the cause of the “clustered explosion” pattern.

This is also a useful signal for judging whether “the event loop may have been frozen”: if a large number of the same kind of timeout errors are squeezed into one very short window instead of being evenly distributed, you can strongly suspect a scheduling-layer problem rather than actual resource pressure being that severe.

4. Process heartbeat ≠ response time of a single endpoint

Gunicorn decides whether a worker is “alive” based on whether that worker is sending heartbeats on time. That is a process-level liveness check, which is a completely different dimension from how long any specific request has been running. As long as the event loop itself is still scheduling normally, the heartbeat can continue on time. Even if a request runs for several minutes, as long as it is “politely awaiting” rather than “hogging the thread with a naked synchronous call,” the process heartbeat is unaffected, and gunicorn won’t kill the worker.

Conversely, “process heartbeat is normal” can never prove that “every endpoint is responding in time”. Those are two independent things. Relying on one mechanism to cover for the other is itself an architectural blind spot that’s easy to overlook.

7. Fix strategy

What actually needed fixing was that “fake async” call: just honestly move it into a thread pool:

variants = await asyncio.to_thread(cls._client.fetch_v2, user)

With this change, even if the SDK call really blocks for several seconds, it only occupies one thread slot in the thread pool. The event loop itself remains completely unaffected, and other coroutines—including health checks—continue to be scheduled normally.

While I was at it, I also revisited the relationship between several internal timeout values: the database connection pool queue timeout, the load balancer’s health check threshold, and gunicorn’s own worker heartbeat timeout. Originally they had all been chosen more or less by gut feel, and by coincidence they landed in the same order of magnitude. Once resource pressure actually happened, several mechanisms fired almost simultaneously and reinforced each other into “it really is dead,” which amplified the avalanche effect instead.

Afterward, I reworked the priority of these values according to the principle that the mechanism with the smaller blast radius should always fail first: local resource wait timeouts should be clearly shorter than process-level watchdog timeouts. That way, under resource pressure, what happens first is always “this one request fails gracefully,” rather than escalating to a much more violent mechanism like “kill the whole process and take a bunch of otherwise healthy requests down with it.”

Summary

The biggest takeaway from this investigation was finally separating two concepts that are easy to blur together: process liveness and single-request health. They are two independent mechanisms. One only cares whether “the event loop is still turning,” while the other cares “how long this specific request has been running.” Neither can substitute for the other.

The other, more straightforward lesson is: async def is only a syntactic shell; it does not automatically verify whether the calls inside are truly asynchronous. When integrating a third-party SDK, even if the docs or docstring confidently claim “this already runs in a thread and won’t block,” it’s still worth taking the time to inspect whether that is actually true underneath—especially for old wrapper code that may not have kept pace with the rest of the project’s async evolution. Those are exactly the kinds of places where this sort of “fake async” likes to hide. And once you really step on it, the cost is often not just “this one endpoint gets a bit slower,” but, like this time, dragging a whole pile of seemingly unrelated mechanisms down together along the event loop’s single scheduling axis.

An Emergency Production Incident: Redis Memory Exhaustion, Precise Cleanup, and an Architectural Postmortem

Fri, 21 Aug 2026 00:00:00 +0800

Introduction

Today I received a monitoring alert: memory usage on the primary Redis instance in production had spiked straight to 99.98%, and Evictions on the dashboard had started climbing continuously.

Once this metric fires, it’s actually pretty dangerous, because this instance is configured with the allkeys-lru eviction policy. In other words, once memory is completely full, Redis will start randomly evicting old keys using the LRU strategy so it can keep accepting new writes. But this Redis instance is also serving user sessions, API rate-limit counters, and some foundational business caches. If I just let it keep evicting keys indiscriminately like this, it would only be a matter of time before critical business data got taken out by mistake.

Since I had confirmed Redis was indeed full, the first step definitely wasn’t to panic-delete things at random. I needed to first understand the situation clearly: analyze memory usage distribution -> identify the biggest consumer -> design a safe cleanup plan -> eliminate the underlying risk.

So here’s a write-up of this investigation process, the practical script I used to precisely clean up 50% of the data based on hit patterns, and a few architectural issues that surfaced during the postmortem.

(Image placeholder: a CloudWatch / Datadog dashboard showing Redis memory hitting 100% and Evictions starting to increase)

1. No Guessing in Production: Analyze Memory Distribution Across the Entire Keyspace

When dealing with a production Redis that’s completely full, the number one taboo is deleting keys based on gut feeling. The number two taboo is taking the “easy” route and running KEYS * or FLUSHDB directly (KEYS * can block the single-threaded server hard, and FLUSHDB can take down all online traffic immediately).

I had to first figure out what was actually consuming the memory. So I wrote a lightweight Python script that uses SCAN cursors to iterate through the entire keyspace in batches, sampling and estimating key counts and memory usage by prefix.

At the time, the memory distribution across the whole database looked roughly like this (sensitive business prefixes have been masked):

Key Prefix (masked)	Key Count	Share by Count	Avg per Key	Estimated Memory Usage	Business Purpose
*`prod:search_emb:`**	585k	22.8%	14.5 KB	~8.07 GB (87.5%)	🔴 Search query vector cache
`prod:img_url:*`	1.063M	41.4%	537 B	~544 MB (5.9%)	Presigned image URL cache
`service_go:prod:*`	188k	7.3%	1.95 KB	~350 MB (3.8%)	Microservice acceleration cache
`prod:card_cache:*`	38k	1.5%	2.03 KB	~72.7 MB (0.8%)	Operations card cache
`prod:session:*`	428k	16.7%	136 B	~55.5 MB (0.6%)	User session state
`prod:qclass:*`	171k	6.7%	211 B	~34.4 MB (0.4%)	Classification tagging cache
`prod:user:` / `LIMITS:`	~30k	1.2%	100~400 B	~15 MB	User info, rate limiting, and other core business data

Once I saw this table, the case was basically solved on the spot: the entire Redis instance only had about 9.2 GB of usable memory, and prod:search_emb:* alone was consuming 8.07 GB (87.5%)! Sessions, rate limiting, and other core business data were actually taking up only a tiny fraction. They were simply getting squeezed out by this one giant memory hog.

Here’s the non-blocking script I used to analyze the full keyspace distribution:

import redis

r = redis.Redis(host="your-redis-host", port=6379, socket_timeout=10)

cursor = 0
stats = {}

while True:
    cursor, keys = r.scan(cursor=cursor, count=5000)
    for k in keys:
        k_str = k.decode("utf-8", errors="replace")
        parts = k_str.split(":")
        prefix = parts[0] + ":" + parts[1] if len(parts) >= 2 else parts[0]
        
        if prefix not in stats:
            stats[prefix] = {"count": 0, "sample_bytes": 0, "sample_cnt": 0}
        
        stats[prefix]["count"] += 1
        if stats[prefix]["sample_cnt"] < 50:
            m = r.memory_usage(k)
            if m:
                stats[prefix]["sample_bytes"] += m
                stats[prefix]["sample_cnt"] += 1
    if cursor == 0:
        break

total_keys = sum(s["count"] for s in stats.values())
print(f"Total keys: {total_keys}")
for p, s in sorted(stats.items(), key=lambda x: x[1]["count"] * (x[1]["sample_bytes"]/max(1, x[1]["sample_cnt"])), reverse=True):
    avg_m = s["sample_bytes"] / max(1, s["sample_cnt"])
    est_mb = (s["count"] * avg_m) / (1024 * 1024)
    print(f"{p:<30} | {s['count']:>8} | {avg_m:>8.1f}B | {est_mb:>10.2f} MB")

2. Check the Code: Can This Biggest Consumer Be Cleaned Up?

After identifying the prefix, I immediately went to the codebase to inspect the corresponding implementation.

It turned out this was used for semantic search: caching the 1024-dimensional embedding vectors generated by an LLM for user search queries.

The logic in the code followed a standard Read-Through pattern:

A user enters a search query, and the system first checks Redis to see whether the query’s embedding is already there;
If it’s a hit (Cache Hit), it directly uses the stored vector to query the vector database;
If it’s a miss (Cache Miss), it calls the LLM API to generate the 1024-dimensional vector, then writes it back to Redis with a 3-day (ex=3*86400) TTL.

That explained why there were 585k keys: given the search volume over the past few days, users were entering all kinds of different queries, and the 3-day accumulation had piled up into 8 GB of vector data.

Can it be cleaned up? And how?

From a business-logic perspective, this is purely an acceleration cache. Deleting it won’t cause any data inconsistency or business errors—because if the cache is missing, the code will automatically fall back to the LLM and regenerate it.

But can I just wipe all of it in one shot? No. If I deleted all 585k vectors at once while search traffic was still live, every request would suddenly become a Cache Miss. That would send all concurrent search traffic straight to the downstream LLM API, likely triggering rate limits or timeouts on the model service and causing an even bigger secondary incident.

So the best solution was: reduce memory usage without clearing everything, prioritize deleting the coldest data that hasn’t been accessed for a long time, and keep the hot cache intact.

3. Fine-Grained Cleanup: Precisely Remove the Coldest 50% by Hit Pattern (Idle Time)

So the next question was: how do I determine which vector cache entries in Redis are “cold data” that haven’t been hit in a long time?

The answer was Redis’s native OBJECT IDLETIME key. Under allkeys-lru, Redis maintains an LRU clock for each key. With OBJECT IDLETIME, I can precisely check how many seconds have passed since the key was last read or written (and this query only reads the clock—it does not alter the key’s access time).

I first used a Pipeline to collect the idle-time distribution across all 585k vector keys:

Idle Time (since last hit)	Key Count	Share	Decision
> 2 days (no access for over 48 hours)	149k	25.59%	🔴 Delete candidate (very cold data)
1 ~ 2 days (no access for 24~48 hours)	144k	24.72%	🔴 Delete candidate (cold data)
12 ~ 24 hours	118k	20.28%	🟢 Keep (warm data)
6 ~ 12 hours	69k	11.82%	🟢 Keep (active data)
1 ~ 6 hours	80k	13.73%	🟢 Keep (high-frequency hot data)
< 1 hour	22k	3.86%	🟢 Keep (core hot data)

Once the numbers came out, the picture was very clear: keys with idle time $\ge 24$ hours (queries nobody had searched for in a full day) accounted for exactly 50.3% (294k keys)!

Those 294k keys were very likely long-tail, low-frequency search queries. Keeping them around brought basically no value other than consuming memory. The remaining 49.7%, on the other hand, had all been hit within the past 24 hours and represented the actually popular queries.

Execute a Safe Asynchronous Cleanup

To guarantee zero impact on production traffic, I used SCAN cursors + batched Pipeline checks + UNLINK (which frees memory asynchronously in a background thread and does not block the main event loop), processing 1000 keys per batch with a 10ms sleep between batches:

import redis
import time

r = redis.Redis(host="your-redis-host", port=6379, socket_timeout=10)

# Threshold: cold data not hit for more than 24 hours
IDLE_THRESHOLD = 86400 

cursor = 0
total_scanned = 0
total_unlinked = 0
batch_to_delete = []

print(f"清理前内存: {r.info('memory').get('used_memory_human')}")

while True:
    cursor, keys = r.scan(cursor=cursor, match="prod:search_emb:*", count=2000)
    if keys:
        pipe = r.pipeline(transaction=False)
        for k in keys:
            pipe.object("idletime", k)
        idles = pipe.execute()

        for k, it in zip(keys, idles):
            total_scanned += 1
            if it is not None and it >= IDLE_THRESHOLD:
                batch_to_delete.append(k)

            # 每凑齐 1000 个调用一次异步 UNLINK
            if len(batch_to_delete) >= 1000:
                r.unlink(*batch_to_delete)
                total_unlinked += len(batch_to_delete)
                batch_to_delete = []
                time.sleep(0.01)  # 短暂停顿，避免主线程抖动

    if cursor == 0:
        break

if batch_to_delete:
    r.unlink(*batch_to_delete)
    total_unlinked += len(batch_to_delete)

time.sleep(2)
print(f"清理完成! 共扫描 {total_scanned} 个，异步清理冷 Key {total_unlinked} 个")
print(f"清理后内存: {r.info('memory').get('used_memory_human')}")

Cleanup Results

The script ran for about 20 seconds:

It scanned 585,152 keys and precisely deleted 294,350 cold keys (50.30%);
Redis memory dropped immediately from 9.19 GB to 5.22 GB, freeing up 4.0 GB of precious space;
Memory usage fell straight from 99.98% back down to 56.5%, Evictions on the monitoring dashboard dropped to zero, and all alerts affecting Sessions and other core business traffic were fully resolved;
The downstream LLM API saw no traffic spike at all, and the system got through the incident smoothly.

(Image placeholder: after cleanup, the monitoring dashboard shows memory usage back down to 56% and business traffic returning to a stable state)

4. Postmortem: What Architectural Problems Did This Expose?

Although the issue was resolved within a little over ten minutes, this memory saturation incident essentially exposed several hard flaws in the earlier architecture design:

1. The serialization format was extremely wasteful (string JSON vs binary pack)

When I checked the code, I found that this 1024-dimensional float vector was being written into Redis using json.dumps(vector) directly. A 1024-dimensional float array, once converted into a JSON string, stores every float with a bunch of decimal digits, plus commas and brackets, so each key ended up taking 14.5 KB ~ 21 KB!

But in reality, 1024 float32 values packed into binary using Python’s struct.pack('<1024f', *vector) only take a fixed 4096 bytes (4 KB). Even if I wrapped it in Base64 for compatibility, it would still only be about 5.5 KB. If binary packing had been used from the start, memory usage for the same number of keys would have dropped by 62% immediately, and the original 8 GB of data would have gone down to under 3 GB.

This Redis instance was currently being used as a giant “everything bucket”: it was serving both large-volume, disposable, recomputable read-through acceleration caches such as search vectors and presigned URLs, and also small-volume, low-latency, absolutely must-not-be-lost core business data such as Sessions, user info, and API rate limiting.

As soon as one acceleration-cache feature gets rolled out or traffic spikes, it can immediately consume the entire instance’s memory and cause core business keys to be evicted by LRU. This design is obviously unreasonable. Going forward, high-consumption caches for search/algorithms must be physically separated from the core business Redis.

3. The instance size itself was too small

The current primary instance was a cache.r4.large (12 GB total physical memory, with only 9.2 GB actually usable by Redis after reserving 25%). By comparison, Redis instances used by our recommendation system and other compute-heavy services are commonly sized at 64 GB. As search and AI features are being used more and more frequently, a 9.2 GB capacity ceiling is clearly no longer keeping up with business growth. Upgrading the instance size or migrating it is basically inevitable.

Summary

When troubleshooting incidents in high-risk production components, the rhythm is usually: use monitoring metrics to quickly classify the problem -> use read-only / low-overhead methods to locate the root cause -> weigh business risk and design the lowest-cost mitigation -> finally close the hole at the code and architecture level.

This time, cutting by LRU idle time let me quickly push memory usage back into the safe zone while preserving cache hit rate as much as possible. But more importantly, the postmortem helped firmly surface the real issues around serialization compression, physical instance isolation, and capacity planning.

How to Elegantly Connect a Local GUI Client to a Private Redis on AWS / GCP

Fri, 21 Aug 2026 00:00:00 +0800

Introduction

Recently, I was troubleshooting a Redis memory issue in production where usage was getting dangerously close to full. I wanted to dig into which keys were growing like crazy, whether there were any big keys, or data without a TTL set. In situations like this, the most intuitive approach is usually to fire up a local GUI tool—something like Another Redis Desktop Manager or RedisInsight—connect to Redis, and search or browse the data visually.

But if you’ve ever used managed Redis on AWS or GCP, you’ve probably run into the same wall: cloud providers simply do not let you expose Redis to the public internet.

Quick disclaimer first: in large companies with very strict compliance processes, connecting directly from your local machine to a production Redis instance might get you a lot of attention from the security team. But in many startups, small teams, or when you’re acting as the tech lead / ops person handling an urgent incident, being able to observe and diagnose production data quickly and intuitively is often far more practical than fighting through a pile of approval workflows.

So today I figured I’d write down why cloud providers absolutely refuse to give Redis a public endpoint, and how I moved away from the painful SSM command-line port forwarding workflow to a much cleaner setup using an SSH tunnel directly inside a GUI client.

1. Why can RDS be public, but Redis absolutely cannot?

When I first started using cloud services, this felt a little counterintuitive to me: on AWS or GCP, if you buy an RDS instance (PostgreSQL / MySQL) or Cloud SQL, you can usually just click a few buttons in the console, assign a Public IP, whitelist your own IP in the security group, and connect directly from your laptop.

But when it comes to ElastiCache or GCP Memorystore for Redis, the console doesn’t even offer a “public access” toggle. It’s hard-locked into a private subnet.

This really isn’t cloud providers being lazy. It’s mostly a consequence of Redis itself:

Historical baggage and weak brute-force resistance: Redis was originally designed with the assumption that it would run inside a fully trusted internal network. Early versions didn’t even have password authentication. Even though modern Redis supports Auth and ACLs, Redis can still handle hundreds of thousands of requests per second on a single instance, which makes password brute-forcing over the public internet extremely cheap for attackers.
Dangerous commands and script escape risks: Redis has a fairly broad privilege boundary. Commands like CONFIG SET, EVAL (Lua scripts), and module loading have all historically been involved in serious exploits where attackers used unauthorized access or weak passwords to escape the Lua sandbox, write files to the host machine, or even spawn a reverse shell.
Its single-threaded model is easy to block: Redis core request handling is single-threaded. If you expose it directly to the public internet, even a small amount of malicious traffic—or someone accidentally running an expensive KEYS *—can freeze the whole instance instantly and trigger a cascading failure across downstream services.

So cloud providers are very consistent on this point: no matter how you configure it, Redis instances must stay inside the VPC private network.

2. Why does AWS SSM port forwarding always feel awkward?

Since direct access isn’t possible, the traditional approach is to use an EC2 bastion host in the same VPC and rely on AWS SSM for local port forwarding:

aws ssm start-session \
  --target i-0123456789abcdef0 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["my-redis-prod.xxxxxx.ng.0001.use1.cache.amazonaws.com"],"portNumber":["6379"],"localPortNumber":["6379"]}'

Once this command is running, your local 127.0.0.1:6379 is effectively mapped to the Redis instance in the cloud, and your GUI client can just connect to localhost:6379.

If you only need to run an occasional local test from code, this is actually pretty convenient. But if you frequently inspect data in a management tool, a few annoying pain points show up quickly:

Switching between environments is extremely tedious: In real life you usually have dev, stage, and prod, and sometimes a dedicated Redis for recommendation systems or search. Every time you want to switch environments, you have to Ctrl+C the current session in the terminal, change the host parameter, and rerun the command. Or you end up opening a bunch of local ports like 6379, 6380, 6381, and after a while you can’t even remember which port maps to which environment.
Idle sessions disconnect all the time: SSM sessions have heartbeat and timeout behavior. If you step away from your computer for a bit without sending requests, or your laptop goes to sleep, the connection in the terminal quietly dies—often stuck at Waiting for connections.... Then you go back to the GUI tool, hit refresh, get a timeout, and have to restart everything from the terminal again.
You have to keep a terminal window open all the time: Every day you end up wasting a terminal tab just to keep SSM running, constantly worried you’ll close it by accident.

3. A better approach: use a local key to connect through the bastion host directly

In fact, most modern Redis desktop clients—like Another Redis Desktop Manager and RedisInsight—already support SSH Tunnel natively.

(Image placeholder: the SSH Tunnel configuration screen in a desktop client; can be replaced during review)

If you let the desktop client manage the SSH tunnel itself, save each Redis environment as a separate connection card, and just click whichever one you need, it can automatically bring up the tunnel and put it to sleep when idle. The experience becomes much smoother.

A lot of people get stuck here because they can’t find the old PEM key for the bastion host, or their access has already expired.

But if you already have permission to run AWS SSM commands locally, then you don’t need to ask anyone for a key again. You can simply generate a new public key on your own machine and use SSM to inject it into the bastion host in one shot.

1. Generate a dedicated SSH key pair locally

Run a single command in your local terminal to generate a standard RSA key pair (without a passphrase, so GUI tools can load it directly):

# Generate a key dedicated to the Redis tunnel
ssh-keygen -t rsa -b 2048 -m PEM -f ~/.ssh/my_redis_tunnel.pem -N "" -C "redis_tunnel_key"

2. Use AWS SSM to inject the public key into the bastion host

Find the EC2 instance ID of the bastion host in the same VPC that has a public IP, then send a CLI command that appends your public key to the bastion host’s authorized_keys:

PUB_KEY=$(cat ~/.ssh/my_redis_tunnel.pem.pub)

aws ssm send-command \
  --instance-ids "i-0123456789abcdef0" \
  --document-name "AWS-RunShellScript" \
  --parameters "commands=[
    \"mkdir -p /home/ubuntu/.ssh\",
    \"chmod 700 /home/ubuntu/.ssh\",
    \"echo '$PUB_KEY' >> /home/ubuntu/.ssh/authorized_keys\",
    \"chmod 600 /home/ubuntu/.ssh/authorized_keys\",
    \"chown -R ubuntu:ubuntu /home/ubuntu/.ssh\"
  ]"

The idea is simple: the AWS SSM Agent runs inside EC2 as a background daemon with root privileges. Once it receives instructions from the AWS API, it can write your public key directly into the allowlist, completely bypassing the classic deadlock of “you need to log into the server first in order to add your public key.”

3. Configure the connection in your Redis client

Now the bastion host already trusts your local private key, so open your Redis desktop client and create a new connection:

General connection settings:
- Host: enter the Redis private endpoint (for example, my-redis-prod.xxxxxx.ng.0001.use1.cache.amazonaws.com)
- Port: 6379
- Auth: leave blank if no password is configured
- TLS / Cluster: choose based on your actual setup (for a standard primary-replica deployment, this is usually left unchecked)
SSH Tunnel:
- Check ☑️ SSH
- Address: enter the bastion host’s public IP (for example, 54.xxx.xxx.xxx)
- Port: 22
- Username: ubuntu (or the default username for your system image)
- Private Key: browse and select the ~/.ssh/my_redis_tunnel.pem you just generated
- Password / Passphrase: leave blank

Click test connection, and it should work immediately.

Summary

Once this is set up, the whole development experience improves a lot.

You can save Prod-Redis, Stage-Redis, and Dev-Redis as separate connection profiles in your GUI client. All of them can share the same bastion host and private key, while each one points to its own internal Redis host. From then on, whenever you need to inspect a specific environment, just double-click and connect. If the connection drops, the client can usually reconnect automatically, and you can finally stop keeping a pile of SSM terminal sessions running in the background.

In day-to-day development and operations, a lot of infrastructure security policies—like private network isolation—are non-negotiable. But by making good use of a bastion host and the native capabilities of your client tools, you can absolutely stay within the security boundary while still maximizing your local troubleshooting efficiency.

Investigating a Slow Memory Leak: Creating a New OpenAI Client Every Time, with the Real Issue in the Event Loop

Wed, 19 Aug 2026 00:00:00 +0800

Introduction

Recently, I was investigating a slow memory leak in one of our services. After watching the memory curve in production, I noticed the process RSS was steadily creeping upward. It didn’t look like the stepwise growth you’d expect from increasing business data volume. Instead, it was that kind of “slow but constant” diagonal line that just keeps going no matter how long you wait. Given enough runtime, it was clearly going to hit the container memory limit.

This was actually pretty obvious in monitoring, but it hadn’t turned into a real incident yet—mainly because the business iterates quickly, and we deploy basically every day. Each deployment restarts the machines, and RSS gets reset back to baseline every time, so the leak never had enough time to accumulate all the way to the top before being dragged back to the starting point. But that doesn’t mean the problem wasn’t there. Extend the curve far enough and the conclusion is obvious: as long as the service runs continuously for long enough without restarting, sooner or later it will hit the container memory limit. At that point, either gunicorn will decide memory usage is too high and start killing workers, or some lower-level mechanism will kill it more forcefully. Frequent releases were only delaying the inevitable by accident.

(Image placeholder: memory leak curve, RSS keeps climbing over time without converging, eventually approaching the container memory limit during long-running execution)

After tracing through the code, I found the culprit was a piece of “asset enrichment” logic a teammate had written earlier: when a user uploads an asset, the backend calls an LLM to do tagging and classification, then computes an embedding and writes it into the search index. This logic itself isn’t fast, and we didn’t want it occupying the main thread’s event loop—after all, the main loop still needs to handle incoming requests normally, and nobody wants a single LLM call to stall the whole service. So the original idea was to throw this work into a dedicated background thread pool. That idea itself was fine. The problem was in how the async SDK was being called inside the thread pool. That’s where a subtle trap got buried.

This post is a record of how I investigated it, why this client couldn’t just be reused directly, and what solution we ended up shipping.

1. The original approach: start another event loop inside the thread pool

First, here’s roughly what the original code looked like:

def submit_background_job(coro_func, *args, **kwargs):
    # Throw it into the thread pool so the current request isn't blocked
    executor.submit(asyncio.run, coro_func(*args, **kwargs))

async def enrich_asset(asset_id: int):
    client = AsyncOpenAI(api_key=API_KEY, base_url=BASE_URL)
    resp = await client.chat.completions.create(...)
    ...

It’s not hard to guess why someone would write it this way: the main service is built on an async framework, but the enrichment logic needs to call an external LLM API, and we still wanted to keep using the async SDK style of programming (retries, timeouts, streaming, etc. are already built into the SDK, so there’s no need to wrap a separate sync version ourselves). So the implementation picked a very common combination: inside each thread from the thread pool, use asyncio.run() to spin up a separate event loop, and then await the async SDK inside that loop. This way it doesn’t occupy the main service’s event loop, while still letting us keep the async style. On the surface, it looks like the best of both worlds.

2. The cost: you have to create a brand-new client every time

What asyncio.run(coro) does is: create a new event loop, run the coroutine, then destroy that loop. In other words, in the code above, every call to enrich_asset goes through a full cycle of “create a brand-new loop -> use it once -> throw it away.”

The problem starts at this line: AsyncOpenAI(api_key=..., base_url=...). Under the hood, this client wraps an httpx.AsyncClient, and httpx.AsyncClient internally maintains a real connection pool plus SSL handshake context. Those things are bound to the event loop they were created on. Since every call runs on a brand-new loop, the client is naturally one-shot as well—it can’t really be reused after this request, because the next request will run on a completely different new loop. So the code chose the “seems easiest” option: just create a fresh one every time.

But as soon as AsyncOpenAI is instantiated, it really does allocate the underlying connection pool (even if no request is ever sent). If you create one every time and then leave it to GC after use, in theory that connection pool and its SSL context should be reclaimed together with that event loop. But in practice, cleanup wasn’t happening cleanly. At the time, I pulled a long-window container memory curve from Datadog—about 14 hours—and the fitted numbers looked like this:

Metric	Value
Overall memory growth rate over 14 hours	about 393 MB/h
Total net memory increase over 14 hours	about 5.35 GB
Peak memory per instance	about 13.4 GB (close to the 16 GB container limit)

I also ran a simple stress test script that repeatedly hit this enrichment path. The memory curve climbed almost in lockstep with the number of calls and never stopped. It had nothing to do with business data volume. That basically confirmed it: the leak point was this client that was “created fresh every time and never reused”, and at this growth rate, the container would eventually be eaten alive by this logic itself.

3. So why not just reuse the client directly?

Once you see “creating a new client every time,” the obvious question is: why not just cache the client and reuse a single global instance?

_client = AsyncOpenAI(api_key=API_KEY, base_url=BASE_URL)

async def enrich_asset(asset_id: int):
    resp = await _client.chat.completions.create(...)
    ...

The answer is: with the current execution model, it simply cannot be reused. The reason goes back to the previous section: the httpx.AsyncClient inside AsyncOpenAI is bound to the event loop that existed at the moment it was created. And in this service, background tasks are run via asyncio.run(), which means every call to enrich_asset runs on a brand-new event loop, different from the previous one. Even if you store the client in a global variable, it can only be created on the loop from the first call—and that loop is destroyed immediately after use. All later loops would then be trying to send requests through a client that is effectively “attached to a dead loop.” At the mechanism level, that’s simply wrong. Resources from different event loops can’t be shared across them like that.

In other words, the prerequisite for “reusing the client” is first having an event loop that itself can be reused. As long as every task still creates a fresh loop and throws it away, the client has nowhere stable to “live,” so caching it or not makes no real difference.

4. The fix: don’t touch business logic, bind the thread pool to long-lived event loops and reuse them

The constraint I set for this fix was: don’t modify the enrich_asset business logic itself. How it calls the LLM and how it processes the result should remain unchanged. The only thing to change is the outer execution layer—how this logic gets run. Since the client must be bound to a fixed event loop in order to be safely reused, the solution should be inverted: instead of creating a throwaway loop for every task, give each thread in the thread pool its own long-lived event loop, and run all tasks dispatched to that thread on the same loop.

The concrete implementation uses thread-local storage (threading.local) to keep a persistent asyncio.Runner per worker thread (asyncio.Runner was introduced in Python 3.11+ as a persistent event loop wrapper; functionally it’s equivalent to manually maintaining an event loop that isn’t destroyed):

import threading
import contextvars
from concurrent.futures import ThreadPoolExecutor

_state = threading.local()  # each thread has its own .runner

def _get_runner() -> "asyncio.Runner":
    runner = getattr(_state, "runner", None)
    if runner is None:
        runner = _state.runner = asyncio.Runner()
    return runner

def _run_on_worker_loop(ctx, coro_fn, args, kwargs):
    return _get_runner().run(coro_fn(*args, **kwargs), context=ctx)

executor = ThreadPoolExecutor(max_workers=4, thread_name_prefix="bg-worker")

def submit(coro_fn, *args, **kwargs):
    ctx = contextvars.copy_context()
    return executor.submit(_run_on_worker_loop, ctx, coro_fn, args, kwargs)

With this setup, every task running on the same thread uses the same event loop. Once that prerequisite is in place, client caching finally becomes valid:

_clients = {}  # (id(loop), api_key, base_url) -> client

def get_client(api_key: str, base_url: str):
    loop = asyncio.get_running_loop()
    key = (id(loop), api_key, base_url)
    client = _clients.get(key)
    if client is None:
        client = AsyncOpenAI(api_key=api_key, base_url=base_url)
        _clients[key] = client
    return client

Including id(loop) in the cache key is basically saying: “this client may only be used on the loop it was created on; different loops maintain their own separate instances.” Since each thread in the thread pool now has exactly one long-lived loop, the practical effect is that the whole process will only ever have as many clients as the thread pool size—for example, 4 threads means 4 clients—instead of growing without bound as the number of calls increases.

When the process exits or the thread pool shuts down, remember to close the client on that thread’s own loop (await client.close()). Don’t directly close another thread’s connection pool across threads, or you may run into some very weird errors.

That leads to the core conclusion of this fix: “Can an async client be reused?” is fundamentally the same question as “Can the event loop behind it be reused?” The two are tightly coupled. Client caching only becomes meaningful once the event loop itself is long-lived and reusable.

5. Results

After deploying the fix, I compared two runtime windows of similar length—both around 14 hours:

Metric dimension	Before fix	After fix	Change
Growth rate over 14 hours	about 393 MB/h	about 210 MB/h	down about 47%
Total net memory increase over 14 hours	about 5.35 GB	about 2.99 GB	net increase reduced by about 44%
Peak memory per instance (14h)	about 13.4 GB (close to 16 GB limit)	about 9.4 GB	about 6.5 GB of safety margin left

(Image placeholder: overlay comparison of memory curves before and after the fix over similar runtime windows, with a clearly reduced slope)

The growth rate was cut almost in half, and the peak moved completely out of the danger zone near the container limit. So the biggest part of the problem was effectively contained.

Summary

At the end of the day, this whole issue boils down to one sentence: “Can an async client be reused?” is really asking “Can the event loop behind it be reused?” The two are tightly bound together, and you can’t solve only the surface-level part.

A few takeaways worth writing down:

asyncio.run() creates a brand-new event loop every time. Wrapping each background task with it is a very common convenience pattern, but if the task uses resources that are bound to the event loop (connection pools, SSL contexts), this “one-shot loop” model makes those resources impossible to safely reuse. The result is that you have to recreate them every time, and the cost is connection-pool leakage.
On the flip side, simply turning the client into a global singleton doesn’t necessarily help. If the event loop itself is still one-shot, then the global singleton just gets permanently bound to the already-dead loop from the first call, and becomes unusable.
The correct order is to first make the event loop long-lived and reusable (for example, by binding a persistent loop to each thread in the thread pool), and only then cache resources that are bound to that loop. That’s the real fix.
Even after deploying the fix, I didn’t assume the story was over. The growth rate was cut in half, but the curve didn’t become perfectly flat. There’s still a small residual increase of around 200 MB per hour, which is probably due to a few other smaller issues in other modules and not the same root cause as this one. This wasn’t a magical “one fix cures all” patch. It just plugged the biggest hole. The remaining small tails still need to be shaved down one by one.
For this kind of “slow leak,” a stress test script plus the memory curve is the most direct qualitative tool. If the curve rises in sync with the number of calls, you can usually narrow the suspect range down pretty quickly to “what resource isn’t being properly reused or closed.”

One last note. Strictly speaking, the way this business logic is written isn’t really best practice. Throwing a slow, heavy LLM call directly into the web service’s own thread pool always comes with trade-offs. But the code had already been written this way and had been running in production for a long time, and there wasn’t a strong business reason to rewrite it. So for this fix, I deliberately set the principle to zero changes to business logic, and only worked on the layer of “how this logic gets executed,” keeping both the scope of change and the risk as small as possible. And in practice, the result did meet expectations.

Postscript

That said, if I were designing this from scratch, the better approach would be not to run this kind of heavy task inside the web container at all. Move it to offline machines, or process it with offline compute resources like Lambda. There are mainly two reasons:

Isolation: this kind of heavy task that consumes both CPU and IO will naturally interfere with the web service if they share the same process. And the web service itself has much stricter requirements for stability and low latency, so the cost of interference is much higher. It’s far more cost-effective to let a less latency-sensitive offline task absorb that risk than to make the web service shake along with it.
Cost: memory in web services is usually much more expensive than offline compute resources. This kind of heavy logic—loading model requests once, processing large objects—even if everything can eventually be reclaimed by GC, will still create memory spikes during execution. And if you’re even slightly careless (like this time), some of it may stay stuck in the heap forever and turn into a real leak. Putting this kind of high-memory operation on cheaper offline resources is simply the more economical architectural choice.

A Database Avalanche Incident Investigation

Tue, 18 Aug 2026 00:00:00 +0800

Preface

That day I was busy with something else when I suddenly got pulled into an incident chat. The only message was one line: every single service endpoint is timing out like crazy.

When you see a description like “timeouts across the board,” the first instinct is: is this that old issue happening again? We had investigated a similar avalanche before, and the root cause back then was a call that was “written inside an async function but was actually synchronously blocking,” freezing the event loop and then dragging down health checks, which caused the container orchestration platform to mark instances unhealthy, remove traffic, and restart them. (I wrote up the full investigation of that one in a separate post, so I won’t repeat it here.) I started digging with that assumption in mind, but this time it turned out to be a completely different pit from start to finish. Still, once I kept digging, it pulled out more problems than I expected, and the chain was much longer too.

This post starts with damage control, then moves on to the retrospective—because when something breaks, what everyone wants to know first is always “how is it now,” and the analysis can come later.

1. First, rule out the possibility of “the old problem happening again”

Based on last time’s experience, the first step as usual was to check the service’s own basic metrics: CPU usage, memory pressure. Everything looked normal, with no abnormal spikes at all.

Since what people were reporting was “all endpoints are timing out,” what mattered more was really the latency of the core business endpoints themselves. But since I suspected the old issue, the first thing I still did was take a look at the TP95 of the health check endpoint—if this was another event-loop-freeze situation, the health check would very likely be slowed down too, which makes it a very useful signal. This time, though, the health check TP95 was completely normal.

Basic service metrics normal + health check normal—once those two were confirmed, I could basically rule out a recurrence of the same old issue. Since the application process itself looked healthy, the problem was probably not in the application layer. Time to look toward the middleware layer.

2. Shift to middleware: database CPU was already maxed out

I immediately checked the database—and one look was enough to tell something was seriously wrong. CPU had already hit the ceiling:

Database CPU usage during the incident window. Usually calm and flat, but here it went straight to 100%

I quickly went to the AWS console hoping to see more detailed execution info, only to find that this database did not have Performance Insights enabled, so I couldn’t directly inspect historical Top SQL, wait event distribution, or other fine-grained data. Fortunately, the console still exposed some basic SQL latency stats. I glanced at the numbers—and the durations were basically all in the thousands of seconds, which honestly startled me. For a moment I even wondered whether I had misread the unit or whether the dashboard was broken. Later I confirmed I hadn’t misread it. At that scale, some statements had already been stuck running in the database for close to an hour, or even longer.

Without Performance Insights, fine-grained historical analysis was off the table. So I had to fall back to the dumb method: connect directly with a DBA account and manually inspect the live state using system views like pg_stat_activity.

3. Dig directly into pg_stat_activity: two deadlocked offenders surfaced

SELECT pid, state, wait_event_type, wait_event, now() - query_start AS duration,
       left(query, 150) AS query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY duration DESC
LIMIT 30;

Sorting by duration and scanning the results, two obvious offenders jumped out immediately:

The first one: a complex analytical query (cross-table aggregate statistics, something like “count the distribution of dialogue rounds per game from the event table”). Its state was active, and it had been running continuously for more than two days. It wasn’t just hanging there either—wait_event_type showed that it was genuinely burning CPU, and it even had two parallel worker processes attached. In other words, this was not some “forgotten idle transaction quietly lying around.” It was a large query that had been actively consuming CPU for two and a half days, with parallelism enabled.

The second one: a batch of UPDATE statements with identical parameters and only different PIDs, all targeting the same table. Their state was also active, and the wait events were all over the place—row locks, transaction locks, internal buffer locks. Their durations ranged from several minutes to nearly 5 hours. At a glance, it was obvious this was the same scheduled job being triggered over and over, piling up into a queue layer by layer—the oldest one had already blocked nearly 30 later arrivals.

At that point the profiles of both offenders were pretty clear: one was continuously bleeding the system dry, and the other was deadlocked in a queue. And both were the kind where letting them continue was completely pointless.

4. Stop the bleeding first: kill them and let the database breathe

At an incident site, the first principle is always stop the bleeding before investigating the cause. Once I confirmed these sessions were effectively stuck and there was no value in letting them continue, I cleared them directly to free up resources. This wasn’t just killing the two “representatives” mentioned above—I cleaned them up in batches by category:

-- First cancel the longest-running session in that analytical query chain;
-- the other parallel workers will terminate along with it
select pg_cancel_backend(<pid>);

The analytical query itself plus its two parallel workers: 3 sessions terminated together;
The 32 queued UPDATE sessions in the lock chain: after confirming their durations and states one by one, I cleared all of them, not just the head of the queue;
Sessions stuck on leaderboard count queries due to missing indexes and doing full table scans: there were 738 of them running at the time, and I batch-canceled those too.

After that, I watched the database metrics closely: CPU dropped immediately, active sessions fell from several thousand down to a few dozen, and connection count dropped along with them. A few minutes later, timeout errors across all online endpoints had basically disappeared, and containers were no longer repeatedly being marked unhealthy by health checks, restarted, and removed from traffic—the avalanche chain had been cut off at the root.

(Placeholder image: after killing the two stuck session groups, database active sessions / CPU usage dropped sharply back to normal levels)

One thing to make clear: this step was only damage control. The two “lesions” were removed, but it answered none of the “why” questions—why that analytical query could run for two and a half days without anyone noticing, why a scheduled job that looked protected could still pile up nearly 30 concurrent runs, and so on. Those were the things I only had time to investigate after the system was stabilized.

5. Retrospective question one: why did it only blow up after two and a half days?

Tracing backward through logs and the incident timeline, the first question was: that analytical query had clearly started two and a half days earlier, so why did nothing happen for so long, and why did it collapse at this exact moment?

The answer had to do with the machine spec of this database. This instance was quite beefy (db.m7i.8xlarge): 32 vCPUs, 128 GB memory, and high-performance storage. On a machine of that size, even with one heavy query continuously running at full parallelism, it might not be enough to crush the system outright in the short term. More often it just “quietly consumes more resources” and “slows background cleanup tasks a bit” in ways that aren’t very obvious, while the business side barely notices.

(Placeholder image: database instance spec screenshot, dozens of CPU cores / over 100 GB memory / high-performance storage)

In other words, the database had actually been brute-forcing its way through for more than 50 hours. That analytical query had been bleeding resources in the background the whole time, garbage collection had been slowed down, and tables had been bloating day by day—but the machine had enough headroom to survive it. It wasn’t until the day’s normal traffic peak arrived that the database, which had already lost a big chunk of its margin, finally couldn’t take it anymore, and the problem exploded all at once. That’s exactly what makes this kind of “chronic resource drain” issue so nasty: it doesn’t alert immediately. Instead, it suddenly blows up at the moment the remaining headroom is exhausted, making your first reaction “how did this happen with no warning?”—when in reality the warning signs were there the whole time, just masked by the machine’s excess capacity.

6. Retrospective question two: how did the scheduled job’s row-lock avalanche build up?

The second offender I killed during damage control came from a very plain scheduled job: find records stuck in “creating” status for too long without updates, and mark them as failed once they time out. The code explicitly used a Redis distributed lock to prevent multiple instances in the cluster from processing the same batch of data at the same time. (To avoid exposing the exact implementation, field names and variable names have been replaced, but the logic is unchanged.)

LOCK_KEY = f"{env}:item:monitor_creating:lock"
LOCK_TIMEOUT = SCHEDULE_MIN_INTERVAL  # 300 seconds, same as the minimum scheduling interval

lock = redis_client.lock(LOCK_KEY, timeout=LOCK_TIMEOUT, blocking=False)
if not lock.acquire():
    return  # Lock is occupied, skip this round
try:
    # ... execute batch UPDATE ...
finally:
    lock.release()

At first glance, all the expected protections seemed to be there. So why did the live incident still end up with nearly 30 concurrent copies of the same UPDATE, with the longest one stuck for almost 5 hours? There were two separate paths here that could both make the lock expire too early, and the real problem was the combination of both:

The TTL itself was too short: the job ran every 7.5 minutes, but the lock TTL was only 300 seconds (5 minutes). Once the database was already under heavy pressure and a single execution of this UPDATE got dragged out to several hours, the lock would automatically expire before the job had actually finished—Redis neither knows nor cares whether the previous round is still executing.
When a container was removed from traffic and restarted, the lock was “gracefully” released early: during database overload, lots of containers were marked timed out by health checks because responses got too slow, and were then removed and restarted. If a container holding the lock while executing this UPDATE got taken down, the cleanup logic in finally: lock.release() would still run, actively releasing the lock—even if the database transaction it was responsible for had not committed yet and was still holding a bunch of row locks. Once the lock was released, a newly started container could immediately acquire it and launch another round of the same UPDATE with the same conditions, competing for locks on the exact same rows with the previous transaction that still hadn’t committed.

Both paths lead to the same situation: the lock looks free, but the underlying transaction is not actually done. So new rounds of the job keep pouring in and queueing up, one after another, piling higher and higher. In this kind of batch-processing scenario, “the lock has been released” should not be treated as equivalent to “it is now safe to start the next round.” The semantics of the distributed lock need to cover whether the transaction it protects has truly ended, not just whether the scheduler layer should launch another invocation.

7. Side note: I thought it was an index issue, and then EXPLAIN slapped me in the face

Since this UPDATE itself was taking hours to run, my first instinct was of course to suspect a bad or missing index. I checked the table and the size was indeed scary: nearly 800 GB and more than 46 million rows. Then I checked how many rows matched the target status—only a little over a hundred in the whole table. That number actually made me even more convinced it was an index problem. I guessed the existing index didn’t cover the time field, so even after narrowing down to those hundred-odd candidate rows, it still had to fetch rows one by one to evaluate the timeout condition, and on a nearly 800 GB table that random I/O could be very expensive. I was almost ready to recommend adding a composite index on the spot.

Fortunately, before touching anything, I took one extra step and ran EXPLAIN. The result showed it was already using an index scan, and the cost was negligible—an existing composite index had already filtered those hundred-odd rows very cleanly. There was nothing wrong with the execution plan of this statement itself. Adding a new index had nothing to do with this incident; my earlier line of reasoning was simply wrong.

This side note is worth remembering: when you have “the table is huge” + “very few rows match,” it’s extremely easy to instinctively suspect indexing, but instinct is not a substitute for EXPLAIN. Luckily, all I did was run one extra read-only verification step. I didn’t actually go and perform an unnecessary index change on a nearly 800 GB table—which would also have consumed extra I/O. When the system is already under pressure, this kind of “trying to help but making it worse” can cost more than doing nothing.

8. Retrospective question three: where was the real load coming from?

If you only looked at that batch of lock-queued UPDATEs, it still didn’t explain why CPU had hit the ceiling. The reason they were stuck looked more like a consequence of “the whole instance is already under heavy pressure,” not the cause of “these few statements dragged the whole instance down.”

Only after comparing several load metrics did the picture become clear: active sessions spiked instantly to 2000+, and peak load briefly reached 2332—while the safe baseline for this 32-core instance under normal conditions was only around 32, meaning it was overloaded by more than 70x. At the same time, total database connections approached 2700. The real load hog was a very unremarkable count query on another table:

SELECT count(*) FROM t_item_result WHERE game_id = $1 AND public IS true;

This query backed a leaderboard-style endpoint. The existing index on the underlying table did not have game_id as its leading column, so filtering by game_id couldn’t use it effectively. Under normal low traffic, that was tolerable. But when the day’s traffic peak hit, hundreds of concurrent requests all landed on it at once, and every single one became a full table scan, instantly burning CPU to the ground. At the time, there were nearly 500 queries that had been running for more than 60 seconds. Only about one-third of them were related to the earlier version table; the remaining two-thirds were innocent queries dragged down by this environment—which also confirmed that the main load source was not the UPDATE path. There was also another comment-related table where a join field lacked supporting indexes, which added fuel to the fire too, though not at the same scale as this count query.

9. Making things worse: even the last self-healing escape route was blocked

In theory, when the database hits this kind of bloat and backlog, there is still one last line of defense: autovacuum, which can slowly clean things up and let the situation recover on its own. But when I checked the live state, even that escape route had been blocked. The autovacuum process responsible for cleaning a large TOAST storage area had been stuck for 3 hours and 07 minutes without finishing. The autovacuum processes on two other core tables had also been dragged down by throttling for 12 minutes 49 seconds and 6 minutes 01 seconds respectively, unable to make normal progress.

The fact that three autovacuum processes were each stuck on different tables showed this was not a localized issue on one table. The whole instance was already so resource-starved that even background cleanup tasks couldn’t get enough CPU time slices to run. At that point, self-healing was completely impossible. Manual intervention was the only option—which was exactly the step described earlier in section 4.

10. Putting the full chain together

If you connect the answers to the previous questions, the escalation path looks roughly like this:

Two and a half days earlier, an analytical query with parallelism started running and never exited. It not only consumed resources itself, but because its transaction stayed open for so long, it pinned the global MVCC snapshot horizon, preventing autovacuum from cleaning dead tuples normally, so core tables kept bloating.
The database had enough capacity to brute-force through more than 50 hours without obvious symptoms, until a routine traffic peak arrived. A hot count query missing the right index got amplified by concurrency into a large number of full table scans, and CPU was instantly burned through.
In that CPU-overloaded environment, a scheduled job that was originally protected by a distributed lock started queueing up round after round of the same UPDATE, because the TTL was too short and the lock was also being released early when containers were removed and restarted. It piled up to nearly 30 sessions.
Nowhere in the whole chain was there any statement-level timeout configured. So a large number of requests could only wait indefinitely, until the containers serving them were themselves marked timed out by health checks and removed/restarted, at which point the coroutines were passively canceled. Containers were replaced in waves—two waves in total, adding up to more than a dozen containers being marked unhealthy and restarted due to probe timeouts—which in turn fed step 3: newly started containers immediately grabbed the just-released lock and launched another round of conflicting UPDATEs, amplifying the loop.
By then, the self-healing mechanism (autovacuum) had already been dragged to a standstill, so only manual intervention could stop the bleeding.

With all five stacked together, the database and every service depending on it slowed down across the board. From the business side, the direct symptom was simply “all endpoints are timing out.” And once those two root sources were killed, the chain snapped immediately and all metrics quickly returned to normal.

11. The real holes that need patching

What we did on-site was only first aid. It solved none of the root causes. After sorting it all out, there are quite a few things to fix, roughly in this priority order:

1. Add statement-level timeouts to database connections

I checked, and right now there is no statement-level timeout configured at all on the database connection layer (statement_timeout in PostgreSQL). That means how long a SQL statement can stay stuck in the database depends entirely on which outer layer times out first. Leaving this unset is basically giving every slow query an unlimited line of credit.

The setup is straightforward. PostgreSQL supports several levels, so you can choose based on need:

-- Option 1: session level, run once after connection is established; only affects the current connection
SET statement_timeout = '15s';

-- Option 2: per database role; automatically applies to all future connections for this user
ALTER ROLE app_user SET statement_timeout = '15s';

-- Option 3: instance-level default (configured in postgresql.conf or a cloud provider parameter group)
statement_timeout = 15000   -- unit is milliseconds

If you’re using a connection pool / ORM, it’s even better to pass it directly when creating connections, for example with Python asyncpg:

create_async_engine(
    DATABASE_URL,
    connect_args={"server_settings": {"statement_timeout": "15000"}},  # milliseconds
)

The exact timeout value can’t just be guessed—you need to tier it by endpoint type. User-facing core read endpoints usually get single-digit to low double-digit seconds; background batch jobs and offline tasks can be looser, maybe tens of seconds. But no matter which tier, it must be an explicit number, not “unset.” In this incident, the longest batch of queries was stuck for nearly 5 hours. Even if we had set a very relaxed timeout like 60 seconds, it would never have dragged on to the point where manual intervention was required.

2. Change the scheduled job to commit in batches; the distributed lock cannot rely on a fixed TTL alone

The table is still growing, and the one-shot large-batch UPDATE approach itself does not age well as data volume increases. It should be split into smaller batches with incremental commits. On the distributed lock side, having a TTL that cannot cover “how long the job might take in the worst case” is itself a hidden risk, and the lock release timing needs to be redesigned too—service restarts should not automatically mean “it is safe to release the lock and wake up the next round.” Either add a renewal mechanism, or tie the lock lifecycle much more tightly to the transaction state it is protecting.

3. Add the two indexes that are actually missing

Add an index on the table hit by the hot count query that actually covers the real query conditions, using a non-blocking index build. Also add the corresponding index for the field used by the comment join query. This incident was also a reminder to myself: during troubleshooting, don’t decide by intuition which table “should get an index.” Use EXPLAIN or historical load analysis tools to find the real load source first, then make changes.

4. Add caching or periodic pre-aggregation for the hot count endpoint, instead of doing a real-time count(*) every time, to reduce direct pressure on the underlying large table.

5. Add a read replica dedicated to offline analysis: even just having one read-only instance that does not serve online business traffic, dedicated to offline analysis and data verification queries, would avoid situations like “someone forgets to close an analysis script connection and drags down the production primary database.” Letting the primary be used everywhere for ad hoc analysis creates too much risk exposure.

6. Fill in the missing monitoring and alerts: core database load metrics, slow SQL distribution, and so on. This incident also confirmed one thing: tools like Performance Insights, which let you inspect historical Top SQL and wait events, do cost a bit, but the difference between being able to locate the root cause in a few minutes during an incident versus not being able to is huge. For core databases, it’s worth enabling them by default.

Summary

My biggest takeaway from this investigation is: production incidents are rarely caused by a single factor. Most of the time, they’re several small issues that each look non-fatal on their own, but happen to line up at the same moment. A forgotten analytical script, a hot endpoint missing an index, a distributed lock whose TTL was too short and then got released at exactly the wrong time, plus no statement-level timeout anywhere in the chain—looked at separately, each one is the kind of oversight people can easily forgive. Stack them together, and you get a very real avalanche. And because the machine itself had strong enough performance, the problem stayed latent for more than 50 hours before being triggered by a traffic peak. That kind of “delayed explosion” is even more dangerous than “immediate alerting.”

Another more concrete lesson is this: during troubleshooting, even conclusions that “look very reasonable” are worth validating with tools before you act. At one point I was convinced it was an index issue. If I hadn’t run that extra EXPLAIN, I might have made a completely unnecessary index change on a nearly 800 GB table and consumed even more I/O in the process. Investigating a blocking chain can tell you who is waiting on whom, but identifying the real load source always comes back to one question: “who is actually burning CPU?” The two are not interchangeable. And when an incident happens, stopping the bleeding and giving the system room to breathe is always more important than trying to analyze everything slowly while still under pressure.

I Built an IntelliJ IDEA Plugin That Flattens All the REST APIs in a Project into a Single Panel

Thu, 30 Jul 2026 00:00:00 +0800

Preface

Back when I was writing in other languages, I’d already used this kind of “API browsing” plugin, and I always found it pretty handy. Once a project gets big, you really end up with a ton of endpoints. Old and new APIs get mixed together, and trying to dig them out of the codebase purely by memory is basically impossible—especially in a huge monorepo. You might vaguely remember “this one is user-related,” but what it’s called and which class it lives in? Total guesswork, and it’s hard to pinpoint precisely.

The IDE’s built-in global search doesn’t help that much either, because the / in an endpoint path is easily treated as a separator, so you can’t do a continuous match on the whole string. The most typical scenario: you copy a full path straight from Chrome DevTools’ Network panel, like /api/v1/user/detail/list, paste it into global search, and… nothing. You’re forced to split the path and try word by word, which is super annoying. This is especially deadly for backend folks—an API endpoint is the entry point of the entire data chain. From there you trace down through Service, Repository, and finally to the DB. You’re following that full chain; if you can’t even locate the entry point, everything after that is pointless.

What really pushed me to write my own was a few other things. First: as IDEA keeps updating, the plugins I used to rely on gradually became incompatible. Every so often I had to go hunt for replacements, and that hassle alone is irritating. Second: while looking for alternatives, I tried a bunch of similar plugins and found they were stuffed with bloated features I didn’t need—not that those features are bad; I get and respect other people’s product thinking. It’s just not what I want. Third, and most directly: lately I’ve been writing more and more Python. I searched the market for this category of plugin and almost all of them only recognize Java-framework Controller annotations. FastAPI? Nobody cares. But the amount of backend APIs written in Python is absolutely not less than Java.

So I just wrote one myself: cover both the Java ecosystem frameworks and FastAPI first, and stick as hard as possible to a single responsibility—“API browsing + navigation”—without piling on features I don’t use. If you need other languages/frameworks, feel free to open a PR. Let’s test together and fill in this puzzle piece by piece.

Main

First, draw a clear line: what this version does and doesn’t do

I took a look at RestfulBox screenshots and realized it’s a bit more ambitious: it even has panels for sending requests and viewing responses—kind of like embedding Postman into the IDE. I’m not planning to follow that path—first, that feature set isn’t trivial in complexity; second, there are already dedicated tools for sending requests, so there’s no need to reinvent it inside an IDE plugin. It also aligns with the “don’t be bloated” idea I mentioned earlier. This version is going all-in on one scenario: browse + navigate. Scan all endpoints, select one, jump straight to the code. That’s enough.

Architecture: one extension point to cover four frameworks

The core is an extracted extension point interface, ApiParserContributor, with one implementation per framework:

interface ApiParserContributor {
    fun collect(project: Project, scope: GlobalSearchScope): List<ApiEndpoint>
}

For the annotation recognition logic of Java frameworks like Spring, JAX-RS, and Micronaut, I didn’t reinvent the wheel—this is directly ported and adapted from RestfulHelper (MIT-licensed open source), including its robust value-extraction logic for handling string concatenation and constant references. After the adaptation, everything is unified to fit this project’s own ApiEndpoint model, and the license and attribution are properly preserved in NOTICE. The FastAPI part is newly written, based on Python PSI to parse path composition across @app.get(...), APIRouter(prefix=...), and include_router.

For Python support I used a small trick: in plugin.xml, declare an optional dependency with <depends optional="true">com.intellij.modules.python</depends>. That way, users who don’t have JetBrains’ official Python plugin installed can still install this plugin without errors—they just won’t see FastAPI endpoints. Everything else works as usual.

The UI layer doesn’t care which framework an endpoint comes from; it only consumes List<ApiEndpoint>. In the tool window, you can switch grouping across four dimensions: path prefix, method, file/class, and module. There’s also a search box for real-time filtering.

One pitfall: full scans blocking the UI thread

During development I ran into a pretty classic IDEA plugin pitfall: at first, every keystroke in the filter box would synchronously call all parser contributors to rescan the entire project, and then wait for results on the EDT (the UI main thread). On small projects you don’t notice it, but once the number of endpoints grows, the UI freezes on input and the experience is awful. Fundamentally, I was doing heavy index access on the wrong thread.

Later I switched to non-blocking async refresh: scan results are cached in a project-level service; when PSI changes, only the affected files are rescanned incrementally instead of doing a full rescan; searching and switching groupings operate only on an in-memory snapshot of the cache and no longer touch the index. Along the way I also found and fixed a related issue—when refresh failed or got canceled, the cache state used to be incorrectly marked as “clean,” which caused it to not rescan when it actually should have.

Besides the tree list in the tool window, I also added a global hotkey (Cmd+Option+\ / Ctrl+\) to bring up a fuzzy-search popup. Type a path fragment or method name and you can locate and jump directly—no need to manually open the tool window first and then hunt around.

There’s a small pitfall here too: across different OSes and user-customized keymaps, the actual effective shortcut can vary. If you hardcode a hint string, you’ll mislead people. So I built a dedicated settings page (Settings → Tools → RestfulController) that reads the currently active keymap and displays the shortcut the user can actually press, rather than a hardcoded default. On that page you can also change the shortcut directly, and adjust whether the popup appears centered on the screen or follows the mouse. After changes, it writes back into the IDE’s own keymap—the same data you see under Settings → Keymap—so there’s no “plugin stores its own copy and gets out of sync with the system” situation.

The settings page reads the currently active Keymap and shows the shortcut the user can actually press; you can also adjust the popup position here

All endpoints in the tool window are grouped by the selected dimension, with real-time search filtering

Fuzzy search triggered by the global shortcut—enter a path fragment and press Enter to jump to the corresponding handler method

Development story: I kept writing, then “someone else” took over and kept writing

This plugin was completed entirely with AI assistance, and in the middle I even swapped “people” a few times in a relay. At the very beginning I used another AI coding tool to kick things off, set the design, and build the first implementation. Halfway through, that tool’s session context was basically used up, so I dumped the handoff chat logs to Claude and had it continue. Claude pushed it forward a lot—the main features like the four framework parsers, the tool window, and global search were largely done in that phase. Then, as I kept going, Claude’s token budget expired too, so I switched to Codex to finish it off: the settings page, async refresh, shortcuts, and other UX details. I also did a round of performance and architecture self-check (the EDT-freeze pitfall above was found during this round).

At first I worried that having multiple different AIs take turns writing the same project would produce stylistically fragmented code. In practice it wasn’t a big problem—as long as the design docs and implementation plan are clear enough, and you hand those docs over during the transition, the new AI can align context quickly and keep moving. The human in the loop mainly steers direction, runs tests, and catches the obviously wrong bits.

Afterword

The plugin is now up and running, and the code is open-sourced at github.com/oreoft/restful-controller. Overall, my takeaway is: for a multi-framework IDE utility like this, the hard part isn’t the parsing details of any single framework (a lot of that can be learned from open source), but designing the extension point cleanly enough—adding a new framework shouldn’t require touching the UI or other parsers. This time, I think I nailed that.

Going forward I’ll probably keep polishing it based on the little annoyances I hit in real use. For now I don’t plan to expand toward “sending requests”—for me, browsing plus navigation already solves the most painful problem. As for support for other languages: same line as before, I’m waiting for your PR.

Oreoft

Turning My Windows Mini PC Loaded with Random Chores into PVE

Introduction

1. More and More Work Piled on Windows

2. Stability Issues Never Went Away

3. Three Iterations of Service Deployment

1. The Hyper-V Era

2. The WSL2 Era

3. All In Docker based on Windows

4. Why Switch to PVE

5. Cleaning Out the Dust First

6. Backing Up the Data First

7. BIOS Configuration and PVE Installation

8. Allocating Virtual Machines

Conclusion

Installing Tailscale on Alibaba Cloud Blocked All of Alibaba Cloud’s Own Internal Services

Introduction

1. What this machine is supposed to do

2. After installing Tailscale, DNS resolution completely broke

3. First workaround: switch DNS to public DNS

4. The same problem showed up again with apt

5. Put an allow rule at the very top of the INPUT chain

6. The internal address for the image registry wasn’t in that range

7. Community workarounds, and Tailscale’s official solution

8. If anti-spoofing is disabled, what replaces it?

9. What I actually changed

1. Add a tag and a node attribute in the policy file

2. Tag this machine with tag:aliyun

3. Enable rp_filter

4. Remove the old systemd unit

Verification

10. Alibaba Cloud’s built-in sysctl config can override yours

11. Retrospective

Summary

References

After Upgrading the Client, Redis Started Returning Tons of MOVED Errors Again: A Local Config That Had Been Lurking for Seven Months

Introduction

1. First, figure out which machines are connecting

2. Hypothesis 1: the removed socket_timeout

3. Hypothesis 2: MOVED and connection rebuilds are triggering each other

4. Couldn’t reproduce it on the bastion host

5. A reproduction that didn’t hold up

6. I could reproduce the symptom, but not the production cause

7. One INFO line in the startup logs

8. Why nothing happened for seven months, and why the API service was fine

9. The fix

10. Retrospective

Summary

Adding Redis Sharding Took Production Writes Down for Over an Hour

Introduction

1. Errors started 12 seconds after the shard was added

2. I thought a restart would fix it. It didn’t.

3. The None at the end of the error was the real clue

4. How the client lost all its nodes

5. This isn’t new—people had already hit it on GitHub

6. Why another service was unaffected: the version was three major releases newer

7. Temporary mitigation: add a watchdog

8. Postmortem: a few things more worth remembering than the bug itself

Summary

A Synchronous Blocking Call Froze the Entire Event Loop

Introduction

1. First, rule out the two most common suspects

2. Change the angle: who removed the containers?

3. Trace backward: what exactly was happening during those few seconds?

4. Find the real culprit: a call that “looked async but was actually sync”

5. One remaining question: why did the downstream storm last longer than the trigger itself?

6. Break down the mechanics: why can one synchronous call drag so many things down together?

1. “Written inside an async function” does not mean “this line of code is asynchronous”

2. If we rewrote the health check endpoint as synchronous, could that avoid frequent task removal?

3. Why did the timeouts explode in a cluster?

4. Process heartbeat ≠ response time of a single endpoint

7. Fix strategy

Summary

An Emergency Production Incident: Redis Memory Exhaustion, Precise Cleanup, and an Architectural Postmortem

Introduction

1. No Guessing in Production: Analyze Memory Distribution Across the Entire Keyspace

2. Check the Code: Can This Biggest Consumer Be Cleaned Up?

Can it be cleaned up? And how?

3. Fine-Grained Cleanup: Precisely Remove the Coldest 50% by Hit Pattern (Idle Time)

Execute a Safe Asynchronous Cleanup

4. The same problem showed up again with `apt`

2. Tag this machine with `tag:aliyun`

3. Enable `rp_filter`

2. Hypothesis 1: the removed `socket_timeout`

3. The `None` at the end of the error was the real clue