Back to all articles 9 min read
Security
• • 9 min read

The 100-Terabyte Ghost: Unmasking a Stealth Linux DDoS Botnet Using Kernel Bind Mounts

A real-world incident response post-mortem on how two production cloud servers unknowingly transmitted over 100 Terabytes of DDoS attack traffic, and how a subtle Linux kernel trick kept the malware invisible.

KloudBoy Security Team
Engineering Team
The 100-Terabyte Ghost: Unmasking a Stealth Linux DDoS Botnet Using Kernel Bind Mounts

Every DevOps engineer and systems administrator dreads the sudden, unexpected traffic spike notification from their cloud hosting provider.

In a recent emergency investigation handled by the KloudBoy security team, a client reached out with a shocking visual: their monthly bandwidth meters on their cloud hosting console were pinned at maximum capacity.

  • Server Alpha: 58.662 TB / 20 TB (100% capacity exceeded)
  • Server Beta: 48.766 TB / 20 TB (100% capacity exceeded)
[ OUTGOING TRAFFIC: 58.662 TB / 20 TB ]
[████████████████████████████████████████] 100%
"If you exceed the included traffic, the cost for every commenced TB extra is €1.00."

Combined, both servers had transmitted over 107 Terabytes of unauthorized outbound bandwidth in less than two weeks. At standard cloud bandwidth overage rates, an unexpected bill was compounding by the hour.

What was happening? Was an internal mail service being abused as an open spam relay? Were media files being scraped or hotlinked? Or had the machines been hijacked into a cyberweapon?

Here is the step-by-step forensic deep-dive into how we investigated the breach, uncovered an ingenious kernel-level concealment technique, and eradicated the botnet.

1. Initial Reconnaissance: Listening to the Network Interface

When facing an unexplained bandwidth explosion, the first place to look is the raw Linux kernel interface counters in /proc/net/dev.

Checking the primary network interface (eth0) on Server Alpha immediately revealed the astronomical scale:

cat /proc/net/dev | grep eth0
Inter-|   Receive                                                |  Transmit
 face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
  eth0: 13289820216 101675297    0    0    0     0          0         0 66541661577599 46957176343    0    0    0     0       0          0

Notice the numbers:

  • RX (Inbound): 13.2 GB across 101 Million packets
  • TX (Outbound): 66.54 Terabytes across 46.95 Billion packets

Doing some basic math on the average outbound packet size:

66,541,661,577,599 bytes / 46,957,176,343 packets ≈ 1,417 bytes/packet

A 1,417-byte average payload size matches the standard Ethernet Maximum Transmission Unit (MTU) of 1,500 bytes. This wasn’t standard web traffic, DNS queries, or outbound email—this was full-frame, line-rate packet streaming designed to completely saturate the server’s 1 Gbps uplink.

2. The Red Herrings

Our initial suspicions turned up empty:

  1. Mail Queues: Checking Postfix and Sendmail queues (mailq, postqueue -p) showed only standard internal system notifications. No mass-spam campaign.
  2. Web Server Logs: Examining Nginx access and error logs across all hosted virtual hosts revealed log files barely exceeding 10 MB in total. No massive file downloads, no open proxies.
  3. CPU Utilization: Running top showed modest CPU loads (average load under 0.6). The server appeared remarkably calm.

How could a server push 500 GB in a single hour without maxing out CPU or leaving trails in web server logs?

3. Querying the Metrics Database: Isolating the Attack Window

Fortunately, the server control panel on these machines records historical system metrics (CPU, disk I/O, network) into a local SQLite database (system.db).

We wrote a Python script to scan the differential in cumulative transmission (total_up) per day:

import sqlite3, datetime

conn = sqlite3.connect('system.db')
c = conn.cursor()
q = """
SELECT date(addtime, 'unixepoch') as d, max(total_up) - min(total_up) as diff_bytes
FROM network
GROUP BY d
HAVING diff_bytes > 107374182400
ORDER BY diff_bytes DESC
"""
c.execute(q)
for r in c.fetchall():
    print(f"Date: {r[0]} -> {r[1]/(1024**4):.2f} TB")

The output was staggering:

Date: Day 10 -> 3.86 TB
Date: Day 4  -> 3.76 TB
Date: Day 11 -> 3.54 TB
Date: Day 8  -> 3.52 TB
Date: Day 6  -> 3.35 TB
Date: Day 3  -> 3.30 TB
Date: Day 5  -> 2.97 TB
Date: Day 9  -> 2.70 TB
Date: Day 12 -> 2.68 TB

Drilling down to hourly metrics during the peak surge:

Hour 09:00 -> 105.95 GB
Hour 10:00 -> 500.69 GB   <-- Over 1.1 Gbps uplink saturation!
Hour 11:00 -> 281.96 GB
Hour 17:00 -> 213.04 GB
Hour 18:00 -> 230.90 GB
Hour 19:00 -> 280.29 GB

For nearly two weeks straight, the servers were pushing between 2.5 TB and 3.8 TB every single day, hitting peak burst rates of 500 GB in an hour.

4. The Smoking Gun: @m_bot_lock_instance

To find the culprit process, we inspected active network sockets using socket statistics (ss):

ss -t -u -a -n -p

Two suspicious entries stood out among thousands of benign sockets on Server Alpha:

tcp  ESTAB  0  0  198.51.100.24:48278  168.220.248.xxx:24032  users:(("python2.7",pid=4758,fd=9))
tcp  ESTAB  0  0  198.51.100.24:48246  168.220.248.xxx:24032  users:(("python2.7",pid=4758,fd=10))

And checking /proc/net/unix for abstract namespace UNIX domain sockets:

00000000578cb20b: 00000002 00000000 00000000 0002 01 37177065 @m_bot_lock_instance

On Server Beta, we ran the exact same audit:

tcp  ESTAB  0  0  198.51.100.88:52136  168.220.248.xxx:24032  users:(("qemu-ga",pid=2974251,fd=9))
00000000578cb20b: 00000002 00000000 00000000 0002 01 552763552 @m_bot_lock_instance

Both servers were actively connected to the exact same external IP and port (168.220.248.xxx:24032) and sharing the exact same internal singleton lock socket: @m_bot_lock_instance!

An IP lookup on the remote destination confirmed it was an external bulletproof Command & Control (C2) server.

5. The Master of Disguise: Linux Kernel Bind Mounts

We proceeded to inspect the binary behind the suspicious process PID on Server Alpha:

ls -la /proc/4758/exe
# lrwxrwxrwx. 1 root root 0 Sep 28 04:40 /proc/4758/exe -> /usr/sbin/python2.7

ls -la /usr/sbin/python2.7
# -rwxr-xr-x. 1 root root 7144 Jun 28  2022 /usr/sbin/python2.7

file /usr/sbin/python2.7
# /usr/sbin/python2.7: ELF 64-bit LSB executable, x86-64, version 1 (SYSV)

At first glance, /usr/sbin/python2.7 looked like a normal 7 KB system Python executable. But when we attempted to delete it:

rm -f /usr/sbin/python2.7
# rm: cannot remove ‘/usr/sbin/python2.7’: Device or resource busy

Why would a regular file report Device or resource busy?

We checked /proc/mounts:

cat /proc/mounts | grep python
# /dev/sda1 /usr/sbin/python2.7 ext4 rw,seclabel,relatime,data=ordered 0 0

The Trick Revealed:

The attacker had leveraged a Linux bind mount:

  1. They copied their 292 KB compiled malware to /usr/sbin/python2.7 and launched it.
  2. Once running in memory, they executed: 
    mount --bind /usr/bin/python2.7 /usr/sbin/python2.7
  3. Because /usr/bin/python2.7 (the genuine vendor Python binary) was mounted on top of /usr/sbin/python2.7, any sysadmin running ls -l, sha256sum, or file integrity monitors (like AIDE or Tripwire) only saw the legitimate, vendor-signed Python binary!
  4. Meanwhile, the Linux kernel kept the original executable open in RAM under the running PID.

We immediately unmounted the disguise:

umount -l /usr/sbin/python2.7
ls -la /usr/sbin/python2.7
# -rwxr-xr-x. 1 root root 292472 Jul  4 20:58 /usr/sbin/python2.7

The mask came off. The underlying file wasn’t a 7 KB Python binary at all—it was a 292 KB compiled malicious ELF executable!

We checked Server Beta and found the exact same technique, this time mounting over /usr/lib/qemu-ga:

umount -l /usr/lib/qemu-ga
ls -la /usr/lib/qemu-ga
# -rwxr-xr-x 1 root root 292472 Jul  4 20:38 /usr/lib/qemu-ga

Both files on both servers produced the exact same cryptographic fingerprint:

SHA256: 120b3b7c013a5f859edbbd60957fa560eaab797a42884b20b2238c8a99458f01

6. Dissecting the Malware: What Was It Doing?

We dumped strings from the memory image /proc/<PID>/exe:

CNC-BOT-RATCHET-INIT-V1
CNC-BOT-SESSION-KEY-V2
m_bot_lock_instance
attack_rand_next
_attack_rand_state
http_get_random_user_agent
HTTP_USER_AGENTS
http_generate_random_string
get_attack_mode
tcp_udp_checksum
http_set_nonblocking

Analysis of Malware Capabilities:

  • Mirai / Mozi Variant: Uses CNC-BOT-RATCHET-INIT-V1 session handshakes with encrypted C2 channels.
  • Raw Socket Forgery: Contains tcp_udp_checksum and attack_rand_next for crafting high-velocity spoofed UDP floods, TCP SYN floods, and ACK floods.
  • Layer-7 HTTP Flood Engine: Uses http_get_random_user_agent to launch application-layer denial-of-service floods against targeted web infrastructure.
  • No Exfiltration or Ransomware: Crucially, the binary contained no database query routines, no ransomware encryption routines, and no document scrapers. The attacker’s sole intent was to turn the servers into high-powered “zombies” for their DDoS-for-hire botnet.

7. The Backdoor: Reconstructing the Intrusion

How did the attackers get in?

Checking /etc/passwd and /etc/sudoers.d/ revealed an account created during the initial breach:

# /etc/passwd:
systemctl:x:1005:1006::/home/systemctl:/bin/bash

# /etc/sudoers.d/systemctl:
systemctl ALL=(ALL) NOPASSWD:ALL

The attacker had chosen the name systemctl so that casual audits of ps or /etc/passwd would assume it was a standard Linux systemd daemon account. In reality, it was a rogue shell account with full passwordless sudo rights.

Checking the authentication logs revealed the attack vector:

  • Both servers had PasswordAuthentication yes enabled on port 22.
  • Authentication logs recorded over 76,000 automated password brute-force attempts per week.
  • Attackers successfully guessed a password via brute-force, created the systemctl user, and planted the bot.
  • Later, the operators logged into the systemctl account over SSH, activated the flood engines, and blasted over 107 TB of traffic.

8. Incident Response & Hardening Playbook

Here are the concrete steps we took to completely eradicate the threat and secure both servers:

Step 1: Sever C2 Connections and Terminate Processes

# Terminate the malware PID
kill -9 <PID>

# Block the C2 IP and bot port at the firewall immediately
iptables -I OUTPUT -d 168.220.248.xxx -j DROP
iptables -I INPUT -s 168.220.248.xxx -j DROP
iptables -I OUTPUT -p tcp --dport 24032 -j DROP

Step 2: Unmask and Delete the Hidden Binaries

# Unmount the stealth bind mounts
umount -l /usr/sbin/python2.7
umount -l /usr/lib/qemu-ga

# Permanently delete the malware files
rm -f /usr/sbin/python2.7 /usr/lib/qemu-ga /usr/sbin/qemu-g

Step 3: Purge the Rogue Backdoor User

userdel -r -f systemctl
rm -f /etc/sudoers.d/systemctl
sed -i '/systemctl/d' /etc/sudoers.d/* /etc/sudoers

Step 4: Lock Down SSH (Key-Only Authentication)

Password authentication should never be enabled on public-facing production servers.

Edit /etc/ssh/sshd_config:

PasswordAuthentication no
ChallengeResponseAuthentication no
PermitRootLogin prohibit-password
PubkeyAuthentication yes

Reload the SSH daemon:

systemctl reload sshd || systemctl reload ssh

Step 5: Make Firewall Rules Permanent

On CentOS/RHEL (firewalld):

firewall-cmd --permanent --add-rich-rule='rule family="ipv4" destination address="168.220.248.xxx" drop'
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="168.220.248.xxx" drop'
firewall-cmd --reload

On Debian/Ubuntu (UFW):

ufw deny out to 168.220.248.xxx
ufw deny from 168.220.248.xxx
ufw reload

9. Key Lessons for Sysadmins

  1. Beware the Bind Mount Evasion: If a process is running a binary that cannot be removed or seems to have anomalous memory footprints compared to its disk size, check cat /proc/mounts. Modern rootkits and botnets increasingly use mount namespaces and bind mounts to mask malicious files under standard operating system paths.
  2. Abstract Sockets Don’t Lie: Regular file tools won’t show abstract UNIX sockets. Inspect /proc/net/unix for @-prefixed sockets (like @m_bot_lock_instance).
  3. Never Allow SSH Passwords: Even strong passwords can fall to credential stuffing or brute force if exposed to 76,000 attempts weekly. Enforce SSH key authentication globally.
  4. Monitor Outbound Bandwidth Baseline: Alert on sustained outbound traffic anomalies before you hit monthly quota limits. A server suddenly pushing gigabit line rates is almost always participating in an active DDoS attack.

Need emergency assistance with a compromised server, malware removal, or infrastructure hardening? Reach out to the KloudBoy Security Team.

Website hacked or dealing with malware?

Our 24/7 security engineers clean malware, harden servers, and restore uptime in under 2 hours.

Get Emergency Security Help