Debugging WiFi Router build on RPi with OpenWrt

Posted by Abiramy.r | Thu, 05 Sep 2024 | Tagged: tutorial, openwrt, linux

In any software project, defects are inevitable and requires debugging to root cause it, and this eventually leads to a fix. The key lies in understanding the issue, analyzing it, and choosing the right approach to resolve it. Debugging issues may seem overwhelming at first, but with a clear understanding of the problem and the right tools, the process becomes more manageable.

This article explores the debugging sequence for common issues encountered while working with the Linux WiFi stack.

Common Issues

While developing code or testing a particular feature, various issues may arise, such as:

  • Device Not Detected

  • Driver Not Loaded

  • Process Not Running

  • Crashes & Kernel Panics

  • Performance Bottlenecks

  • Compatibility Issues

Each of these requires different approaches of analysis and debugging.

Lessons from the Field: Debugging WiFi Issues

While working with the Openwrt target based on ath12k driver, which supports Qualcomm Wi-Fi 7 (IEEE 802.11be) devices, several challenges were encountered. Here are some key issues and how they were tackled:

  • Crashes and Kernel Panics : Misconfigurations, firmware incompatibilities, or unhandled exceptions led to unexpected crashes.

  • Packet Drops and Performance Issues : Unstable firmware, hardware limitations caused connectivity problems.

  • Hostapd and Wpa_supplicant Failures : Misconfigured network parameters and hidden driver bugs resulted in failures.

Now, let’s visualize how these issues manifest across different layers of the WiFi stack.

/static/images/wifi-stack.png

Debugging Essentials: Tracing, Crashes, Network Insights

Linux provides a variety of powerful tools to track down and resolve issues efficiently. Some essential debugging tools which were involved in our analysis:

  • Print Statements and Logs: The simplest yet most effective debugging method. Inserting printk (for kernel debugging) or printf statements (for user-space debugging) helps to track the program flow and identify issues.

  • GDB (GNU Debugger): A powerful tool for inspecting and controlling program execution, setting breakpoints, and analyzing crashes.

  • Tcpdump: A command-line tool for capturing and analyzing network packets, essential for diagnosing network-related issues.

  • Wireshark: A graphical tool for deep packet inspection, useful for troubleshooting network communications and protocol-level issues.

Debugging through Hostapd and Wpa_supplicant Logs

In Openwrt, by default hostapd or wpa_supplicant runs in less verbose mode, while debugging we may need a detailed log. To get the logs, we may need to start the hostapd or wpad in higher verbose mode. We would also like to run the hostapd in foreground to lively watch the logs in console.

  • wpad (Wi-Fi Protected Access Daemon) is the init script for hostapd (Host Access Point Daemon) when hostapd is built for both AP and STA mode support. 

# /etc/init.d/wpad stop

  • Stopping wpad script would allow us to run hostapd in foreground for manual debugging.

  • Start hostapd in debug mode and capture logs. 

# hostapd -ddd /var/run/hostapd-phy0_band* > /tmp/hostapd_log.txt &

  • hostapd - Manages wireless access points.

  • -ddd - Enables detailed debug logs.

  • /var/run/hostapd-phy0_band* - Loads configs for multiple bands.

  • > /tmp/hostapd_log.txt - Saves logs for review.

  • & - Runs in the background.

configuration files coresponding to each radio would be named as hostapd-phy0_band[123].conf

Modify hostapd.sh used by netifd to enable detailed logging of Wpa_Supplicant.

Edit the script located at:

# /lib/netifd/hostapd.sh

Modify the wpa_supplicant_run() function:

wpa_supplicant_run() {
        local ifname="$1"; shift

        _wpa_supplicant_common "$ifname"

        /usr/sbin/wpa_supplicant -ddd \
                ${network_bridge:+-b $network_bridge} \
                -P "/var/run/wpa_supplicant-${ifname}.pid" \
                -D ${_w_driver:-wext} \
                -i "$ifname" \
                -c "$_config" \
                -C "$_rpath" > /tmp/wpa_supplicant.log &
                #"$@"

        ....
}

  • wpa_supplicant - Handles client-side Wi-Fi authentication and association.

  • -ddd - Enables detailed debug logs.

  • > /tmp/wpa_supplicant.log - Redirects logs to a file for analysis.

  • & - Runs wpa_supplicant as a background process.

  • Reload the wifi settings as below, which stops and restarts the Wi-Fi interfaces. 

# wifi

  • Check if logs being generated. 

# ls -l /tmp/wpa_supplicant.log

  • With the detailed logs acquired we can look for any anamolies to futher root cause the issue.

Debugging Crashes with GDB

If the issue is a crash of hostapd/wpa_supplicant and core file is generated, we can debug that with GDB.

  • Need to use the multiarch gdb from the toolchain to support respective architecture. 

# gdb-multiarch /home/user/openwrt/build_dir/target-aarch64_cortex-a53+neon-vfpv4_musl/
linux-ipq53xx_generic/target-dir-8fd27a5d/usr/sbin/hostapd

  • Set the sysroot parameter of gdb to access the relevant libraries from the build specific to the target. 

# set sysroot /home/user/openwrt/build_dir/target-aarch64_cortex-a53+neon-vfpv4_musl/
linux-ipq53xx_generic/target-dir-8fd27a5d/

  • Load the core file which has been obtained from the crash of "hostapd" or "wpa_supplicant", usually a core file is generated in /tmp. Load the core file in GDB as below 

# core-file hostapd.1732089837.495.11.core

Conclusion

Debugging user-space applications is just the first step. In the upcoming article, we’ll venture into the depths of kernel debugging and unraveling techniques to diagnose and resolve kernel-level issues with precision.

Stay tuned for an exciting deep dive into the heart of the Linux kernel!