Sunday, June 24, 2012

case study: irqbalance is not useful for AMD SMP

The original usage of IRQBALANCE is for Intel SMP (symmetric multi-processing CPU) that distributes attendant interrupts (IRQ) over the processors and cores. It is designed for finding a balance between power savings and optimal performance. Obviously, it is developed for Intel SMP at the moment of 2006 (you can find the article here), a time when Intel quad-core CPU has double group of two-cores sharing a single level 2 (L2) cache.

However, time always changes and IT industry flies faster. When AMD released its third generation  SMP CPU in 2007, take quad-core Opteron for example, there is no shared L2 cache. All L1 and L2 caches are dedicated to reduce cache pollution, while they are sharing a large L3 cache for buffering main memory.

So IRQBALANCE is obsolete. Its latest release code is irqbalance-0.56, which is published at the same period in 2006. At old time, system may get some benefits from balancing IRQ distributed among groups of cache domains. Now, with dedicated L2 cache in AMD SMP, IRQ balancing will dampen performance, especially for Linux network stack of large steaming packets, where threading performance is much more critical than load balancing. Because each L2 cache is a separated one, switching IRQ between cores even between multi-way CPU sockets will increase cache missing rates. As a result, performance for handling SFB IRQ is decreased. On the other hand, if the same CPU handles the same IRQ almost all the time, the IRQ handler function will unlikely leave the CPU cache, boosting the kernel performance.

There are some complains about IRQBALANCE. Here, a Chinese post shows an example: <irqbalance服务可能拖累双核、多核cpu>. Also another example is here: 
<Improving desktop performance in Fedora 14 by disabling irqbalance>.

To disable IRQBALANCE is quite easy. In Ubuntu, it is as follows:
ubuntu: set ENABLED=0 in /etc/default/irqbalance or sudo apt-get purge irqbalance


In Fedora, it is as follows:
fedora: service irqbalance stop or chkconfig irqbalance off or yum remove -y irqbalance