{"id":3931,"date":"2019-10-21T13:51:10","date_gmt":"2019-10-21T05:51:10","guid":{"rendered":"https:\/\/damogame.cn\/wordpress\/?p=3931"},"modified":"2019-10-21T13:51:10","modified_gmt":"2019-10-21T05:51:10","slug":"linux-3-10-%e7%9a%84%e5%8f%88%e4%b8%80%e6%ac%a1hung","status":"publish","type":"post","link":"https:\/\/i007.cc\/wordpress\/archives\/3931","title":{"rendered":"linux 3.10 \u7684\u53c8\u4e00\u6b21hung"},"content":{"rendered":"

\u539f\u6587\u5730\u5740<\/a><\/p>\n

 <\/p>\n

\n

\u6700\u8fd1\u53c8\u9047\u5230\u4e00\u6b21hung\uff0cdmesg\u4e2d\u5806\u6808\u5982\u4e0b\uff1a<\/p>\n

\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n
[176223.913270] ------------[ cut here ]------------\r\n[176223.913280] WARNING: CPU: 10 PID: 0 at net\/sched\/sch_generic.c:300 dev_watchdog+0x248\/0x260-----\u6ce8\u610fcpu\u53f7\u662f10\r\n[176223.913282] NETDEV WATCHDOG: eth4 (ixgbe): transmit queue 5 timed out----------------warn\u7684\u5185\u5bb9\r\n[176223.913283] Modules linked in: witdriver(OE) mysendmsg(OE) fuse tipc(OE) ossmod(OE) iptable_filter mptctl mptbase bonding dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ses enclosure sg mei_me mei shpchp lpc_ich i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_cpufreq acpi_pad tcp_bbr sch_fq binfmt_misc ip_tables ext4 mbcache jbd2 raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci libata ixgbe(OE) i40e(OE) mpt3sas(OE)\r\n[176223.913340]  ptp raid_class pps_core scsi_transport_sas i2c_core dca [last unloaded: witdriver]\r\n[176223.913347] CPU: 10 PID: 0 Comm: swapper\/10 Tainted: G           OE  ------------   3.10.0-693.21.1.el7.x86_64 #1\r\n[176223.913348] Hardware name: ZTE ZXCDN\/SC621DI-16F, BIOS 1.0b 09\/21\/2017\r\n[176223.913350] Call Trace:\r\n[176223.913351]  <IRQ>  [<ffffffff816b2f9e>] dump_stack+0x19\/0x1b\r\n[176223.913361]  [<ffffffff8108ae58>] __warn+0xd8\/0x100----------------------warn\u4e86\uff0c\u5185\u5bb9\u89c1\u7b2c\u4e8c\u884c\r\n[176223.913364]  [<ffffffff8108aedf>] warn_slowpath_fmt+0x5f\/0x80\r\n[176223.913369]  [<ffffffff815bcc28>] dev_watchdog+0x248\/0x260----------------\u5728\u5904\u7406\u5230dev_watchdog\u65f6\u5019\u51fa\u73b0\u95ee\u9898\r\n[176223.913372]  [<ffffffff815bc9e0>] ? dev_deactivate_queue.constprop.33+0x60\/0x60\r\n[176223.913375]  [<ffffffff8109a9c8>] call_timer_fn+0x38\/0x110----------------\u5904\u7406\u56de\u8c03\r\n[176223.913378]  [<ffffffff815bc9e0>] ? dev_deactivate_queue.constprop.33+0x60\/0x60\r\n[176223.913382]  [<ffffffff8109ceed>] run_timer_softirq+0x22d\/0x310-----------\u8fd9\u4e2a\u8f6f\u4e2d\u65ad\u662ftimer\r\n[176223.913386]  [<ffffffff8109404d>] __do_softirq+0xfd\/0x290\r\n[176223.913390]  [<ffffffff816c8afc>] call_softirq+0x1c\/0x30\r\n[176223.913394]  [<ffffffff8102d435>] do_softirq+0x65\/0xa0---------------------\u786c\u4e2d\u65ad\u5904\u7406\u540e\uff0c\u987a\u4fbf\u5904\u7406\u8f6f\u4e2d\u65ad\r\n[176223.913397]  [<ffffffff81094495>] irq_exit+0x175\/0x180---------------------\u786c\u4e2d\u65ad\u7ed3\u675f\r\n[176223.913400]  [<ffffffff816c9e88>] smp_apic_timer_interrupt+0x48\/0x60\r\n[176223.913402]  [<ffffffff816c6732>] apic_timer_interrupt+0x162\/0x170---------\u786c\u4e2d\u65ad\u5f00\u59cb\r\n[176223.913403]  <EOI>  [<ffffffff81534357>] ? cpuidle_enter_state+0x57\/0xd0\r\n[176223.913410]  [<ffffffff815344ae>] cpuidle_idle_call+0xde\/0x230\r\n[176223.913414]  [<ffffffff81034f8e>] arch_cpu_idle+0xe\/0x40\r\n[176223.913420]  [<ffffffff810edc4a>] cpu_startup_entry+0x14a\/0x1c0\r\n[176223.913423]  [<ffffffff81052222>] start_secondary+0x1f2\/0x270\r\n[176223.913425] ---[ end trace 30e7271cf4a53655 ]---\r\n[176223.913430] ixgbe 0000:86:00.0 eth4: Fake Tx hang detected with timeout of 5 seconds\r\n[176224.918915] ixgbe 0000:86:00.1 eth5: Fake Tx hang detected with timeout of 5 seconds\r\n[176224.918917] ixgbe 0000:af:00.1 eth7: Fake Tx hang detected with timeout of 5 seconds\r\n[176224.918948] ixgbe 0000:af:00.0 eth6: Fake Tx hang detected with timeout of 5 seconds\r\n[176224.934872] ixgbe 0000:18:00.1 eth1: Fake Tx hang detected with timeout of 5 seconds\r\n[176226.403293] hrtimer_gaq_time = 912735, soft_irq_time = 42216, cpu = 7\r\n[176233.921764] ixgbe 0000:86:00.0 eth4: Fake Tx hang detected with timeout of 10 seconds\r\n[176234.911438] ixgbe 0000:af:00.1 eth7: Fake Tx hang detected with timeout of 10 seconds\r\n[176234.911441] ixgbe 0000:86:00.1 eth5: Fake Tx hang detected with timeout of 10 seconds\r\n[176234.911474] ixgbe 0000:af:00.0 eth6: Fake Tx hang detected with timeout of 10 seconds\r\n[176234.943364] ixgbe 0000:18:00.1 eth1: Fake Tx hang detected with timeout of 10 seconds\r\n[176237.664127] hrtimer_gaq_time = 854268, soft_irq_time = 72171, cpu = 43<\/pre>\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n<\/div>\n
\u5982\u679c\u53ea\u76ef\u7740\u8fd9\u4e2a\u5806\u6808\u770b\uff0c\u53ef\u4ee5\u770b\u51fa\uff0c\u7f51\u5361\u7684\u00a0dev_watchdog \u51fd\u6570\u68c0\u6d4b\u5230\u4e86eth4\u7684queue 5 \u51fa\u73b0\u4e86\u00a0trans_timeout\u3002<\/p>\n
\u8d85\u65f6\u7684\u68c0\u6d4b\u5468\u671f\uff0c\u4e0d\u540c\u7684\u8bbe\u5907\u662f\u4e0d\u4e00\u6837\u7684\uff0cintel\u7684ixgbe\u5bf9\u5e94\u7684\u8d85\u65f6\u65f6\u95f4\u662f ixgbe_main.c \u4e2d\u8bbe\u7f6e\u7684<\/p>\n
\n
netdev->watchdog_timeo = 5 * HZ;<\/pre>\n<\/div>\n
\u68c0\u6d4b\u7684\u539f\u7406\u5c31\u662f\uff0c\u6bcf\u5f53\u53d1\u5305\u6210\u529f\u7684\u65f6\u5019\uff0c\u8bbe\u7f6e\u00a0txq->trans_start \uff0c\u5982\u679c\u8fd9\u4e2a\u503c\u5728timer\u5230\u671f\u68c0\u6d4b\u7684\u65f6\u5019\uff0c\u548c\u5f53\u524d\u65f6\u95f4\u76f8\u5dee \u4e00\u4e2a\u8d85\u65f6\u5468\u671f(\u9ed8\u8ba4\u662f5s\uff0c\u5404\u79cd\u9a71\u52a8\u53ef\u80fd\u4e0d\u540c)\uff0c\u5219\u8ba4\u4e3a\u51fa\u73b0\u53d1\u9001<\/p>\n
\u8d85\u65f6\u3002\u4f9d\u8d56\u4e8e\u5b9a\u65f6\u5668\u8f6f\u4e2d\u65ad\uff0c\u68c0\u6d4b\u7684\u5bf9\u8c61\u662f\u5904\u4e8erunning\u72b6\u6001\u7684\u7f51\u5361\u5df2\u7ecf\u5904\u4e8estop\u72b6\u6001\u7684\u53d1\u9001\u961f\u5217\uff0c\u8bdd\u8bf4\u7f51\u5361\u5982\u679c\u6ca1\u6709running\uff0c\u68c0\u6d4b\u4e5f\u6ca1\u6709\u610f\u4e49\u3002<\/p>\n
\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n
static inline netdev_tx_t netdev_start_xmit(struct sk_buff *skb, struct net_device *dev,\r\n                        struct netdev_queue *txq, bool more)\r\n{\r\n    const struct net_device_ops *ops = dev->netdev_ops;\r\n    int rc;\r\n    \/*__netdev_start_xmit \u91cc\u9762\u5c31\u5b8c\u5168\u662f\u4f7f\u7528driver \u7684ops\u53bb\u53d1\u5305\u4e86\uff0c\u5176\u5b9e\u5230\u6b64\u4e3a\u6b62\uff0c\u4e00\u4e2askb\u5df2\u7ecf\u4ecenetdevice\u5c42\u9001\u5230driver\u5c42\u4e86\uff0c\u63a5\u4e0b\u6765\u4f1a\u7b49\u5f85driver\u7684\u8fd4\u56de*\/\r\n    rc = __netdev_start_xmit(ops, skb, dev, more);\r\n    if (rc == NETDEV_TX_OK)\r\n        txq_trans_update(txq);\r\n\r\n    return rc;\r\n}<\/pre>\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n<\/div>\n
\n
static inline void txq_trans_update(struct netdev_queue *txq)\r\n{\r\n    if (txq->xmit_lock_owner != -1)\r\n        txq->trans_start = jiffies;---------------------\u66f4\u65b0\u5f53\u524d\u53d1\u9001\u961f\u5217txq\u7684\u6700\u540e\u53d1\u9001\u65f6\u95f4\r\n}<\/pre>\n<\/div>\n
ixgbe\u9a71\u52a8\u4f1a\u68c0\u6d4b\u662f\u5426pending\uff0c\u4e5f\u5c31\u662f\u6709\u7684tx\u4e2d\u7684ring\u6ca1\u53d8\u52a8\u4e86\uff0c\u90a3\u4e48\u6253\u5370warning\u3002<\/p>\n
\u63a5\u4e0b\u6765\uff0c\u53c8\u6709\u4e00\u4e2a\u5806\u6808\uff1a<\/p>\n
\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n
[176242.505583] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [kthread_send\/9:178838]-----------\u6ce8\u610f\u8fd9\u4e2a\u65f6\u5019\u662fcpu9\r\n[176242.506532] Modules linked in: witdriver(OE) mysendmsg(OE) fuse tipc(OE) ossmod(OE) iptable_filter mptctl mptbase bonding dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ses enclosure sg mei_me mei shpchp lpc_ich i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_cpufreq acpi_pad tcp_bbr sch_fq binfmt_misc ip_tables ext4 mbcache jbd2 raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci libata ixgbe(OE) i40e(OE) mpt3sas(OE)\r\n[176242.506588]  ptp raid_class pps_core scsi_transport_sas i2c_core dca [last unloaded: witdriver]\r\n[176242.506597] CPU: 9 PID: 178838 Comm: kthread_send\/9 Tainted: G        W  OE  ------------   3.10.0-693.21.1.el7.x86_64 #1\r\n[176242.506599] Hardware name: ZTE ZXCDN\/SC621DI-16F, BIOS 1.0b 09\/21\/2017\r\n[176242.506601] task: ffff88290f7ddee0 ti: ffff882b680d0000 task.ti: ffff882b680d0000\r\n[176242.506603] RIP: 0010:[<ffffffff811005ce>]  [<ffffffff811005ce>] native_queued_spin_lock_slowpath+0x1ce\/0x200\r\n[176242.506610] RSP: 0018:ffff882fbe843dc0  EFLAGS: 00000202\r\n[176242.506611] RAX: 0000000000000001 RBX: ffff882fbe843d50 RCX: 0000000000000001\r\n[176242.506613] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff882fb80a0bc0-------\u5927\u5bb6\u90fd\u5728\u62a2\u7684\u9501\r\n[176242.506614] RBP: ffff882fbe843dc0 R08: 0000000000000101 R09: ffff8801363f4000\r\n[176242.506616] R10: 0000000000000000 R11: ffff882fbe843da8 R12: ffff882fbe843d38\r\n[176242.506617] R13: ffffffff816c6732 R14: ffff882fbe843dc0 R15: ffff882fb80a0b40\r\n[176242.506619] FS:  0000000000000000(0000) GS:ffff882fbe840000(0000) knlGS:0000000000000000\r\n[176242.506621] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033\r\n[176242.506622] CR2: 00007fd86fd46316 CR3: 0000000001a0a000 CR4: 00000000003607e0\r\n[176242.506624] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000\r\n[176242.506626] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400\r\n[176242.506627] Call Trace:\r\n[176242.506629]  <IRQ>\r\n[176242.506630]\r\n[176242.506635]  [<ffffffff816adeee>] queued_spin_lock_slowpath+0xb\/0xf\r\n[176242.506640]  [<ffffffff816bb080>] _raw_spin_lock+0x20\/0x30\r\n[176242.506645]  [<ffffffff815bca52>] dev_watchdog+0x72\/0x260-----------------------\u540c\u4e0a\u5206\u6790\uff0c\u4e0d\u8fc7\u8fd9\u6b21\u662f\u62ff\u4e0d\u5230\u9501\r\n[176242.506649]  [<ffffffff815bc9e0>] ? dev_deactivate_queue.constprop.33+0x60\/0x60\r\n[176242.506653]  [<ffffffff8109a9c8>] call_timer_fn+0x38\/0x110\r\n[176242.506655]  [<ffffffff815bc9e0>] ? dev_deactivate_queue.constprop.33+0x60\/0x60\r\n[176242.506658]  [<ffffffff8109ceed>] run_timer_softirq+0x22d\/0x310\r\n[176242.506663]  [<ffffffff8109404d>] __do_softirq+0xfd\/0x290\r\n[176242.506667]  [<ffffffff816c8afc>] call_softirq+0x1c\/0x30\r\n[176242.506672]  [<ffffffff8102d435>] do_softirq+0x65\/0xa0\r\n[176242.506675]  [<ffffffff81094495>] irq_exit+0x175\/0x180\r\n[176242.506679]  [<ffffffff816c9e88>] smp_apic_timer_interrupt+0x48\/0x60\r\n[176242.506682]  [<ffffffff816c6732>] apic_timer_interrupt+0x162\/0x170----------------\u88ab\u786c\u4e2d\u65ad\u6253\u65ad\r\n[176242.506684]  <EOI><\/pre>\n
[176242.506702] [<ffffffffc01299e3>] ? ixgbe_xmit_frame_ring+0x53\/0xf30 [ixgbe]
\n[176242.506710] [<ffffffffc012a918>] ixgbe_xmit_frame+0x58\/0xd0 [ixgbe]
\n[176242.506715] [<ffffffffc043b63c>] wit_send_tasklet+0x73c\/0xae0 [witdriver]
\n[176242.506719] [<ffffffffc043ba95>] wit_kthread_xmit_fn+0xb5\/0x150 [witdriver]———-\u6211\u4eec\u7684\u5185\u6838\u7ebf\u7a0bkthread_send\u8c03\u7528\u7684\u53d1\u5305\u51fd\u6570
\n[176242.506723] [<ffffffffc043b9e0>] ? wit_send_tasklet+0xae0\/0xae0 [witdriver]
\n[176242.506726] [<ffffffff810b5241>] kthread+0xd1\/0xe0
\n[176242.506729] [<ffffffff810b5170>] ? insert_kthread_work+0x40\/0x40
\n[176242.506733] [<ffffffff816c5577>] ret_from_fork+0x77\/0xb0
\n[176242.506736] [<ffffffff810b5170>] ? insert_kthread_work+0x40\/0x40<\/p>\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n<\/div>\n
\u5206\u6790\u8fd9\u4e2a\u5806\u6808\uff0cdev_watchdog \u8fd9\u6b21\u62ff\u4e0d\u5230\u9501\u4e86\uff0c\u5df2\u7ecf\u6301\u7eed\u4e8622s\uff0c\u8d70\u67e5\u4ee3\u7801\uff0c\u5b83\u9700\u8981\u54ea\u4e00\u628a\u9501\u5462\uff1f<\/p>\n
\n
etif_tx_lock(dev);<\/pre>\n<\/div>\n
\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n
static inline void netif_tx_lock(struct net_device *dev)\r\n{\r\n    unsigned int i;\r\n    int cpu;\r\n\r\n    spin_lock(&dev->tx_global_lock);\r\n    cpu = smp_processor_id();\r\n    for (i = 0; i < dev->num_tx_queues; i++) {\r\n        struct netdev_queue *txq = netdev_get_tx_queue(dev, i);\r\n\r\n        \/* We are the only thread of execution doing a\r\n         * freeze, but we have to grab the _xmit_lock in\r\n         * order to synchronize with threads which are in\r\n         * the ->hard_start_xmit() handler and already\r\n         * checked the frozen bit.\r\n         *\/\r\n        __netif_tx_lock(txq, cpu);\r\n        set_bit(__QUEUE_STATE_FROZEN, &txq->state);\r\n        __netif_tx_unlock(txq);\r\n    }\r\n}<\/pre>\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n<\/div>\n
 <\/p>\n
\u90a3\u8fd9\u628a\u9501\u88ab\u8c01\u62ff\u4e86\u5462\uff1f\u88ab\u00a0wit_kthread_xmit_fn \u51fd\u6570\u62ff\u4e86\uff0c\u56e0\u4e3a\u6211\u4eec\u6b63\u5728\u53d1\u5305\u554a\u3002<\/p>\n
\u4e5f\u5c31\u662f\u8bf4\uff0c\u6211\u4eec\u7684\u5185\u6838\u7ebf\u7a0b\u53d1\u5305\u7684\u65f6\u5019\uff0c\u62ff\u4e86\u9501\uff0c\u7ed3\u679c\u88ab\u4e2d\u65ad\uff0c\u7136\u540e\u786c\u4e2d\u65ad\u5904\u7406dev_watchdog\u7684\u65f6\u5019\uff0c\u4e5f\u8981\u62ff\u8fd9\u628a\u9501\uff0c\u6b7b\u9501\u4e86\u3002<\/p>\n
\u63a5\u4e0b\u6765\uff0c\u53c8\u5f15\u53d1\u5176\u4ed6cpu\u6b7b\u9501\uff1a\uff08\u6bcf\u6b21\u8fdb\u5165_raw_spin_lock\uff0crdi\u4e2d\u5fc5\u7136\u5b58\u653e\u7684\u5c31\u662f\u9501\uff0cx86\u53cd\u6c47\u7f16\u770b\u7684\u8bdd\u786e\u5b9e\u662frdi\uff0c\u5176\u4ed6\u67b6\u6784\u4e0d\u662f\u5982\u6b64\uff09<\/p>\n
\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n
[176246.611936] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [a.out:189281]-------------------------------cpu25\r\n[176246.612901] Modules linked in: witdriver(OE) mysendmsg(OE) fuse tipc(OE) ossmod(OE) iptable_filter mptctl mptbase bonding dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ses enclosure sg mei_me mei shpchp lpc_ich i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_cpufreq acpi_pad tcp_bbr sch_fq binfmt_misc ip_tables ext4 mbcache jbd2 raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci libata ixgbe(OE) i40e(OE) mpt3sas(OE)\r\n[176246.612961]  ptp raid_class pps_core scsi_transport_sas i2c_core dca [last unloaded: witdriver]\r\n[176246.612969] CPU: 25 PID: 189281 Comm: a.out Tainted: G        W  OEL ------------   3.10.0-693.21.1.el7.x86_64 #1\r\n[176246.612970] Hardware name: ZTE ZXCDN\/SC621DI-16F, BIOS 1.0b 09\/21\/2017\r\n[176246.612973] task: ffff882ba14a0000 ti: ffff882ba92c8000 task.ti: ffff882ba92c8000\r\n[176246.612974] RIP: 0010:[<ffffffff81100556>]  [<ffffffff81100556>] native_queued_spin_lock_slowpath+0x156\/0x200\r\n[176246.612982] RSP: 0018:ffff882ba92cb978  EFLAGS: 00000202\r\n[176246.612983] RAX: 0000000000000101 RBX: 0000090190134ba9 RCX: 0000000000c90000\r\n[176246.612985] RDX: 0000000000c90101 RSI: 0000000000000101 RDI: ffff882fb80a0bc0------------\u5927\u5bb6\u90fd\u5728\u62a2\u7684\u9999\u997d\u997d\u5b58\u653e\u5728\u6b64\uff0c\u5c31\u662f\u8fd9\u628a\u9501\r\n[176246.612986] RBP: ffff882ba92cb978 R08: ffff882fbe959700 R09: 0000000000000000\r\n[176246.612988] R10: 0000000000000001 R11: ffffea017c3e5580 R12: ffff882c18a85c00\r\n[176246.612989] R13: 00000000000005dc R14: ffff882ba2091100 R15: ffffffff811e63d1\r\n[176246.612991] FS:  00007fbf25f9b700(0000) GS:ffff882fbe940000(0000) knlGS:0000000000000000\r\n[176246.612993] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033\r\n[176246.612995] CR2: 0000000000c79000 CR3: 000000593a2ec000 CR4: 00000000003607e0\r\n[176246.612996] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000\r\n[176246.612998] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400\r\n[176246.612999] Call Trace:\r\n[176246.613006]  [<ffffffff816adeee>] queued_spin_lock_slowpath+0xb\/0xf\r\n[176246.613011]  [<ffffffff816bb080>] _raw_spin_lock+0x20\/0x30\r\n[176246.613015]  [<ffffffff815bcd7f>] sch_direct_xmit+0x13f\/0x240------------qdisc\u4e2dsys\u8c03\u7528\u53d1\u5305\r\n[176246.613017]  [<ffffffff815bcf15>] __qdisc_run+0x95\/0x1c0\r\n[176246.613022]  [<ffffffff81596989>] __dev_queue_xmit+0x2c9\/0x550-----------\u8fd9\u4e2a\u662f\u5b9e\u9645\u7269\u7406\u7f51\u5361\u7684\r\n[176246.613025]  [<ffffffff81596c20>] dev_queue_xmit+0x10\/0x20\r\n[176246.613032]  [<ffffffffc03abbe2>] bond_dev_queue_xmit+0x32\/0x80 [bonding]\r\n[176246.613036]  [<ffffffffc03ad7be>] bond_start_xmit+0x1be\/0x420 [bonding]\r\n[176246.613039]  [<ffffffff81593bc0>] dev_hard_start_xmit+0x90\/0x1a0\r\n[176246.613041]  [<ffffffff81596b08>] __dev_queue_xmit+0x448\/0x550-----------\u8fd9\u4e2a\u662fbond\u7684\r\n[176246.613044]  [<ffffffff81596c20>] dev_queue_xmit+0x10\/0x20\r\n[176246.613048]  [<ffffffff815dd8f6>] ip_finish_output+0x546\/0x7a0\r\n[176246.613050]  [<ffffffff815dde53>] ip_output+0x73\/0xe0\r\n[176246.613053]  [<ffffffff815dba46>] ? __ip_local_out_sk+0xf6\/0x100\r\n[176246.613055]  [<ffffffff815dba87>] ip_local_out_sk+0x37\/0x40\r\n[176246.613058]  [<ffffffff815de8a6>] ip_send_skb+0x16\/0x50\r\n[176246.613062]  [<ffffffff81606a9c>] udp_send_skb+0xac\/0x2b0\r\n[176246.613065]  [<ffffffff81606cde>] udp_push_pending_frames+0x3e\/0x60\r\n[176246.613068]  [<ffffffff81608b19>] udp_sendpage+0x119\/0x1c0\r\n[176246.613071]  [<ffffffff81615490>] inet_sendpage+0x70\/0xe0\r\n[176246.613075]  [<ffffffff8157613e>] kernel_sendpage+0x1e\/0x30\r\n[176246.613079]  [<ffffffffc04363a8>] sendfile_slowpath_2+0x1a8\/0x1f0 [witdriver]\r\n[176246.613084]  [<ffffffffc04458a6>] sys_mycall+0x776\/0x8f0 [witdriver]\r\n[176246.613090]  [<ffffffff816c5715>] system_call_fastpath+0x1c\/0x21\r\n[176246.613091] Code: 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 <8b> 17 0f b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84<\/pre>\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n<\/div>\n
\u7531\u4e8e\u6211\u4eec\u62ff\u7684\u662fdev\u7684\u9501\uff0c\u5728\u6ca1\u5f00\u542f\u62a2\u5360\u7684\u60c5\u51b5\u4e0b\u4e0d\u8c03\u5ea6\u51fa\u53bb\uff0c\u4f46\u4e5f\u5e76\u6ca1\u6709\u5173\u4e2d\u65ad\uff0c\u8fd9\u6837\u5176\u4ed6cpu\u5728\u7cfb\u7edf\u8c03\u7528\u4e2d\u8981\u53d1\u5305\uff0c\u4e5f\u4f1a\u53bb\u5c1d\u8bd5\u62ff\u8fd9\u628a\u9501\uff0c\u597d\u5427\uff0c\u4f60\u4e5f\u6b7b\u9501\u4e86\uff0c\u7b49\u7740\u5427\u3002<\/p>\n
\u53ef\u4ee5\u60f3\u8c61\uff0c\u540e\u7eed\u5176\u4ed6cpu\u8981\u62ff\u8fd9\u628a\u9501\u7684\u8bdd\uff0c\u5927\u5bb6\u90fd\u4f1a\u6302\uff0c\u4e00\u4e2acpu\u7684softlock\u5bfc\u81f4\u5176\u4ed6cpu\u5168\u6302\uff0c\u53ea\u8981\u4f60\u8fd9\u4e2acpu\u6765\u62ff\u90a3\u9501\u5c31\u4f1a\u5982\u6b64\u3002<\/p>\n
\u7531\u4e8e\u6211\u4eec\u62ff\u7684\u662fspin_lock,\u5e76\u6ca1\u6709\u5173\u4e2d\u65ad\uff0c\u6240\u4ee5\u5b83\u8fd8\u53ef\u4ee5\u54cd\u5e94\u4e2d\u65ad\uff0c\u770b\u5982\u4e0b\u5806\u6808\u5c31\u660e\u767d\uff0c\u6700\u65e9\u6253\u5370\u7f51\u5361\u5f02\u5e38\u76849\u53f7cpu\uff0c\u5806\u6808\u5728\u6b7b\u7684\u65f6\u5019\u662f\u5982\u4e0b\u8fd9\u6837\u7684\uff1a<\/p>\n
\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n
crash> bt\r\nPID: 178838  TASK: ffff88290f7ddee0  CPU: 9   COMMAND: \"kthread_send\/9\"\r\n #0 [ffff882fbe843a70] machine_kexec at ffffffff8105d77b\r\n #1 [ffff882fbe843ad0] __crash_kexec at ffffffff8110aca2\r\n #2 [ffff882fbe843ba0] panic at ffffffff816ad52f\r\n #3 [ffff882fbe843c20] watchdog_timer_fn at ffffffff81135a51----------------------------hardlock\u68c0\u6d4b\u4e2dhrtimer\u7684\u56de\u8c03\r\n #4 [ffff882fbe843c58] __hrtimer_run_queues at ffffffff810b93a6\r\n #5 [ffff882fbe843cb0] hrtimer_interrupt at ffffffff810b993f\r\n #6 [ffff882fbe843cf8] local_apic_timer_interrupt at ffffffff8105467b\r\n #7 [ffff882fbe843d10] smp_apic_timer_interrupt at ffffffff816c9e83\r\n #8 [ffff882fbe843d28] apic_timer_interrupt at ffffffff816c6732--------------------------\u6b7b\u4e4b\u524d\u53c8\u589e\u52a0\u7684\u5806\u6808\uff0c\u8fd9\u4e2a\u662f\u786c\u4e2d\u65ad\r\n #9 [ffff882fbe843dc8] queued_spin_lock_slowpath at ffffffff816adeee---------------------\u4e00\u5f00\u59cb\u6253\u5370hung\u7684\u5806\u6808\uff0c\u5728\u81ea\u65cb\r\n#10 [ffff882fbe843dd8] _raw_spin_lock at ffffffff816bb080\r\n#11 [ffff882fbe843de8] dev_watchdog at ffffffff815bca52\r\n#12 [ffff882fbe843e28] call_timer_fn at ffffffff8109a9c8\r\n#13 [ffff882fbe843e60] run_timer_softirq at ffffffff8109ceed\r\n#14 [ffff882fbe843ed8] __do_softirq at ffffffff8109404d\r\n#15 [ffff882fbe843f48] call_softirq at ffffffff816c8afc\r\n#16 [ffff882fbe843f60] do_softirq at ffffffff8102d435\r\n#17 [ffff882fbe843f80] irq_exit at ffffffff81094495\r\n#18 [ffff882fbe843f98] smp_apic_timer_interrupt at ffffffff816c9e88\r\n#19 [ffff882fbe843fb0] apic_timer_interrupt at ffffffff816c6732\r\n--- <IRQ stack> ---\r\n#20 [ffff882b680d3c28] apic_timer_interrupt at ffffffff816c6732\r\n    [exception RIP: ixgbe_xmit_frame_ring+83]\r\n    RIP: ffffffffc01299e3  RSP: ffff882b680d3cd0  RFLAGS: 00000212\r\n    RAX: 0000000000000562  RBX: 0000000000000001  RCX: 000000000000403d\r\n    RDX: ffff882fb9331c00  RSI: ffff8828d7b8fac0  RDI: 0000000000000001\r\n    RBP: ffff882b680d3d48   R8: 0000000000000008   R9: 0000a0a5447b9d78\r\n    R10: ffff8828c6e84f00  R11: 000000002b3000b8  R12: ffff8828c0291b00\r\n    R13: 0000000022300000  R14: 0000000000000001  R15: ffff882b680d3cc0\r\n    ORIG_RAX: ffffffffffffff10  CS: 0010  SS: 0018\r\n#21 [ffff882b680d3d50] ixgbe_xmit_frame at ffffffffc012a918 [ixgbe]\r\n#22 [ffff882b680d3d80] wit_send_tasklet at ffffffffc043b63c [witdriver]\r\n#23 [ffff882b680d3e78] wit_kthread_xmit_fn at ffffffffc043ba95 [witdriver]\r\n#24 [ffff882b680d3ec8] kthread at ffffffff810b5241\r\n#25 [ffff882b680d3f50] ret_from_fork at ffffffff816c5577<\/pre>\n
\"\u590d\u5236\u4ee3\u7801\"<\/a><\/span><\/div>\n<\/div>\n
\u603b\u7ed3\uff1a<\/p>\n
1\u3001\u89e3\u51b3\u529e\u6cd5\u5f88\u7b80\u5355\uff0c\u5c31\u662f\u5728\u00a0wit_kthread_xmit_fn \u51fd\u6570\u4e2d\uff0c\u52a0\u4e86\u4e00\u4e2a\u5173\u95ed\u8f6f\u4e2d\u65ad\u5c31\u884c\uff0c\u56e0\u4e3a\u5c31\u662f\u8f6f\u4e2d\u65ad\u548csys\u62ff\u9501\u51b2\u7a81\u3002<\/p>\n
2\u3001\u5206\u6790\u8f6f\u9501\uff0c\u6700\u597d\u4ecedmesg\u4e2d\u5f00\u59cb\u5206\u6790\uff0c\u90a3\u4e2a\u65f6\u5019\u65f6\u95f4\u8fd8\u65e9\uff0c\u4f1a\u6309\u65f6\u95f4\u987a\u5e8f\u6765\u6253\u5370\u5806\u6808\uff0c\u6bd4\u8d77\u76f4\u63a5crash\u6765\u5206\u6790\u8981\u5feb\u4e00\u4e9b\uff0c\u7279\u522b\u662f\u5728\u5bf9\u81ea\u5df1\u6a21\u5757\u4ee3\u7801\u6bd4\u8f83\u719f\u6089\u7684\u60c5\u51b5\u4e0b\u3002<\/p>\n
 <\/p>\n<\/div>\n
\u6c34\u5e73\u6709\u9650\uff0c\u5982\u679c\u6709\u9519\u8bef\uff0c\u8bf7\u5e2e\u5fd9\u63d0\u9192\u6211\u3002\u5982\u679c\u60a8\u89c9\u5f97\u672c\u6587\u5bf9\u60a8\u6709\u5e2e\u52a9\uff0c\u53ef\u4ee5\u70b9\u51fb\u4e0b\u9762\u7684 \u63a8\u8350 \u652f\u6301\u4e00\u4e0b\u6211\u3002\u7248\u6743\u6240\u6709\uff0c\u9700\u8981\u8f6c\u53d1\u8bf7\u5e26\u4e0a\u672c\u6587\u6e90\u5730\u5740\uff0c\u535a\u5ba2\u4e00\u76f4\u5728\u66f4\u65b0\uff0c\u6b22\u8fce \u5173\u6ce8 \u3002<\/div>\n","protected":false},"excerpt":{"rendered":"
\u539f\u6587\u5730\u5740   \u6700\u8fd1\u53c8\u9047\u5230\u4e00\u6b21h<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"colormag_page_container_layout":"default_layout","colormag_page_sidebar_layout":"default_layout","footnotes":""},"categories":[],"tags":[117],"class_list":["post-3931","post","type-post","status-publish","format-standard","hentry","tag-01-linux"],"_links":{"self":[{"href":"https:\/\/i007.cc\/wordpress\/wp-json\/wp\/v2\/posts\/3931","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/i007.cc\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/i007.cc\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/i007.cc\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/i007.cc\/wordpress\/wp-json\/wp\/v2\/comments?post=3931"}],"version-history":[{"count":0,"href":"https:\/\/i007.cc\/wordpress\/wp-json\/wp\/v2\/posts\/3931\/revisions"}],"wp:attachment":[{"href":"https:\/\/i007.cc\/wordpress\/wp-json\/wp\/v2\/media?parent=3931"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/i007.cc\/wordpress\/wp-json\/wp\/v2\/categories?post=3931"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/i007.cc\/wordpress\/wp-json\/wp\/v2\/tags?post=3931"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}