Intel LogoIntel ® Ethernet 700 Series
Linux Performance Tuning Guide
NEX Cloud Networking Group (NCNG)
Rev. 1.2
Kaum Ob Hlis 2024

Cov ntsiab lus zais
1 Kev kho keeb kwm
2 Taw qhia
3 Daim ntawv teev thawj zaug
4 Lub hauv paus ntsuas kev ua tau zoo thiab kev ntsuas kev ntsuas
5 Tuning i40e Driver Settings
6 Platform Tuning (i40e Tsis-Specific)
7 Adapter Bonding
8 Kev daws teeb meem kev ua haujlwm
9 Cov lus pom zoo rau Kev Ua Haujlwm Zoo Tshaj Plaws
10 Cov ntaub ntawv / Cov ntaub ntawv
10.1 Cov ntaub ntawv

Kev kho keeb kwm

Kev kho dua Hnub tim Cov lus pom
1.2 Kaum Ob Hlis 2024 · Ntxiv Cov Lus Qhia Ntxiv Kev Tswj Xyuas Hluav Taws Xob.
· Ntxiv Intel* Turbo Boost.
· Ntxiv Network Ntaus Backlog.
· Ntxiv Platform-Specific Configurations thiab Tuning.
· Ntxiv 4th Generation Intel* %eon* Scalable Processors.
· Ntxiv AMD EPYC.
· Hloov Kho Kho Qhov System Hardware Capabilities.
· Hloov kho iPerf2.
· Hloov kho iPerf3.
· Hloov tshiab Tx/Rx Queues.
· Hloov kho kev cuam tshuam kev cuam tshuam.
· Hloov Nplhaib Loj.
· Hloov Kho Platform Tuning (i40e Tsis-Specific).
· Hloov kho BIOS nqis.
· Hloov kho C-State Control.
· Hloov tshiab CPU zaus Scaling.
· Hloov kho daim ntawv thov chaw.
· Hloov Kho Kev Ua Haujlwm/Kernel Chaw.
· Hloov kho IP xa mus.
· Hloov kho Low Latency.
Lub Yim Hli 2023 Kev hloov pauv rau daim ntawv no indude:
· Ntxiv Cov Lus Qhia Ntxiv.
· Ntxiv kom ntseeg tau tias DDP Pob tau thauj khoom kom raug.
· Ntxiv iPerf2.
· Ntxiv iPerf3.
· Ntxiv netperf.
· Hloov kho IRQ Affinity.
· Ntxiv Tx/Rx Queues.
· Hloov Nplhaib Loj.
· Ntxiv Jumbo Ncej.
· Ntxiv Adapter Bonding.
· Ntxiv Intel svr-info Tool.
1.0 Peb Hlis 2016 Thawj Tso Tawm (Intel Public).

Taw qhia

Cov lus qhia no yog npaj los muab kev taw qhia rau kev sib tw ib puag ncig rau kev ua haujlwm zoo sib tham siv Intel ® Ethernet 700 Series NICs hauv Linux ib puag ncig. Nws tsom mus rau kho vajtse, tsav tsheb, thiab kev ua haujlwm ntawm cov txheej txheem thiab cov chaw uas tuaj yeem txhim kho kev ua haujlwm network. Nws yuav tsum raug sau tseg tias kev sib txuas lus tuaj yeem cuam tshuam los ntawm ib qho ntawm cov kev cuam tshuam sab nraud, tsuas yog feem ntau thiab ua rau cov no muaj nyob hauv phau ntawv qhia no.
1.1 Cov ntaub ntawv ntsig txog

Daim ntawv teev thawj zaug

2.1 Hloov tshiab Tsav Tsheb / Firmware Versions
Xyuas cov tsav tsheb / firmware versions siv ethtool -i ethx.
Hloov kho cov hauv qab no raws li xav tau:

2.2 Nyeem README
Tshawb xyuas cov teeb meem paub thiab tau txais cov lus qhia txog kev teeb tsa tshiab los ntawm README file suav nrog hauv pob khoom i40e.
2.3 Txheeb xyuas tias koj lub PCI nthuav qhia (PCIe) qhov yog x8
Qee qhov PCIe x8 qhov tseeb tau teeb tsa ua x4 qhov. Cov slots no muaj tsis txaus bandwidth rau tag nrho cov kab nqi nrog dual chaw nres nkoj thiab quad chaw nres nkoj li. Tsis tas li ntawd, yog tias koj muab PCIe v3.0-muaj peev xwm adapter rau hauv PCIe v2.x qhov, koj tsis tuaj yeem tau txais tag nrho bandwidth. Tus neeg tsav tsheb software kuaj pom qhov xwm txheej no thiab sau cov lus hauv qab no hauv lub kaw lus:
PCI-Express bandwidth muaj rau daim npav no tsis txaus rau kev ua haujlwm zoo. Rau qhov ua tau zoo tshaj plaws yuav tsum muaj x8 PCI-Express qhov.
Yog tias qhov yuam kev no tshwm sim, txav koj lub adapter mus rau qhov tseeb PCIe v3.0 x8 qhov los daws qhov teeb meem.
2.4 Xyuas System Hardware Capabilities
Ntawm 10 Gbps, 25 Gbps, thiab 40 Gbps Ethernet, muaj qee qhov tsawg kawg nkaus CPU thiab qhov yuav tsum tau ua. Nyob rau hauv dav dav, niaj hnub neeg rau zaub mov chav kawm processor thiab kev pom kev nco configuration rau koj lub platform yuav tsum txaus, tab sis cov kev xav tau sib txawv nyob ntawm seb koj ua hauj lwm. Txhua lub cim xeeb raws yuav tsum tau muaj nyob thiab nco ua haujlwm hom yuav tsum tau qhib hauv BIOS. Xyuas kom tseeb tias koj lub CPU thiab lub cim xeeb teeb tsa muaj peev xwm txhawb nqa qib ntawm kev ua haujlwm network koj xav tau rau koj cov haujlwm.
Nco tseg
XL710 yog 40 GbE maub los. Lub 2 x 40 GbE adapter siv tus maub los no tsis yog tsim los ua 2 x 40 GbE tab sis 1 x 40 GbE nrog qhov chaw nres nkoj rov qab. Thaum sim siv cov kab-tus nqi tsheb uas cuam tshuam nrog ob lub chaw nres nkoj, qhov hloov pauv sab hauv yog saturated thiab kev sib xyaw bandwidth ntawm ob qhov chaw nres nkoj raug txwv rau tag nrho SO Gbps.
2.4.1 Kernel Boot Parameters
Yog tias Intel® Virtualization Technology rau Directed I/O (Intel® VT-d) tau qhib rau hauv BIOS, Intel xav kom IOMMU nyob rau hauv hom dhau los rau kev ua haujlwm zoo hauv lub network. Qhov no tshem tawm DMA nyiaj siv ua haujlwm ntawm tus tswv tsheb thaum ua haujlwm Virtual Machine (VMs) tseem muaj txiaj ntsig ntawm Intel® VT-d. Qhov no yog ua tiav los ntawm kev ntxiv cov kab hauv qab no rau cov khoom siv khau raj: fommu-pt.
2.5 Xyuas kom DDP Pob yog thauj khoom kom raug
140ea thiab 140eb lub hauv paus tsav tsheb tsis muaj kev txhawb nqa ncaj qha rau Dynamic Device Personalization (DDP). Txhawm rau siv DDP nrog 700 Series khoom siv, DDP profile tuaj yeem siv nrog daim ntawv thov testpmd.
Yog xav paub ntxiv ntawm DDP profiles, thiab yuav ua li cas thov DDP profile nrog testpmd ntawm 700 Series cov khoom siv, xa mus rau Intel® Ethernet 700 Series Dynamic Device Personalization (DDP) Technology Guide.
Txhawm rau txheeb xyuas yog DDP profile tau loaded tiav:
testpmd> ddp tau daim ntawv teev npe 0 Profile num yog: 1
Nco tseg
Yog profile tus lej yog 0, tsis muaj pob DDP thauj khoom. Nyob rau hauv qhov xwm txheej ntawm DDP pob load yuam kev, lub cuab yeej ua haujlwm rau hom kev nyab xeeb thiab ntau yam kev ua haujlwm tsis muaj. Yog tias muaj teeb meem cuam tshuam txog kev thauj khoom DDP pob, nws yuav ua rau muaj teeb meem kev ua haujlwm. Rau cov kauj ruam daws teeb meem, xa mus rau Inte/* Ethernet 700 Series Dynamic Device Personalization (DDP) Technology Guide.

Lub hauv paus ntsuas kev ua tau zoo thiab kev ntsuas kev ntsuas

3.1 Network Performance Benchmarks
Ua ntej pib qhov kev tawm dag zog, nws yog ib qho tseem ceeb uas yuav tsum muaj qhov ntsuas qhov zoo ntawm koj lub network kev ua tau zoo. Feem ntau ntxiv rau qhov tau txais kev ntsuas thawj zaug ntawm koj daim ntawv thov tshwj xeeb / kev ua haujlwm ntawm kev ua haujlwm, nws yog ib lub tswv yim zoo los siv tus qauv kev ua haujlwm hauv network los xyuas kom meej tias koj lub cuab yeej hauv lub network yog nyob rau hauv lub xeev zoo.
Rau ib qho kev ua kom zoo dua qub, netperf lossis iperf thiab NetPIPE yog tag nrho cov khoom siv opensource dawb uas ua rau koj ntxhov siab txog kev sib txuas thiab kuaj xyuas cov teeb meem kev ua haujlwm.
Netperf muaj zog rau ob qho tib si throughput thiab latency xeem. NetPIPE yog cov cuab yeej tshwj xeeb latency tab sis tuaj yeem muab tso ua ke rau txhua qhov chaw.
Nco tseg
Qhov ntsuas TCP_RR hauv netperf rov latency hauv tus nqi ntawm kev hloov pauv / sec. Qhov no yog tus lej ncig mus ncig. Ib-txoj kev latency tuaj yeem suav nrog kev sib npaug hauv qab no:
Latency(usec) = (1⁄2) / [Transactions/sec] * 1,000,000
3.1.1 iPerf2
Intel pom zoo iperf2 dhau iperf3 rau feem ntau qhov kev ntsuas ntsuas vim qhov yooj yim ntawm kev siv thiab kev txhawb nqa ntawm ntau cov xov hauv ib qho piv txwv. Intel pom zoo kom khiav nrog -P xaiv nrog 2-4 xov rau 25G kev sib txuas thiab ib ncig ntawm 4-6 xov rau 40G kev sib txuas.

  • Txhawm rau khiav uni-directional tsheb los ntawm tus neeg siv khoom mus rau server: Server command example: ipf2 s
    Client command example: ip2c -P
  • Txhawm rau khiav ob txoj kev khiav tsheb los ntawm tus neeg siv khoom mus rau server (thiab rov ua dua): Server hais kom ua example: iperf2 -s -p
    Client command example:
    ib 2-c -p -P --full-duplex LOS YOG
    ib 2-c -p -P -d

Nco tseg
Ob qhov -full-duplex thiab -d kev xaiv hauv iperf2 tso cai rau cov neeg siv ua kev sim bidirectional. Txawm li cas los xij, -full-duplex kev xaiv tshwj xeeb tsom rau kev sim tag nrho duplex.
Nco tseg
Thaum kuaj iperf2 hla ntau lub chaw nres nkoj neeg rau zaub mov, tus chij -d tuaj yeem muab ntxiv rau cov neeg rau zaub mov hais kom khiav tag nrho cov neeg rau zaub mov hauv keeb kwm yav dhau los ntawm tib lub davhlau ya nyob twg qhov rais. Tus chij -d kuj tseem siv tau thaum cov neeg rau zaub mov hais kom muab tso rau hauv ib lub voj rau hauv tsab ntawv.
Nco tseg
Thaum khiav lub network throughput xeem nrog ib tug kwj / xov (example: P1), AMD processors yuav tsis muab qhov kev cia siab, tshwj xeeb tshaj yog bandwidth siab dua NICs (yog tias ceev yog> = 25G bandwidth). Yog li ntawd, daim ntawv thov pinning rau cov cores tshwj xeeb yog yuav tsum tau ua kom tiav siab dua. Saib Daim Ntawv Thov Chaw nyob ntawm nplooj 22.
3.1.2 iPerf3
Yog tias siv iperf3, ntau zaus ntawm daim ntawv thov yuav tsum tau siv advantage ntawm ntau cov xov, RSS, thiab kho vajtse queues. Intel pom zoo kom khiav nrog 2-4 daim ntawv thov ntu rau 25G kev sib txuas thiab ib puag ncig 4-6 zaug rau 40G kev sib txuas. Txhua qhov kev sib tham yuav tsum qhia qhov tshwj xeeb TCP chaw nres nkoj tus nqi siv qhov kev xaiv -p.

  • Txhawm rau khiav uni-directional tsheb los ntawm tus neeg siv khoom mus rau server:
    Server command example:
    iperf3 -s -p
    Client command example:
    ib 3-c -p
  • Txhawm rau khiav ob txoj kev khiav tsheb los ntawm cov neeg siv khoom mus rau server (thiab rov ua dua):
    Server command example:
    iperf3 -s -p
    Client command example: ip3c -p -P --ib
  • Txhawm rau pib ntau qhov xwm txheej (xov) ntawm iperf3, qhov kev pom zoo yog siv lub voj voog los qhia cov xov rau TCP cov chaw nres nkoj thiab khiav iperf3 hauv keeb kwm yav dhau los siv & tsim ntau cov txheej txheem nyob rau hauv parallel.
    Server command example, pib 4 threads: port =””; rau kuv hauv {0..3}; ua port=520$i; bash -c "iperf3 -s -p $port &"; ua tiav; Client command example, pib 4 threads – Transmit test port =””; rau kuv hauv {0..3}; ua port=520$i; bash -c "iperf3 -c $ serverIP -p $ chaw nres nkoj &"; ua tiav; Client command example, pib 4 threads – Tau txais qhov chaw xeem =””; rau kuv hauv {0..3}; ua port=520$i; bash -c "iperf3 -R -c $ serverIP -p $port &"; ua tiav; Rau 40G kev sib txuas, nce lub voj voog kom tsim tau txog 6 zaus / xov.

Nco tseg
Thaum khiav lub network throughput xeem nrog ib tug kwj / xov (example: P1), AMD processors yuav tsis muab qhov kev cia siab, tshwj xeeb tshaj yog bandwidth siab dua
NICs (yog tias ceev yog> = 25G bandwidth). Yog li ntawd, daim ntawv thov pinning rau cov cores tshwj xeeb yog yuav tsum tau ua kom tiav siab dua. Saib Daim Ntawv Thov Chaw nyob ntawm nplooj 22 thiab AMD EPYC ntawm nplooj 26.
3.1.3 NPE
Cov cuab yeej netperf yog qhov kev xaiv muaj zog rau ob qho tib si throughput thiab latency xeem.

  • Kev xeem TCP_STREAM hauv netperf ntsuas cov peev txheej dhau los ntawm lub cuab yeej. Server command example: netserver Client command example: netperf -t TCP_STREAM -l 30 -H
  • TCP_RR xeem hauv netperf rov latency nyob rau hauv tus nqi ntawm kev pauv / thib ob. Qhov no yog tus lej ncig mus ncig. Nws raug pom zoo kom siv -T x, x xaiv, yog x yog CPU hauv zos rau lub cuab yeej. Lub latency ib-txoj kev tuaj yeem suav nrog: Latency(usec) = (1⁄2) / [Transactions/sec]*1,000,\ Server command example: netserver
    Client command example: netperf -t TCP_RR -l 30 -H -T x, x
  • Txhawm rau pib ntau qhov xwm txheej (xov) ntawm netperf, qhov kev pom zoo yog siv lub voj voog los qhia cov xov rau TCP cov chaw nres nkoj thiab khiav netperf hauv keeb kwm yav dhau los siv & tsim ntau cov txheej txheem sib luag.
    Server command example, pib 8 threads:
    port = ""; rau kuv hauv {0..7}; ua port=520$i; bash -c "netserver -L $serverIP -p $port &"; ua tiav;
    Client command example, pib 8 threads: port=""; rau kuv hauv {0..7}; ua port=520$i; bash -c “netperf -H $serverIP -p $port -t TCP_STREAM -l 30 &”; ua tiav;

3.2 Kev Sib Txuas Lus
Tsom ntsoov rau ib qho kev hloov pauv ntawm ib lub sijhawm kom koj paub tias qhov kev hloov pauv twg ua rau koj qhov kev xeem. Cov txheej txheem ntau dua koj nyob rau hauv cov txheej txheem kho, nws yuav yooj yim dua los txheeb xyuas thiab hais txog qhov ua rau kev ua haujlwm tsis zoo.

Tuning i40e Driver Settings

4.1 IRQ Affinity
Configuring IRQ affinity yog li ntawd cuam tshuam rau cov sib txawv network queues affinitized rau txawv CPU cores yuav muaj kev cuam tshuam loj rau kev ua tau zoo, tshwj xeeb tshaj yog multithread throughput tests.
Txhawm rau teeb tsa IRQ affinity, nres irqbalance thiab tom qab ntawd siv cov ntawv set_irq_affinity los ntawm i40e qhov pob lossis tus pin queues manually. Disable user-space IRQ balancer kom pab kom kab pinning:

  • systemctl lov tes taw irqbalance
  • systemctl nres irqbalance
    Siv cov ntawv set_irq_affinity los ntawm i40e qhov pob (pom zoo):
  • Txhawm rau siv tag nrho cov cores:
    [path-to-i40epackage]/scripts/set_irq_affinity -X tag nrho ethX
  • Txhawm rau siv cov cores nkaus xwb ntawm NUMA lub qhov (socket): [path-to-i40epackage]/scripts/set_irq_affinity -X hauv zos ethX
  • Koj tuaj yeem xaiv ntau yam ntawm cov cores. Tsis txhob siv cpu0 vim nws khiav haujlwm timer. [path-to-i40epackage]/scripts/set_irq_affinity 1-2 ethX

Nco tseg
Cov ntawv affinity enables Transmit Packet Steering (XPS) ua ib feem ntawm cov txheej txheem pinning thaum qhov kev xaiv -x tau teev tseg. Thaum XPS tau qhib, Intel xav kom koj lov tes taw irqbalance, vim tias cov ntsiav tshuaj sib npaug nrog XPS tuaj yeem ua rau kev ua haujlwm tsis txaus ntseeg. Cov ntawv affinity disables XPS thaum qhov kev xaiv -X tau teev tseg. Disabling XPS thiab ua kom symmetric queues muaj txiaj ntsig zoo rau kev ua haujlwm uas qhov kev ua tau zoo tshaj plaws tau ua tiav thaum Tx thiab Rx tsheb tau txais kev pabcuam ntawm tib cov kab ke.
Configuring symmetric queues nyob rau hauv Linux yuav tsum tau kho lub network interface tus tsav tsheb tsis tau kom symmetric tau txais queues (Rx) thiab symmetric transmit queues (Tx) rau kev txhawb network adapters.
Nco tseg

  • Symmetric queues yog qhov kev sib txuas siab tshaj plaws, thiab tsis yog tag nrho 700 series network adapters lossis tsav tsheb txhawb nqa lawv.
  • Xyuas kom koj muaj tus tsav tsheb tsim nyog thiab kev txhawb nqa kho vajtse ua ntej sim teeb tsa cov kab sib luag.

Txhawm rau teeb tsa symmetric queues, ua raws cov kauj ruam dav dav no:

  1. Kho kom raug Network Interface Configuration File: Siv cov ntawv nyeem (example, vi, nano, lossis gedit) los hloov kho lub network interface configuration file. Cov file feem ntau nyob hauv qab /etc/sysconfig/network-scripts/ directory thiab muaj lub npe zoo li ifcfg-ethX, qhov twg ethX yog lub npe ntawm koj lub network interface.
  2. Ntxiv Symmetric Queue Parameters. Ntxiv cov kab hauv qab no rau lub network interface configuration file: ETHTOOL_OPTS=”rx-queues 8 tx-queues 8″
  3. Restart Network Service.
    Tom qab ua qhov kev hloov pauv, rov pib lub network kev pabcuam txhawm rau siv cov kev teeb tsa tshiab. sudo systemctl rov pib lub network

manually:

  • Nrhiav cov processors txuas rau txhua qhov siv: numactl –hardware lscpu
  • Nrhiav lub ntsej muag me ntsis rau txhua tus processors:
  • Piv txwv tias cores 0-11 rau ntawm 0: [1,2,4,8,10,20,40,80,100,200,400,800]
  • Nrhiav IRQs muab rau qhov chaw nres nkoj tau muab: grep ethX / proc / cuam tshuam thiab nco ntsoov IRQ qhov tseem ceeb Rau example, 181-192 rau 12 vectors loaded.
  • Echo tus nqi SMP affinity rau hauv IRQ nkag. Nco ntsoov tias qhov no yuav tsum tau ua rau txhua qhov IRQ nkag: ncha 1 > /proc/irq/181/smp_affinity echo 2 > /proc/irq/182/smp_affinity echo 4 > /proc/irq/183/smp_affinity Qhia IRQ affinity:
  • Txhawm rau qhia IRQ affinity rau tag nrho cov cores: /scripts/set_irq_affinity -s ethX
  • Txhawm rau qhia tsuas yog cores ntawm NUMA lub qhov (socket) hauv zos: /scripts/set_irq_affinity -s hauv zos ethX
  • Koj tuaj yeem xaiv ntau yam ntawm cov cores: /scripts/set_irq_affinity -s 40-0-8,16 ethX

Nco tseg
Cov ntawv set_irq_affinity txhawb nqa tus chij -s hauv i40e tsav tsheb version 2.16.11 thiab tom qab ntawd.
4.2 Tx/Rx Queues
Lub neej ntawd tus naj npawb ntawm cov kab tau qhib rau txhua qhov chaw nres nkoj Ethernet los ntawm tus tsav tsheb ntawm kev pib yog sib npaug rau tag nrho cov CPUs muaj nyob hauv lub platform. Qhov no ua haujlwm zoo rau ntau lub platform thiab kev teeb tsa ua haujlwm. Txawm li cas los xij, hauv cov platforms nrog cov tub ntxhais kawm siab thiab / lossis siab Ethernet chaw nres nkoj siab, qhov kev teeb tsa no tuaj yeem ua rau muaj kev sib cav sib ceg. Yog li ntawd, nws yuav tsim nyog nyob rau hauv qee qhov xwm txheej los hloov lub neej ntawd rau txhua qhov chaw nres nkoj hauv qhov system.
Lub neej ntawd tus lej ntawm Tx / Rx queues tuaj yeem sib txawv nyob ntawm tus qauv tshwj xeeb thiab tus tsav tsheb. Tus naj npawb ntawm cov kab tuaj yeem hloov kho siv ethtool -L hais kom ua hauv qab no.
Nco tseg
Hauv cov xwm txheej no, Intel xav kom koj txo qis cov kab ke ua ntej suav rau txhua qhov chaw nres nkoj kom tsis pub ntau tshaj li cov CPUs muaj nyob hauv NUMA node hauv zos rau qhov chaw nres nkoj adapter. Qee qhov xwm txheej, thaum sim sib npaug cov peev txheej ntawm kev siv cov chaw nres nkoj siab, nws yuav tsim nyog los txo tus lej no ntxiv.
Txhawm rau hloov kho kab teeb tsa:
Cov nram qab no example sets the port to 32 Tx/Rx queues: ethtool -L ethX 32
Examptso zis:
ethtool -l ethX
Channel parameters rau ethX: Pre-set siab tshaj plaws:
ib: 96
TXL: 96
Lwm yam: 1
sib: 96
Tam sim no hardware settings:
ib: 0
TXL: 0
Lwm yam: 1
sib: 32
4.3 Interrupt Moderation
Adaptive interrupt moderation yog nyob rau ntawm lub neej ntawd, thiab yog tsim los muab kev sib npaug ntawm kev siv CPU tsawg thiab kev ua haujlwm siab. Txawm li cas los xij, koj tuaj yeem sim kho qhov cuam tshuam kev teeb tsa manually kom haum rau koj cov ntaub ntawv siv.
Qhov ntau ntawm 0-235 microseconds muab qhov ua tau zoo ntawm 4,310 txog 250,000 cuam tshuam ib ob. Tus nqi ntawm rx-μsecs-siab tuaj yeem tsim kev ywj pheej ntawm rx-μsecs thiab tx-μsecs hauv tib ethtool hais kom ua, thiab tseem muaj kev ywj pheej ntawm kev hloov pauv kev cuam tshuam nruab nrab algorithm. Cov khoom siv hauv qab no txhawb nqa granularity hauv 2microsecond intervals, yog li cov txiaj ntsig uas nyob ib sab tuaj yeem ua rau tib tus nqi cuam tshuam.

  • Txhawm rau tua cov kev cuam tshuam cuam tshuam kev hloov pauv: ethtool -C ethX adaptive-rx tawm adaptive-tx tawm
  • Txhawm rau qhib kev hloov pauv hloov pauv hloov pauv: ethtool -C ethX adaptive-rx ntawm adaptive-tx rau

Qhov chaw zoo pib rau kev sib tw dav dav yog 84 μs, lossis ~ 12000 cuam tshuam / s. Yog tias koj pom rx_dropped counters tab tom khiav thaum lub tsheb khiav (siv ethtool -S ethX) ces tej zaum koj yuav muaj qeeb heev ntawm CPU, tsis txaus buffers los ntawm lub adapter lub nplhaib loj (ethtool -G) los tuav cov pob ntawv rau 84 μs lossis qis qis ntawm tus nqi cuam tshuam.

  • Txhawm rau teeb tsa kev cuam tshuam kev cuam tshuam rau qhov kev cuam tshuam tsis tu ncua ntawm 84 μs ntawm kev cuam tshuam (12000 cuam tshuam / s): ethtool -C ethX adaptive-rx tawm adaptive-tx tawm rx-usecs 84 tx-usecs 84 Tus nqi tom ntej sim, yog tias koj tsis maxed tawm ntawm CPU siv, yog 62 Qhov no siv ntau dua CPU, tab sis nws cov kev pabcuam buffers sai dua, thiab yuav tsum muaj tsawg dua cov lus piav qhia (ntiv nplhaib loj, ethtool -G).
  • Txhawm rau teeb tsa kev cuam tshuam kev cuam tshuam rau kev cuam tshuam ruaj khov ntawm 62 usecs ntawm kev cuam tshuam (16000 cuam tshuam / s). ethtool -C ethX adaptive-rx off adaptive-tx off rx-usecs 62 tx-usecs 62
    Yog tias rx_dropped counters nce thaum lub tsheb khiav (siv ethtool -S ethX), tej zaum koj yuav muaj CPU qeeb dhau, tsis txaus buffers los ntawm adapter lub nplhaib loj (ethtool -G), lossis tsawg dhau ntawm tus nqi cuam tshuam. Yog tias koj tsis tau maxed tawm ntawm kev siv CPU, koj tuaj yeem nce tus nqi cuam tshuam los ntawm kev txo tus nqi ITR. Qhov no siv ntau dua CPU, tab sis cov kev pabcuam buffers sai dua, thiab yuav tsum muaj tsawg tus piav qhia (ntiv nplhaib loj, ethtool -G).
    Yog tias koj lub CPU nyob ntawm 100%, ces nce tus nqi cuam tshuam tsis tau qhia. Hauv qee qhov xwm txheej xws li CPU khi ua haujlwm, tej zaum koj yuav xav nce μs tus nqi kom pab tau ntau lub sijhawm CPU rau lwm yam kev siv.
    Yog tias koj xav tau kev ua haujlwm qis qis thiab / lossis muaj ntau CPU los mob siab rau kev ua haujlwm network, koj tuaj yeem lov tes taw kev cuam tshuam tag nrho, uas ua rau muaj kev cuam tshuam rau hluav taws sai li sai tau.
  • To disable interrupt moderation ethtool -C ethX adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0

Nco tseg
Thaum khiav nrog kev cuam tshuam nruab nrab tsis ua haujlwm, qhov cuam tshuam ntawm txhua kab tuaj yeem siab heev. Xav txog suav nrog rx-usec-high parameter los teeb tsa lub siab txwv ntawm kev cuam tshuam tus nqi. Cov lus txib hauv qab no cuam tshuam kev hloov pauv kev cuam tshuam kev cuam tshuam thiab tso cai rau ntau tshaj 5 microseconds ua ntej qhia tias tau txais lossis xa tawm tiav. Es tsis txhob ua rau ntau li 200,000 interrupts ib ob, nws txwv tag nrho interrupts ib ob mus rau 50,000 ntawm lub rx-usec-siab parameter. # ethtool -C ethX adaptive-rx tawm adaptive-tx tawm rx-usecs-high 20 rx-usecs 5 txusecs 5 Sim kho qhov xa tawm / txais / qhov tseem ceeb tshaj plaws coalescing timer siab dua (80/100/150/200) lossis qis dua (25/20/10/5)
4.4 Nplhaib Loj
Yog tias koj pom rx_dropped cov txee hauv ethtool -S ethX (rx_dropped, rx_dropped.nic), lossis xav tias cache siab nrog ntau cov kab ua haujlwm, koj tuaj yeem sim kho lub nplhaib loj los ntawm tus nqi qub. Tus nqi pib yog 512, qhov siab tshaj plaws yog 4096.

  • Txhawm rau txheeb xyuas cov txiaj ntsig tam sim no: ethtool -g ethX
    Yog tias nws xav tias tsis muaj buffering ua rau poob ntawm qhov cuam tshuam tam sim no, koj tuaj yeem sim qhov siab tshaj plaws ua ntej, tom qab ntawd qhov tsawg kawg nkaus, tom qab ntawd txuas ntxiv mus rau hauv kev tshawb nrhiav binary kom txog thaum koj pom qhov ua tau zoo.
    Yog tias cache siab raug xav tias (ntau cov kab ua haujlwm) txo cov buffers los ntawm lub neej ntawd tuaj yeem pab Intel ® Data Direct I/O (Intel ® DDIO) ua haujlwm nrog kev ua haujlwm zoo dua. Intel xav kom sim 128 lossis 256 rau ib kab, paub tias qhov nce ntawm kev cuam tshuam ntawm ethtool -C tej zaum yuav tsim nyog kom tsis txhob muaj qhov nce hauv rx_dropped.
  • Txhawm rau teeb tsa lub nplhaib loj rau tus nqi tas li: ethtool -G eth12 rx 256 tx 256

Nco tseg
Txhawm rau txhim kho Rx pob ntawv poob pom nrog ethtool -S ethX|grep poob, xav txog kev nce lub nplhaib loj rau 4096. Kev sim nrhiav qhov chaw zoo tshaj plaws rau kev ua haujlwm tab sis saib xyuas kev siv lub cim xeeb ntau dua nrog cov nqi siab dua.
4.5 Flow Control
Txheej 2 ntws tswj tuaj yeem cuam tshuam TCP kev ua tau zoo heev thiab raug pom zoo kom ua tsis taus rau feem ntau cov haujlwm. Ib qho kev zam tshwj xeeb yog kev sib tsoo tsheb uas qhov tawg tsis ntev hauv lub sijhawm.
Flow tswj yog neeg xiam los ntawm lub neej ntawd.

  • Txhawm rau kom tswj tau ntws: ethtool -A ethX rx on tx on
  • Txhawm rau lov tes taw ntws tswj: ethtool -A ethX rx tawm tx tawm

Nco tseg
Koj yuav tsum muaj lub peev xwm tswj tau qhov sib txuas tus khub kom ua tiav kev tswj hwm ntws.
4.6 Jumbo Ncej
Thaum qhov kev cia siab ib puag ncig muaj cov ntaub ntawv loj ntawm cov ntaub ntawv raug xa mus, nws yuav muaj txiaj ntsig zoo los pab kom lub jumbo ncej feature. Kev them nyiaj yug Jumbo Frames yog qhib los ntawm kev hloov qhov Maximum Transmission Unit (MTU) mus rau tus nqi loj dua li tus nqi qub ntawm 1500. Qhov no tso cai rau lub cuab yeej hloov cov ntaub ntawv hauv cov pob ntawv loj dua hauv lub network ib puag ncig. Qhov kev teeb tsa no tuaj yeem txhim kho kev xa tawm thiab txo qis kev siv CPU rau cov haujlwm loj I / O. Txawm li cas los xij, nws yuav cuam tshuam cov pob ntawv me me lossis latency-sensitive workloads.
Nco tseg
Jumbo thav duab lossis loj MTU teeb tsa yuav tsum tau teeb tsa kom zoo thoob plaws koj lub network ib puag ncig.
Siv cov lus txib ifconfig kom nce MTU loj. Rau example, nkag mus rau hauv qab no, qhov twg yog tus lej interface: ifconfig ua 9000up
Xwb, koj tuaj yeem siv ip hais kom ua raws li hauv qab no: ip txuas teeb mtu 9000 dev ip link teeb tsa dev

Platform Tuning (i40e Tsis-Specific)

5.1 BIOS nqis

  • Qhib Intel® VT-d rau kev ua haujlwm virtualization.
  • Hyper-threading (logical processors) tuaj yeem cuam tshuam kev ua haujlwm. Sim nrog nws rau lossis tawm rau koj cov haujlwm.
  • Intel® Turbo Boost tso cai rau CPU cores ua haujlwm ntawm ntau zaus dua li lub hauv paus zaus ntawm CPU. Enabling Intel® Turbo Boost tuaj yeem txhim kho kev ua haujlwm rau ntau qhov haujlwm tab sis siv zog ntau dua kom cov cores ntawm ntau zaus. Kev sim nrog Turbo Boost off / on rau koj cov haujlwm.

Nco tseg
Turbo zaus tsis tau lees paub yog tias lub platform tab tom ntsib kev siv CPU tag nrho. Cov tub ntxhais kawm siab dua turbo zaus raug txo qis raws li kev siv CPU tag nrho nce.
5.2 Tswj fais fab
Kev tswj fais fab tuaj yeem cuam tshuam kev ua haujlwm, tshwj xeeb tshaj yog nyob rau hauv kev ua haujlwm qis qis. Yog tias qhov kev ua tau zoo yog qhov tseem ceeb dua li txo qis kev siv hluav taws xob, Intel xav kom koj sim ua kom txo qis cov teebmeem ntawm kev tswj fais fab. Muaj ntau ntau txoj hauv kev los txwv kev tswj fais fab, los ntawm cov cuab yeej ua haujlwm, kev teeb tsa BIOS, thiab cov kernel boot tsis. Xaiv txoj kev zoo tshaj plaws thiab qib kom haum rau koj ib puag ncig.
5.2.1 C-State Control
Kev txwv C-xeev nkag mus rau CO lossis C1 txhim kho kev ua haujlwm thiab nce kev siv hluav taws xob.
Disabling CPU Package C6 lub xeev nkag tuaj yeem txhim kho kev ua haujlwm network. Txawm li cas los xij, qhov no nce kev siv zog.
Cov kev xaiv hauv qab no muaj:

  • Dynamically tswj C-lub xeev nkag:
    Qhib
    /dev/cpu_dma_latency thiab sau qhov siab tshaj plaws tso cai latency rau nws.

Nco tseg
Muaj ib qho kev pab cuam me me hu ua cpudmalatency.c uas tuaj yeem rub tawm los ntawm cov chaw qhib hauv zej zog, muab tso ua ke, thiab khiav los ntawm kab hais kom ua raws nraim qhov no.
Cov nram qab no example tso cai tsib μs ntawm lub sijhawm sawv, thiab yog li tso cai C1 nkag: cpudmalatency 5 &

  • Txwv qhov siab tshaj plaws C-xeev nyob rau hauv lub kernel boot settings:
    Rau Intel CPUs: intel_idle.max_cstates=1
    Rau cov uas tsis yog Intel CPUs: processor.max_cstates=1
  • Siv cov lus txib cpupower los xyuas thiab lov tes taw CPU C6 xeev: Kos: cpupower saib lossis cpupower idle-info
    Disable C6: cpupower idle-set -d3 los yog
    Disable C-States: cpupower idle-set -D0

Nco tseg:

  1. Disable C-states ntawm CPU yog tias lub server muaj Intel® 4th Gen Intel® Xeon® Scalable Processor(s). Thaum Hyper Threading tau qhib lossis xiam oob qhab, kev ua haujlwm tsis siv neeg lub xeev (-D0) tiv thaiv cov cores los ntawm kev nkag mus rau lub xeev qis zog thaum lub sijhawm tsis ua haujlwm thiab txo qis latency rau CPU rau kev hloov pauv ntawm lub xeev tsis ua haujlwm thiab lub xeev nquag.
  2. Kev tswj fais fab ntawm Intel® 4th Gen Intel® Xeon® Scalable Processor yog qhov hnyav heev. Txhawm rau zam cov cores los ntawm kev nkag mus rau cov xeev qis, sim txo cov cores hauv kev siv kom lawv tsaug zog ntev dua (ethtool -L ua ke ). Tsis tas li ntawd, khi cuam tshuam rau cov cores tshwj xeeb siv teeb tsa irq affinity (feem ntau nrog -x hauv zos lossis cov npe ntawm CPU cores), thiab xyuas kom meej tias cov haujlwm ua haujlwm khiav ntawm tib lub cores nrog cov haujlwm lossis numactl. Qhov no txhim kho kev ua tau zoo los ntawm kev ua kom cov cores nquag thiab ua kom zoo rau kev cuam tshuam.

Qhib C6:
cpupower idle-set -d3
Qhib C-States:
cpupower idle-set -E

  • Lwm txoj hauv kev yog siv cov cuab yeej tuned (nrog rau ntau qhov kev faib tawm Linux) los teeb tsa qhov kev ua tau zoo.file. Cov profiles hloov ntau qhov chaw OS uas tuaj yeem cuam tshuam kev ua haujlwm thoob plaws ntau daim ntawv thov. Nws tau pom tias lub network-throughput profile muab kev txhim kho rau feem ntau ntawm kev ua haujlwm.
    Tshawb xyuas:
    tuned-adm nquag
    Teeb:
    tuned-adm profile network-thooj
    Nco tseg
    Tuned kev pabcuam yuav tsum tau khiav rau cov lus txib saum toj no. Txhawm rau kuaj / rov pib dua, tuned: systemctl raws li txoj cai tuned systemctl restart tuned
    Koj tuaj yeem tsis tso cai rau C-xeev nkag los ntawm kev ntxiv cov hauv qab no rau cov kab khau raj kernel:
    idle=poll
  • Txwv lub C-xeev los ntawm lub kaw lus BIOS lub zog tswj kev teeb tsa, uas tej zaum yuav muaj qhov ua tau zoofile muaj.
    Cov cuab yeej xws li turbostat lossis x86_energy_perf_policy tuaj yeem siv los xyuas lossis teeb tsa kev tswj hwm lub zog.

5.2.2 PCIe Fais Fab Tswj Xyuas
Active-State Power Management (ASPM) ua rau lub xeev qis zog rau PCIe txuas thaum lawv tsis siv. Qhov no tuaj yeem ua rau latency siab dua ntawm PCIe network li, yog li Intel xav kom koj lov tes taw ASPM rau latency-sensitive workloads. Disable ASPM los ntawm kev ntxiv cov hauv qab no rau cov kab khau raj kernel: pcie_aspm = tawm
5.2.3 CPU zaus Scaling
CPU zaus scaling (los yog CPU ceev scaling) yog Linux fais fab kev tswj cov txheej txheem nyob rau hauv uas lub kaw lus moos ceev yog kho ntawm ya mus txuag lub zog thiab kub. Ib yam li C-xeev, qhov no tuaj yeem ua rau tsis xav tau latency ntawm kev sib txuas hauv network.
Lub cuab yeej cpupower kuj tuaj yeem siv los xyuas thiab hloov kho CPU kev ua haujlwm tsis raug thiab txwv:

  • Kos: cpupower saib lossis
  • Teem CPUs rau hom kev ua haujlwm: cpupower zaus-set -g kev ua haujlwm

Nco tseg
Kev hloov kho rau CPU zaus txwv tuaj yeem cuam tshuam rau ntau yam haujlwm thiab tuaj yeem cuam tshuam lwm yam, xws li CPU turbo hom.
Txhawm rau lov tes taw CPU zaus scaling, lov tes taw CPU fais fab kev pabcuam los ntawm cov lus txib hauv qab no:
systemctl nres cpupower.service
systemctl lov tes taw cpupower.service
5.2.4 Kev taw qhia txog kev tswj fais fab ntxiv
Cov ntsiab lus ntxiv muaj nyob rau hauv qib siab tshaj noview ntawm ntau lub zog tswj cov yam ntxwv hauv 3rd Generation Intel® Xeon® Scalable processors, nrog rau kev taw qhia txog yuav ua li cas cov yam ntxwv no tuaj yeem koom ua ke ntawm qib platform: https://networkbuilders.intel.com/solutionslibrary/power-management-technologyoverview-technology-guide
5.3 Intel® Turbo Boost
Intel® Turbo Boost ua rau lub processor sai dua thaum xav tau tab sis tuaj yeem siv lub zog ntxiv. Tig tawm Turbo Boost ua rau lub processor ntawm qhov nrawm nrawm, muab koj qib kev ua tau zoo ib yam rau cov haujlwm tshwj xeeb.
5.4 firewalls
Firewalls tuaj yeem cuam tshuam kev ua haujlwm, tshwj xeeb tshaj yog kev ua haujlwm latency.
Disable iptables / firewalld yog tias tsis xav tau.
5.5 Daim ntawv thov chaw
Feem ntau ib leeg xov (uas sib haum mus rau ib lub network queue) tsis txaus kom ua tiav qhov siab tshaj plaws bandwidth. Qee lub platform architectures, xws li AMD, zoo li yuav poob ntau Rx pob ntawv nrog ib qho xov piv rau cov platforms nrog Intel-based processors.
Xav txog kev siv cov cuab yeej xws li taskset lossis numactl los pin daim ntawv thov rau NUMA node lossis CPU cores hauv zos rau lub network ntaus ntawv. Rau qee qhov kev ua haujlwm xws li cia I / O, txav daim ntawv thov mus rau qhov chaw tsis nyob hauv zos muab txiaj ntsig.
Sim ua kom cov xov tooj siv los ntawm koj daim ntawv thov yog tias ua tau.
5.6 Kernel Version
Cov kernel niaj hnub no feem ntau yog qhov tsim nyog rau kev ua tau zoo tab sis, nyob ntawm koj cov ntaub ntawv siv, kev hloov kho cov kernel tuaj yeem muab kev ua tau zoo dua. Kev rub tawm lub hauv paus tseem ua rau koj tuaj yeem qhib / lov tes taw qee yam ua ntej tsim lub kernel.
5.7 Operating System/Kernel Settings
Tham txog kev khiav hauj lwm qhov system tuning guides, xws li Red Hat Enterprise Linux Network Performance Tuning Guide, kom pom ntau ntxiv txog kev khiav hauj lwm qhov nruab nrab.
Qee qhov tsis zoo rau kev qhib tau teev nyob rau hauv cov lus hauv qab no. Nco ntsoov tias cov no tsuas yog pom zoo pib cov ntsiab lus, thiab hloov lawv los ntawm qhov tsis raug yuav ua rau kom cov peev txheej siv rau hauv lub system. Txawm hais tias kev nce nqi tuaj yeem pab txhim kho kev ua tau zoo, nws yog ib qho tsim nyog yuav tsum tau sim nrog cov txiaj ntsig sib txawv los txiav txim siab seb qhov twg ua haujlwm tau zoo tshaj plaws rau lub cev, kev ua haujlwm thiab hom tsheb.
Cov ntsiab lus tsis yog configurable siv sysctl utility hauv Linux raws li qhia hauv qab no.
Rau view lub default values ​​rau rmem thiab wmem ntawm lub system:
sysctl net.core.rmem_default
sysctl net.core.wmem_default
Teem cov nqi rau max (16 MB):
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
Qhov ntau thiab tsawg ntawm lub qhov (socket buffer) , tseem hu ua txais tsis tau (rmem) thiab kis tsis tau (wmem), yog cov kab ke uas qhia meej txog tus nqi ntawm lub cim xeeb tseg rau kev nkag thiab tawm hauv lub network.
Khiav sysctl yam tsis muaj kev sib cav -w teev cov parameter nrog nws qhov chaw tam sim no.

Kev teeb tsa pawg Kev piav qhia
net.core.rmem_default Default Txais Qhov rai Loj
net.core.wmem_default Default Transmit Window Size
net.core.rmem_max Maximum Txais Qhov Loj
net.core.wmem_max Maximum Transmit Window Size
net.core.optmem_max Maximum Option Memory Buffers
net.core.netdev_max_backlog Backlog ntawm cov pob ntawv uas tsis tau ua ua ntej kernel pib poob
net.ipv4.tcp_rmem Nco tseg rau TCP nyeem buffers
net.ipv4.tcp_wmem Nco tseg rau TCP xa buffers

Kernel, pawg network, lub cim xeeb tuav, CPU ceev, thiab kev tswj fais fab tuaj yeem cuam tshuam loj rau kev ua haujlwm hauv network. Ib qho kev pom zoo yog siv rau lub network throughput profile siv cov lus txib tuned. Qhov no hloov qee qhov OS nqis los muab kev nyiam rau kev sib txuas lus.
Tshawb xyuas:
tuned-adm nquag
Teeb:
tuned-adm profile network-thooj
5.8 Network Device Backlog
Cov yam ntxwv no pab txhim kho kev ua haujlwm hauv lub network los ntawm kev tswj cov tsheb khiav tau zoo, txo cov pob ntawv poob, txo qis latency, thiab txhawb nqa cov khoom siv. Qhov no ua rau cov neeg siv kev paub zoo dua thiab cov lus teb sai dua.
Los ntawm lub neej ntawd, nws tau qhib rau hauv feem ntau ntawm Linux operating systems. Txhawm rau txheeb xyuas tus nqi pib:
sysctl net.core.netdev_max_backlog
Tus nqi siab tshaj plaws rau netdev_max_backlog tuaj yeem sib txawv nyob ntawm seb yam xws li ntsiav version, kho vajtse, nco, thiab ua haujlwm. Feem ntau, 8192 tau pom tias yog tus nqi zoo. sysctl -w net.core.netdev_max_backlog=8192
5.9 Platform-Specific Configurations thiab Tuning
5.9.1 4th Generation Intel® Xeon® Scalable Processors
Kev tswj fais fab ntawm Intel® 4th Generation Intel® Xeon® Scalable processor yog qhov hnyav heev piv rau 3rd Generation Intel® Xeon® Scalable processors. Txhawm rau zam cov cores los ntawm kev nkag mus rau cov xeev qis, sim txo cov cores hauv kev siv kom lawv tsaug zog ntev dua.
Pom zoo Bios nqis rau qhov ua tau zoo tshaj plaws

  1. Hyper-threading enable / disable (raws li qhov yuav tsum tau ua haujlwm thiab cov hom phiaj ua haujlwm) ntawm CPU.
  2. Teem lub system profile mus rau Performance rau lub siab tshaj plaws kev ua tau zoo.
    Nco tseg
    Qhov no ua rau muaj kev siv zog ntau dua
  3. Teem lub CPU fais fab kev tswj kom siab tshaj plaws kom ua tau qhov tseem ceeb tshaj plaws CPU kev ua tau zoo tshaj lub zog ua haujlwm.
  4. Qhib Turbo Boost. Disabling Turbo Boost nyob rau hauv lub system BIOS nqis feem ntau tiv thaiv CPU los ntawm dynamically nce nws moos ceev tshaj nws lub hauv paus zaus.
  5. Nco tseg
    Disabling Turbo Boost tej zaum yuav tsim nyog rau qee qhov kev siv uas qhov kev ua tau zoo sib xws, kev siv hluav taws xob, lossis kev tswj xyuas thermal yog qhov tseem ceeb tshaj qhov ua tau zoo tshaj plaws.
  6. Tua ib lub hauv paus I/O Virtualization (SR-IOV), yog tias lub kaw lus tsis siv cov thev naus laus zis virtualization.
  7. Disable C-states kom qhia CPU kom nyob twj ywm thiab tiv thaiv kom tsis txhob nkag mus tob rau hauv lub xeev tsis ua haujlwm.
  8. Disable C1E, kom ntseeg tau tias CPU tseem ua haujlwm thiab tsis nkag mus rau C1E tsis ua haujlwm.
  9. Teem lub uncore zaus mus rau qhov siab tshaj plaws los qhia lub kaw lus ua haujlwm ntawm qhov muaj ntau zaus.
  10. Ntawm Dell platforms, teem ntau APIC Cov lus piav qhia (MADT) core emulation rau Linear (lossis Round-Robin nyob ntawm BIOS) los muab qhov pom tseeb thiab pom tau ntawm CPU cores.

Pom zoo OS Level Tunings rau optimized Performance

  1. Teem CPU zaus scaling tswv xeev rau kev ua haujlwm. cpupower zaus-set -g kev ua haujlwm cpupower zaus-info
  2. Disable C-States. cpupower idle-set -D0
  3. Teem core Rx (rmem) thiab Tx (wmem) buffers rau qhov siab tshaj plaws. sysctl -w net.core.rmem_max=16777216 sysctl -w net.core.wmem_max=16777216
  4. Teem lub network ntaus ntawv backlog. sysctl -w net.core.netdev_max_backlog=8192
  5. Teem tuned profile (kev ua haujlwm nyob ntawm qhov kev xa tawm / latency).
    tuned-adm profile network-thooj

Pom zoo Adapter Level Tunings rau Optimized Performance

  1. Txwv tus naj npawb ntawm cov kab los siv rau kev thov tsheb. Siv qhov tsawg kawg nkaus ntawm cov kab uas yuav tsum tau ua kom cov CPU cores cuam tshuam los tiv thaiv lawv los ntawm kev nkag mus tob rau hauv lub xeev tsis ua haujlwm (hloov rau cov haujlwm ua haujlwm): ethtool -L sib 32
  2. Teem interrupt moderation rates. ethtool - C adaptive-rx off adaptive-tx off rx-usecs-high 50 rx-usecs 50 tx-usecs 50
    Sim kho qhov kev xa tawm / txais / qhov tseem ceeb tshaj plaws coalescing timer siab dua (80/100/150/200) lossis qis dua (25/20/10/5) kom pom tus nqi zoo rau kev ua haujlwm.
  3. Teem Rx/Tx lub nplhaib loj. ethtool - G tx4096 4096
    Nco tseg
    Yog tias koj pom Rx pob ntawv poob nrog ethtool -S| grep poob, sim txo lub nplhaib loj rau <4096. Sim nrhiav tus nqi zoo tshaj plaws rau kev ua haujlwm uas cov pob ntawv tsis poob.
  4. Teem IRQ Affinity. Siv cov cores hauv zos rau NIC, lossis cov ntawv qhia tshwj xeeb (qhov twg # cores sib npaug rau cov lej ntawm cov kab hauv 1 ntawm nplooj ntawv 26. systemctl nres irqbalance set_irq_affinity -X hauv zos LOSSIS set_irq_affinity -X

5.9.2 AMD EPYC
AMD EPYC processors yog cov muaj zog CPUs tsim rau cov servers thiab cov chaw khaws ntaub ntawv, tsim los ntawm AMD's Zen architecture. Cov kev teeb tsa hauv qab no yog los ntawm AMD's 4th tiam EPYC series.
Pom zoo BIOS nqis rau qhov ua tau zoo tshaj plaws

  1. Qhib hom kev cai tso cai rau cov neeg siv los kho CPU kev ua tau zoo, kev siv hluav taws xob, thiab lwm qhov chaw. Qhov no yuav pab tau zoo-tuning system rau qhov zoo tshaj plaws sib npaug ntawm kev ua tau zoo thiab lub zog efficiency.
  2. Pab txhawb kev ua tau zoo ntawm cov tub ntxhais kawm kom tso cai rau CPU tuaj yeem nce nws qhov nrawm los ua haujlwm hnyav dua, txhim kho kev ua haujlwm tag nrho.
  3. Disable thoob ntiaj teb C-xeev tswj, tiv thaiv CPU los ntawm kev nkag mus tob zog txuag lub xeev hu ua C-states, uas muaj peev xwm tswj tau teb.
    Nco tseg
    Disabling C-states tuaj yeem ua rau muaj kev siv hluav taws xob ntxiv thiab ua kom sov sov. Saib xyuas ob qho tib si rau kev ua haujlwm.
  4. Qhib / lov tes taw Simultaneous Multithreading (SMT) ntawm CPU, raws li qhov xav tau ntawm kev ua haujlwm thiab cov hom phiaj ua haujlwm. SMT yog sib npaug rau Hyper Threading ntawm Intel CPUs.
    Nco tseg
    Rau kev ua tau zoo, xa mus rau Tuning i40e Driver Settings ntawm nplooj 13 thiab Platform Tuning (i40e Non-Specific) nyob rau nplooj 19 rau qhov pom zoo OS thiab adapter qib tuning.

Adapter Bonding

Linux kev sib raug zoo yog qhov muaj txiaj ntsig zoo uas tuaj yeem txhim kho lub network kev ua tau zoo, rov ua dua, thiab ua txhaum kev zam txim hauv server ib puag ncig. Txawm li cas los xij, nws yog ib qho tseem ceeb uas yuav tsum nco ntsoov tias nws yuav tsum tau siv cov khoom siv sib txuas sib txuas thiab kev teeb tsa kom raug ntawm ob lub server thiab hloov mus ua haujlwm kom raug.
Tus tsav tsheb sib txuas hauv Linux tso cai rau koj los sau ntau qhov sib txuas ntawm lub cev sib txuas rau hauv qhov sib txuas sib txuas. Qhov kev sib txuas sib txuas no tshwm sim raws li ib qho kev sib txuas virtual network rau kev khiav hauj lwm thiab kev siv.
Nco tseg
Daim ntawv cog lus yog qhov kev sib txuas lus zoo, yog li nws tsis tuaj yeem teeb tsa CPU affinity ncaj qha rau ntawm daim ntawv cog lus interface (rau example, bond0). Ntawd yog, nws tsis muaj kev tswj ncaj qha los cuam tshuam kev tuav lossis CPU affinity. CPU affinity yuav tsum tau teeb tsa rau hauv qab interfaces uas yog ib feem ntawm daim ntawv cog lus.
Bonding muab ntau hom kev ua haujlwm, txhua tus muaj nws tus yam ntxwv.

Hom  Hom
0 Round Robin
1 Active Backup
2 XOR
3 Tshaj tawm
4 LACP
5 Hloov Load Balance
6 Adaptive Load Balance

Muaj ntau txoj hauv kev los tsim kev sib txuas lus hauv Linux. Ib txoj hauv kev zoo tshaj plaws yog los ntawm kev siv lub network teeb tsa files (example, /etc/network/interfaces or /etc/sysconfig/network-scripts/ifcfg-bondX).
Configuration Siv Network Configuration Files
Cov kauj ruam hauv qab no tsim kev sib txuas los ntawm kev teeb tsa lub network files.

  1. Xaiv ob lossis ntau qhov chaw nres nkoj NIC rau kev sib txuas (example, ethX and ethY)
  2. Qhib NIC Configuration Files hauv qab /etc/sysconfig/network-scripts/ rau qhov xav tau NIC Interface (rau example, vi ifcfg-ethX thiab vi ifcfg-ethY) thiab ntxiv cov ntawv hauv qab no:
    MASTER=bondN [Ceeb Toom: N yog tus lej los hais txog tus lej nyiaj.] SLAVE = yog
  3. Tsim ib daim ntawv cog lus network file siv vi /etc/sysconfig/networkscripts/ifcfg-bondN thiab sau cov ntawv hauv qab no:
    DEVICE=bondN [Nco tseg: N yog tus lej los hais txog tus lej nyiaj] ONBOOT = yog USERCTL = tsis muaj BOOTPROTO = dhcp (lossis) tsis muaj
    IPADDR=200.20.2.4 [yuav tsum tau yog BOOTPROTO=none] NETMASK=255.255.255.0 [yuav tsum tau yog BOOTPROTO=none] NETWORK=200.20.2.0 [yuav tsum tau yog BOOTPROTO=none] BROADCAST=200.20.2.255. BOOTPROTO=none] BONDING_OPTS =”mode=1 miimon=100″
    Nco tseg
    Hom tuaj yeem yog tus lej ntawm 0 txog 6 raws li qhov yuav tsum tau ua.
  4. Rov pib cov kev pabcuam network siv kev pabcuam network rov pib dua lossis systemctl rov pib NetworkManager.service

Kev daws teeb meem kev ua haujlwm

7.1 Kev siv CPU
Tshawb xyuas CPU siv ib qho tseem ceeb thaum lub sijhawm ua haujlwm.
Nco ntsoov tias kev siv ib qho tseem ceeb muaj feem cuam tshuam rau kev ua tau zoo dua li kev siv CPU tag nrho vim nws muab lub tswv yim ntawm kev siv CPU rau ib lub network kab. Yog tias koj tsuas muaj ob peb lub xov tooj khiav network khiav, ces koj tsuas yog siv ob peb cores xwb. Txawm li cas los xij, yog tias cov cores no nyob ntawm 100%, ces koj lub network throughput yuav raug txwv los ntawm kev siv CPU thiab nws yog lub sijhawm los ua cov hauv qab no:

  1. Tune IRQ moderation / nplhaib loj raws li cov ncauj lus kom ntxaws hauv Interrupt Moderation.
  2. Nce tus naj npawb ntawm daim ntawv thov xov kom nthuav tawm CPU thauj khoom ntau dua cores. Yog tias tag nrho cov cores khiav ntawm 100% ces koj daim ntawv thov yuav raug CPU khi es tsis yog network khi.

Cov cuab yeej siv tau yooj yim:

  • saum
    - Nias 1 kom nthuav cov npe ntawm CPUs thiab xyuas seb cov twg tau siv.
    - Pom qhov theem ntawm kev siv.
    - Daim ntawv ceeb toom cov txheej txheem twg tau teev tseg ua haujlwm tshaj plaws (sab saum toj ntawm daim ntawv teev npe).
  • mpstat
    Cov nram qab no example kab hais kom ua tau sim ntawm Red Hat Enterprise Linux 7.x.
    Nws qhia txog kev siv CPU rau ib tus tub ntxhais kawm (los ntawm kev nrhiav tag nrho cov feem pua ​​​​tsis ua haujlwm thiab rho tawm los ntawm 100) thiab qhia txog qhov tseem ceeb tshaj 80% liab. mpstat -P ALL 1 1 | grep -v nruab nrab | tail -n +5 | taub hau -n -1 | awk '{ print (100-$13)}' | egrep -color=always '[^\.][8-9][0-9][\.]?.*|^[8-9][0-9][\.]?.*| 100 |' | kem
  • perf sab saum toj Saib rau qhov twg cycles raug siv.

7.2i40e ib
Tus tsav tsheb i40e muab cov npe ntev ntawm cov txee rau kev sib tham debug thiab saib xyuas los ntawm ethtool -S ethX hais kom ua. Nws tuaj yeem pab tau saib cov zis thaum lub sijhawm ua haujlwm thiab / lossis sib piv cov nqi suav ua ntej thiab tom qab ua haujlwm.

  • Kom tau txais tag nrho cov pov tseg ntawm i40e cov txee: ethtool -S ethX
  • Mus saib xwb tsis-zero suav: saib -d (ethtool -S ethX) | egrep -v :\ 0 | kem
    Qee yam uas yuav nrhiav tau:
  • rx_dropped txhais tau hais tias CPU tsis ua haujlwm buffers nrawm txaus.
  • port.rx_dropped txhais tau hais tias ib yam dab tsi tsis ceev txaus hauv lub qhov / nco / system.

7.3 Network Counters
Txheeb xyuas netstat -s ua ntej / tom qab ua haujlwm khiav.
Netstat sau cov ntaub ntawv network los ntawm tag nrho cov khoom siv network hauv lub system. Yog li, cov txiaj ntsig tuaj yeem cuam tshuam los ntawm cov tes hauj lwm uas tsis yog lub network hauv kev sim. Cov zis los ntawm netstat -s tuaj yeem yog qhov qhia tau zoo ntawm cov teeb meem kev ua haujlwm hauv Linux operating system lossis ntsiav. Tham txog kev khiav hauj lwm qhov system tuning guides, xws li Red Hat Enterprise Linux Network Performance Tuning Guide, kom pom ntau ntxiv txog kev khiav hauj lwm qhov nruab nrab.
7.4 System Logs
Kos lub kaw lus rau kev ua yuam kev thiab ceeb toom (/var/log/messages, dmesg).
7.5 Intel svr-info Tool
Intel muab cov cuab yeej svr-info (saib https://github.com/intel/svr-info) rau Linux uas ntes cov khoom lag luam thiab cov ntsiab lus ntawm software los ntawm lub server. svr-cov ntaub ntawv tso tawm tuaj yeem pab tau zoo heev los txheeb xyuas cov kev tsis sib haum xeeb lossis kev teeb tsa / tunings uas tsis zoo rau kev ua haujlwm. Thaum qhib rooj plaub kev txhawb nqa nrog Intel rau Ethernet-hais txog kev ua haujlwm teeb meem, nco ntsoov suav nrog svr-info tso tawm (cov ntawv nyeem file) rau txhua tus neeg rau zaub mov Linux hauv kev sim teeb tsa.

  1. Download tau thiab nruab svr-info:
    wget -qO- https://github.com/intel/svr-info/releases/latest/download/svrinfo.tgz| tar xvz cd svr-info
    ./svr-info
    > hostname.txt
  2. Sau cov zis:
    ./svr-info > hostname.txt
  3. Txuas ib phau ntawv (.txt) file rau txhua tus neeg rau zaub mov rau koj cov ntaub ntawv txhawb nqa Intel rau kev tshuaj xyuas.

Cov lus pom zoo rau Kev Ua Haujlwm Zoo Tshaj Plaws

8.1 Kev xa tawm IP

  • Hloov kho lub kernel.
    Qee qhov tsis ntev los no hauv-distro kernels tau degraded routing kev ua tau zoo vim yog cov hloov pauv hauv cov kab ke pib pib nrog kev tshem tawm ntawm cov routing cache vim kev ruaj ntseg. Tsis ntev los no tawm-ntawm-distro kernels yuav tsum muaj thaj ua rau thaj uas txo qhov kev ua tau zoo ntawm cov kev hloov pauv no thiab tuaj yeem muab kev ua tau zoo dua qub.
  • Disable hyper-threading (logical cores).
  • Kho cov kernel boot parameters.
    - Force iommu tawm (intel_iommu = tawm lossis iommu = tawm) los ntawm cov kab khau raj ntsiav tshwj tsis yog xav tau rau virtualization
    - Tua hluav taws xob tswj: processor.max_cstates=1 idle=poll pcie_aspm=off
  • Txwv tus naj npawb ntawm cov kab kom sib npaug ntawm cov cores ntawm lub qhov (socket) hauv zos (12 hauv qhov example). ethtool -L ethX ua ke 12
  • Pin cuam tshuam rau lub qhov (socket) hauv zos nkaus xwb. set_irq_affinity -X local ethX OR set_irq_affinity -X local ethX
    Nco tseg
    -X lossis -x tuaj yeem siv nyob ntawm kev ua haujlwm.
  • Hloov cov Tx thiab Rx nplhaib qhov ntau thiab tsawg raws li xav tau. Tus nqi loj dua yuav siv ntau dua tab sis tuaj yeem muab cov nqi xa mus zoo dua. ethtool -G ethX rx 4096 tx 4096
  • Disable GRO thaum routing.
    Vim muaj qhov paub txog qhov teeb meem, GRO yuav tsum raug muab tua thaum routing/forwarding. ethtool -K ethX gro tawm qhov twg ethX yog Ethernet interface los hloov kho.
  • Disable adaptive interrupt moderation thiab teeb tus nqi zoo li qub. ethtool -C ethX adaptive-rx off adaptive-tx off ethtool -C ethX rx-usecs 64 tx-usecs 64

Nco tseg
Nyob ntawm seb hom processor thiab kev ua haujlwm, cov kev sib koom ua ke rau RX thiab TX tuaj yeem hloov kho kom ua tau zoo dua qub (lossis tsawg dua ncej).

  • Disable lub firewall. sudo systemctl lov tes taw firewalld sudo systemctl nres firewalld
  • Pab kom IP xa mus. sysctl -w net.ipv4.ip_forward=1
  • Configure qhov siab tshaj plaws rau qhov tau txais thiab xa cov qhov (socket) tsis ntau thiab tsawg. sysctl -w net.core.rmem_max=16777216 sysctl -w net.core.wmem_max=16777216

Nco tseg
Nyob ntawm qhov ua haujlwm lossis qhov yuav tsum tau ua, cov txiaj ntsig no tuaj yeem hloov pauv los ntawm lub neej ntawd.
8.2 Tsawg latency

  • Tig hyper-threading (logical cores) OFF.
  • Xyuas kom lub network ntaus ntawv nyob hauv zos rau numa core 0.
  • Pin tus qauv ntsuas rau core 0 siv taskset -c 0.
  • Tig irqbalance tawm siv systemctl nres irqbalance lossis systemctl lov tes taw irqbalance
  • Khiav tus affinity tsab ntawv kom kis thoob plaws cores. Sim hauv zos lossis tag nrho.
  • Tshem tawm kev cuam tshuam kev ua haujlwm. ethtool -C ethX rx-usecs 0 tx-usecs 0 adaptive-rx off adaptive-tx off rxusecs- siab 0
  • Txwv tus naj npawb ntawm cov kab kom sib npaug ntawm cov cores ntawm lub qhov (socket) hauv zos (32 hauv qhov example). ethtool -L ethX ua ke 32
  • Pin cuam tshuam rau lub qhov (socket) hauv zos nkaus xwb (ntawv ntim nrog i40e tsav tsheb qhov chaw). set_irq_affinity -X local ethX
  • Siv tus qauv tsim muaj xws li netperf -t TCP_RR, netperf -t UDP_RR, lossis NetPipe. netperf -t TCP_RR los yog netperf -t UDP_RR
  • Pin benchmark rau ib qho tseem ceeb hauv NUMA node hauv zos. Taskset -c

Intel ® Ethernet 700 Series
Linux Performance Tuning Guide
Kaum Ob Hlis 2024
Doc. No.: 334019, Rev.: 1.2

Cov ntaub ntawv / Cov ntaub ntawv

Intel Ethernet 700 Series Linux Performance Tuning [ua pdf] Cov neeg siv phau ntawv qhia
334019, Ethernet 700 Series Linux Performance Tuning, Ethernet 700 Series, Linux Performance Tuning, Performance Tuning, Tuning

Cov ntaub ntawv

Cia ib saib

Koj email chaw nyob yuav tsis raug luam tawm. Cov teb uas yuav tsum tau muaj yog cim *