From bf56a9edfcca610bc771e0176f72bbce59fcc87a Mon Sep 17 00:00:00 2001 From: Franck Cuny Date: Thu, 26 Dec 2024 19:01:18 -0800 Subject: large cleanup --- content/stuff-about-pcie.md | 266 -------------------------------------------- 1 file changed, 266 deletions(-) delete mode 100644 content/stuff-about-pcie.md (limited to 'content/stuff-about-pcie.md') diff --git a/content/stuff-about-pcie.md b/content/stuff-about-pcie.md deleted file mode 100644 index 311e55f..0000000 --- a/content/stuff-about-pcie.md +++ /dev/null @@ -1,266 +0,0 @@ -+++ -title = "Stuff about PCIe" -date = 2022-01-03 -[taxonomies] -tags = ["hardware"] -+++ - -## Speed - -The most common versions are 3 and 4, while 5 is starting to be -available with newer Intel processors. - -| ver | encoding | transfer rate | x1 | x2 | x4 | x8 | x16 | -| --- | --------- | ------------- | ---------- | ----------- | ---------- | ---------- | ----------- | -| 1 | 8b/10b | 2.5GT/s | 250MB/s | 500MB/s | 1GB/s | 2GB/s | 4GB/s | -| 2 | 8b/10b | 5.0GT/s | 500MB/s | 1GB/s | 2GB/s | 4GB/s | 8GB/s | -| 3 | 128b/130b | 8.0GT/s | 984.6 MB/s | 1.969 GB/s | 3.94 GB/s | 7.88 GB/s | 15.75 GB/s | -| 4 | 128b/130b | 16.0GT/s | 1969 MB/s | 3.938 GB/s | 7.88 GB/s | 15.75 GB/s | 31.51 GB/s | -| 5 | 128b/130b | 32.0GT/s | 3938 MB/s | 7.877 GB/s | 15.75 GB/s | 31.51 GB/s | 63.02 GB/s | -| 6 | 128b/130 | 64.0 GT/s | 7877 MB/s | 15.754 GB/s | 31.51 GB/s | 63.02 GB/s | 126.03 GB/s | - -This is a -[useful](https://community.mellanox.com/s/article/understanding-pcie-configuration-for-maximum-performance) -link to understand the formula: - - Maximum PCIe Bandwidth = SPEED * WIDTH * (1 - ENCODING) - 1Gb/s - -We remove 1Gb/s for protocol overhead and error corrections. The main -difference between the generations besides the supported speed is the -encoding overhead of the packet. For generations 1 and 2, each packet -sent on the PCIe has 20% PCIe headers overhead. This was improved in -generation 3, where the overhead was reduced to 1.5% (2/130) - see -[8b/10b encoding](https://en.wikipedia.org/wiki/8b/10b_encoding) and -[128b/130b encoding](https://en.wikipedia.org/wiki/64b/66b_encoding). - -If we apply the formula, for a PCIe version 3 device we can expect -3.7GB/s of data transfer rate: - - 8GT/s * 4 lanes * (1 - 2/130) - 1G = 32G * 0.985 - 1G = ~30Gb/s -> 3750MB/s - -## Topology - -An easy way to see the PCIe topology is with `lspci`: - - $ lspci -tv - -[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex - +-01.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge - +-01.1-[01]----00.0 OCZ Technology Group, Inc. RD400/400A SSD - +-01.3-[02-03]----00.0-[03]----00.0 ASPEED Technology, Inc. ASPEED Graphics Family - +-01.5-[04]--+-00.0 Intel Corporation I350 Gigabit Network Connection - | +-00.1 Intel Corporation I350 Gigabit Network Connection - | +-00.2 Intel Corporation I350 Gigabit Network Connection - | \-00.3 Intel Corporation I350 Gigabit Network Connection - +-02.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge - +-03.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge - +-04.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge - +-07.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge - +-07.1-[05]--+-00.0 Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function - | +-00.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor - | \-00.3 Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller - +-08.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge - +-08.1-[06]--+-00.0 Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function - | +-00.1 Advanced Micro Devices, Inc. [AMD] Zeppelin Cryptographic Coprocessor NTBCCP - | +-00.2 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] - | \-00.3 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller - +-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller - +-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge - +-18.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 - +-18.1 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 - +-18.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 - +-18.3 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 - +-18.4 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 - +-18.5 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 - +-18.6 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 - \-18.7 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 - -Now, how do we read this ? - -``` -+-[10000:00]-+-02.0-[01]----00.0 Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] -| \-03.0-[02]----00.0 Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] -``` - -This is a lot of information, how do we read this ? - -- The first part in brackets (`[10000:00]`) is the domain and the bus. -- The second part (`02.0` is still unclear to me) -- The third number (between brackets) is the device on the bus - -## View a single device - -```sh -lspci -v -s 0000:01:00.0 -: 01:00.0 Non-Volatile memory controller: OCZ Technology Group, Inc. RD400/400A SSD (rev 01) (prog-if 02 [NVM Express]) -: Subsystem: OCZ Technology Group, Inc. RD400/400A SSD -: Flags: bus master, fast devsel, latency 0, IRQ 41, NUMA node 0 -: Memory at ef800000 (64-bit, non-prefetchable) [size=16K] -: Capabilities: -: Kernel driver in use: nvme -: Kernel modules: nvme -``` - -## Reading `lspci` output - - $ sudo lspci -vvv -s 0000:01:00.0 - 01:00.0 Non-Volatile memory controller: OCZ Technology Group, Inc. RD400/400A SSD (rev 01) (prog-if 02 [NVM Express]) - Subsystem: OCZ Technology Group, Inc. RD400/400A SSD - Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ - Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-