From 31983a04d686f9f90b356072089d8d677e40e776 Mon Sep 17 00:00:00 2001 From: Mauro Carvalho Chehab Date: Wed, 5 Aug 2009 21:16:56 -0300 Subject: Documentation/edac.txt: Add Nehalem specific EDAC characteristics As Nehalem has a different binding to EDAC API, and its own different error injection code, documents it. Signed-off-by: Mauro Carvalho Chehab --- Documentation/edac.txt | 110 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) (limited to 'Documentation') diff --git a/Documentation/edac.txt b/Documentation/edac.txt index 79c533223762..8bc320467c64 100644 --- a/Documentation/edac.txt +++ b/Documentation/edac.txt @@ -6,6 +6,8 @@ Written by Doug Thompson 7 Dec 2005 17 Jul 2007 Updated +(c) Mauro Carvalho Chehab +05 Aug 2009 Nehalem interface EDAC is maintained and written by: @@ -717,3 +719,111 @@ unique drivers for their hardware systems. The 'test_device_edac' sample driver is located at the bluesmoke.sourceforge.net project site for EDAC. +======================================================================= +NEHALEM USAGE OF EDAC APIs + +This chapter documents some EXPERIMENTAL mappings for EDAC API to handle +Nehalem EDAC driver. They will likely be changed on future versions +of the driver. + +Due to the way Nehalem exports Memory Controller data, some adjustments +were done at i7core_edac driver. This chapter will cover those differences + +1) On Nehalem, there are one Memory Controller per Quick Patch Interconnect + (QPI). At the driver, the term "socket" means one QPI. It should also be + associated with the CPU physical socket. + + Each MC have 3 physical read channels, 3 physical write channels and + 3 logic channels. The driver currenty sees it as just 3 channels. + Each channel can have up to 3 DIMMs. + + The minimum known unity is DIMMs. There are no information about csrows. + As EDAC API maps the minimum unity is csrows, the driver exports one + DIMM per csrow. + + Currently, it also exports the several memory controllers as just one. This + limit will be removed on future versions of the driver. + +2) Nehalem MC has the hability to generate errors. The driver implements this + functionality via some error injection nodes: + + For injecting a memory error, there are some sysfs nodes, under + /sys/devices/system/edac/mc/mc0/: + + inject_addrmatch: + Controls the error injection mask register. It is possible to specify + several characteristics of the address to match an error code: + dimm = the affected dimm. Numbers are relative to a channel; + rank = the memory rank; + channel = the channel that will generate an error; + bank = the affected bank; + page = the page address; + column (or col) = the address column. + each of the above values can be set to "any" to match any valid value. + + At driver init, all values are set to any. + + For example, to generate an error at rank 1 of dimm 2, for any channel, + any bank, any page, any column: + echo "dimm:2 rank:1" >/sys/devices/system/edac/mc/mc0/inject_addrmatch + + To return to the default behaviour of matching any, you can do: + echo "dimm:any rank:any" >/sys/devices/system/edac/mc/mc0/inject_addrmatch + + inject_eccmask: + specifies what bits will have troubles, + + inject_section: + specifies what ECC cache section will get the error: + 3 for both + 2 for the highest + 1 for the lowest + + inject_socket: + specifies what QPI (or processor socket) will generate the error. + on Xeon 35xx, it should be 0. + on Xeon 55xx, it should be 0 or 1. + + inject_type: + specifies the type of error, being a combination of the following bits: + bit 0 - repeat + bit 1 - ecc + bit 2 - parity + + inject_enable starts the error generation when something different + than 0 is written. + + All inject vars can be read. root permission is needed for write. + + Datasheet states that the error will only be generated after a write on an + address that matches inject_addrmatch. It seems, however, that reading will + also produce an error. + + For example, the following code will generate an error for any write access + at socket 0, on any DIMM/address on channel 2: + + echo "channel:2" > /sys/devices/system/edac/mc/mc0/inject_addrmatch + echo 2 >/sys/devices/system/edac/mc/mc0/inject_type + echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask + echo 3 >/sys/devices/system/edac/mc/mc0/inject_section + echo 0 >/sys/devices/system/edac/mc/mc0/inject_socket + echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable + dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null + + The generated error message will look like: + + EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error)) + +3) Nehalem specific Corrected Error memory counters + + Nehalem have some registers to count memory errors, reporting it on a + way that it is different from what EDAC API allows. Due to that, a + separate sysfs note were created to handle such counters. + + They can be read by looking at the contents of "corrected_error_counts" + counter: + + $ cat /sys/devices/system/edac/mc/mc0/corrected_error_counts + dimm0: 15866 + dimm1: 0 + dimm2: 27285 -- cgit v1.2.3 From c344436319e898784febbeeea71d1b0f65ef53ae Mon Sep 17 00:00:00 2001 From: Mauro Carvalho Chehab Date: Sat, 5 Sep 2009 05:10:15 -0300 Subject: Documentation/edac.txt: Improve it to reflect the latest changes at the driver Signed-off-by: Mauro Carvalho Chehab --- Documentation/edac.txt | 72 +++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 16 deletions(-) (limited to 'Documentation') diff --git a/Documentation/edac.txt b/Documentation/edac.txt index 8bc320467c64..bd3f8a3905af 100644 --- a/Documentation/edac.txt +++ b/Documentation/edac.txt @@ -730,25 +730,41 @@ Due to the way Nehalem exports Memory Controller data, some adjustments were done at i7core_edac driver. This chapter will cover those differences 1) On Nehalem, there are one Memory Controller per Quick Patch Interconnect - (QPI). At the driver, the term "socket" means one QPI. It should also be - associated with the CPU physical socket. + (QPI). At the driver, the term "socket" means one QPI. This is + associated with a physical CPU socket. Each MC have 3 physical read channels, 3 physical write channels and 3 logic channels. The driver currenty sees it as just 3 channels. Each channel can have up to 3 DIMMs. The minimum known unity is DIMMs. There are no information about csrows. - As EDAC API maps the minimum unity is csrows, the driver exports one + As EDAC API maps the minimum unity is csrows, the driver sequencially + maps channel/dimm into different csrows. + + For example, suposing the following layout: + Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs + dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 + dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400 + Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs + dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 + Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs + dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400 + The driver will map it as: + csrow0: channel 0, dimm0 + csrow1: channel 0, dimm1 + csrow2: channel 1, dimm0 + csrow3: channel 2, dimm0 + +exports one DIMM per csrow. - Currently, it also exports the several memory controllers as just one. This - limit will be removed on future versions of the driver. + Each QPI is exported as a different memory controller. 2) Nehalem MC has the hability to generate errors. The driver implements this functionality via some error injection nodes: For injecting a memory error, there are some sysfs nodes, under - /sys/devices/system/edac/mc/mc0/: + /sys/devices/system/edac/mc/mc?/: inject_addrmatch: Controls the error injection mask register. It is possible to specify @@ -779,11 +795,6 @@ were done at i7core_edac driver. This chapter will cover those differences 2 for the highest 1 for the lowest - inject_socket: - specifies what QPI (or processor socket) will generate the error. - on Xeon 35xx, it should be 0. - on Xeon 55xx, it should be 0 or 1. - inject_type: specifies the type of error, being a combination of the following bits: bit 0 - repeat @@ -806,10 +817,12 @@ were done at i7core_edac driver. This chapter will cover those differences echo 2 >/sys/devices/system/edac/mc/mc0/inject_type echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask echo 3 >/sys/devices/system/edac/mc/mc0/inject_section - echo 0 >/sys/devices/system/edac/mc/mc0/inject_socket echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null + For socket 1, it is needed to replace "mc0" by "mc1" at the above + commands. + The generated error message will look like: EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error)) @@ -821,9 +834,36 @@ were done at i7core_edac driver. This chapter will cover those differences separate sysfs note were created to handle such counters. They can be read by looking at the contents of "corrected_error_counts" - counter: + counter. Due to hardware limits, the output is different on machines + with unregistered memories and machines with registered ones. + + With unregistered memories, it outputs: $ cat /sys/devices/system/edac/mc/mc0/corrected_error_counts - dimm0: 15866 - dimm1: 0 - dimm2: 27285 + all channels UDIMM0: 0 UDIMM1: 0 UDIMM2: 0 + + What happens here is that errors on different csrows, but at the same + dimm number will increment the same counter. + So, in this memory mapping: + csrow0: channel 0, dimm0 + csrow1: channel 0, dimm1 + csrow2: channel 1, dimm0 + csrow3: channel 2, dimm0 + The hardware will increment UDIMM0 for an error at either csrow0, csrow2 + or csrow3. + + With registered memories, it outputs: + + $cat /sys/devices/system/edac/mc/mc0/corrected_error_counts + channel 0 RDIMM0: 0 RDIMM1: 0 RDIMM2: 0 + channel 1 RDIMM0: 0 RDIMM1: 0 RDIMM2: 0 + channel 2 RDIMM0: 0 RDIMM1: 0 RDIMM2: 0 + + So, with registered memories, there's a direct map between a csrow and a + counter. + +4) Standard error counters + + The standard error counters are generated when an mcelog error is received + by the driver. Since it is counted by software, it is possible that some + errors could be lost. -- cgit v1.2.3 From 35be95446734cbb10b088a6b38269ac4a8ac3a86 Mon Sep 17 00:00:00 2001 From: Mauro Carvalho Chehab Date: Thu, 24 Sep 2009 17:28:50 -0300 Subject: Documentation/edac.txt: Reflect the sysfs changes at the document Signed-off-by: Mauro Carvalho Chehab --- Documentation/edac.txt | 56 ++++++++++++++++++++++++++------------------------ 1 file changed, 29 insertions(+), 27 deletions(-) (limited to 'Documentation') diff --git a/Documentation/edac.txt b/Documentation/edac.txt index bd3f8a3905af..0b875e8da969 100644 --- a/Documentation/edac.txt +++ b/Documentation/edac.txt @@ -766,7 +766,7 @@ exports one For injecting a memory error, there are some sysfs nodes, under /sys/devices/system/edac/mc/mc?/: - inject_addrmatch: + inject_addrmatch/*: Controls the error injection mask register. It is possible to specify several characteristics of the address to match an error code: dimm = the affected dimm. Numbers are relative to a channel; @@ -781,10 +781,12 @@ exports one For example, to generate an error at rank 1 of dimm 2, for any channel, any bank, any page, any column: - echo "dimm:2 rank:1" >/sys/devices/system/edac/mc/mc0/inject_addrmatch + echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm + echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank To return to the default behaviour of matching any, you can do: - echo "dimm:any rank:any" >/sys/devices/system/edac/mc/mc0/inject_addrmatch + echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm + echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank inject_eccmask: specifies what bits will have troubles, @@ -813,7 +815,7 @@ exports one For example, the following code will generate an error for any write access at socket 0, on any DIMM/address on channel 2: - echo "channel:2" > /sys/devices/system/edac/mc/mc0/inject_addrmatch + echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel echo 2 >/sys/devices/system/edac/mc/mc0/inject_type echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask echo 3 >/sys/devices/system/edac/mc/mc0/inject_section @@ -829,18 +831,23 @@ exports one 3) Nehalem specific Corrected Error memory counters - Nehalem have some registers to count memory errors, reporting it on a - way that it is different from what EDAC API allows. Due to that, a - separate sysfs note were created to handle such counters. + Nehalem have some registers to count memory errors. The driver uses those + registers to report Corrected Errors on devices with Registered Dimms. - They can be read by looking at the contents of "corrected_error_counts" - counter. Due to hardware limits, the output is different on machines - with unregistered memories and machines with registered ones. + However, those counters don't work with Unregistered Dimms. As the chipset + offers some counters that also work with UDIMMS (but with a worse level of + granularity than the default ones), the driver exposes those registers for + UDIMM memories. - With unregistered memories, it outputs: + They can be read by looking at the contents of all_channel_counts/ - $ cat /sys/devices/system/edac/mc/mc0/corrected_error_counts - all channels UDIMM0: 0 UDIMM1: 0 UDIMM2: 0 + $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done + /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0 + 0 + /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1 + 0 + /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2 + 0 What happens here is that errors on different csrows, but at the same dimm number will increment the same counter. @@ -849,21 +856,16 @@ exports one csrow1: channel 0, dimm1 csrow2: channel 1, dimm0 csrow3: channel 2, dimm0 - The hardware will increment UDIMM0 for an error at either csrow0, csrow2 - or csrow3. - - With registered memories, it outputs: - - $cat /sys/devices/system/edac/mc/mc0/corrected_error_counts - channel 0 RDIMM0: 0 RDIMM1: 0 RDIMM2: 0 - channel 1 RDIMM0: 0 RDIMM1: 0 RDIMM2: 0 - channel 2 RDIMM0: 0 RDIMM1: 0 RDIMM2: 0 - - So, with registered memories, there's a direct map between a csrow and a - counter. + The hardware will increment udimm0 for an error at the first dimm at either + csrow0, csrow2 or csrow3; + The hardware will increment udimm1 for an error at the second dimm at either + csrow0, csrow2 or csrow3; + The hardware will increment udimm2 for an error at the third dimm at either + csrow0, csrow2 or csrow3; 4) Standard error counters The standard error counters are generated when an mcelog error is received - by the driver. Since it is counted by software, it is possible that some - errors could be lost. + by the driver. Since, with udimm, this is counted by software, it is + possible that some errors could be lost. With rdimm's, they displays the + contents of the registers -- cgit v1.2.3