MTD driver to host PNOR issue

[cyrilbur-ibm]

I've been testing out the MTD driver.
Minimal testing has uncovered an issue in that reading from /dev/mtd7 (the host PNOR) is unreliable.
$ dd if=/dev/mtd7 of=/tmp/pnor count=1
Appears to work, what looks like an FFS header is in /tmp/pnor. Repeating that command reads all FF.

Didn't reproduce on /dev/mtd0.

[cyrilbur-ibm]

uname -a
Linux palmetto 4.3.4-fe68a785afbaa305506c87c6324f70ed0605385c #1 Mon Feb 8 16:55:24 AWST 2016 armv5tejl GNU/Linux

[mdmillerii]

I haven't tested pnor a lot. I did see something similar once in the last week where the driver would repeatedly read FF.

pflash can bypass the bmc and was able to see data when this occurred. A reboot for me allowed me to read the flash repeatedly and I was not able to recreate the FF even with tunning.

There is still call to pflash to tune the controller that is happening behind the drivers back, and also a write to the controller register of 2404 in the aspeed hard-coded inits. Also the driver does not fully clear the original modes it finds in the controller; any of these may be related.

[shenki]

@cyrilbur-ibm can you please clarify what the issue is?

Please try to reproduce on the latest image.

[mdmillerii]

I was able to recreate this on my latest 4.7 kernel by:

1. boot kernel
2. pflash -m -i works
3. pflash -i (without -m so using direct register access)
4. pflash -m -i complains chip looks erased.
5. restoring 0x1e630004 with devmem did not fix pflash -m -i
6. devmem 0x30000000 32 is not 0xff data (nor a few small offsets). (did not try /dev/mtd7)
7. pflash -i continued to work
8. reboot fixed pflash -m -i until next direct access (without -m).

[cyrilbur-ibm]

On Thu, 2016-09-15 at 16:30 -0700, Milton D. Miller II wrote:

I was able to recreate this on my latest 4.7 kernel by:

1. boot kernel
2. pflash -m -i works
3. pflash -i (without -m so using direct register access)
4. pflash -m -i complains chip looks erased.
5. restoring 0x1e630004 with devmem did not fix pflash -m -i
6. devmem 0x30000000 32 is not 0xff data (nor a few small offsets).
   (did not try /dev/mtd7)
7. pflash -i continued to work
8. reboot fixed pflash -m -i until next direct access (without -m).

I hesitate to say that this is working as intended...

When you do a -m you're using the kernel driver. If you drop the -m
you're accessing hardware directly as you've correctly pointed out, at
that point the state of the hardware has changed out from under the
kernel, the most common response from the hardware when a drivers
actions do not correspond with the state of the hardware is all 0xff
reads. The kernel doesn't have a way of distinguishing legitimate all
0xff reads and bogus ones so it happily returns the bad reads.

I suppose you're hinting that using devmem to put the hardware back
into the state the kernel thinks it is should restore -m functionality,
and it probably does but, exactly what is that state?

pflash -i should always continue to work as it always inits the
hardware each time you run the binary.

Reboot obviously fixes the kernel as the kernel reinits the hardware
for its self.

What do you expect should happen?

Cyril

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[mdmillerii]

@cyrilbur-ibm actually it did not make it working, but it may be a mismatch between the kernel and the state of the flash eg command size. So in some regards this says we should choose between the kernel knowing about this hardware and pflash managing it with the host.

This sort of goes back to the plan for openbmc/openbmc#553 and my comment in the driver line 415-418

My understanding is there is a ipmi handshake to arbitrate with opal for flash access but I haven't looked it up. I was told hostboot always said not now.

But this does raise the issue: should we totally disable the pnor in the current kernel so its boot does not interrupt the host?

[cyrilbur-ibm]

On Thu, 2016-09-15 at 16:56 -0700, Milton D. Miller II wrote:

@cyrilbur-ibm actually it did not make it working, but it may be a
mismatch between the kernel and the state of the flash eg command
size. So in some regards this says we should choose between the
kernel knowing about this hardware and pflash managing it with the
host.

Were you running pflash on the host? What was the BMC doing during this
time? This is quite a different situation.

I'm quite sure I ripped out direct access by pflash on the host. When
running on the host pflash will access the flash through firmware
(skiboot) via linux and MTD, I believe skiboot and the BMC do have a
method for arbitrating flash access.

This sort of goes back to the plan for openbmc/openbmc#553 and my
comment in the driver line 415-418

This all seems correct.

My understanding is there is a ipmi handshake to arbitrate with opal
for flash access but I haven't looked it up.

AFAIK skiboot and AMI firmware have this. We would need to connect it
up in OpenBMC if it isn't already.

I was told hostboot always said not now.
But this does raise the issue: should we totally disable the pnor in
the current kernel so its boot does not interrupt the host?

Perhaps, this won't save us from people running pflash directly on the
BMC though...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[mdmillerii]

I was running the pflash binary on the BMC that was in the root fs already on my target system (some rev ~120 commits past 1.0.0 according to /etc/os-release). I did not try booting the host. I know pflash will tune for fast-read while the kernel still does slow reads. The kernel driver tries to use a conservative clock value which is in the control register. It should always write the full control register from the values it saves at initialization (driver probe).

Our present skeleton kicks off pflash tune against the host pnor, and also uses pflash to update the BMC. The boot process calls pflash to write the base device tree with the BMC discovered inventory items.

[cyrilbur-ibm]

On Thu, 2016-09-15 at 17:30 -0700, Milton D. Miller II wrote:

I was running the pflash binary on the BMC that was in the root fs
already on my target system (some rev ~120 commits past 1.0.0
according to /etc/os-release).  I did not try booting the host.  I
know pflash will tune for fast-read while the kernel still does slow
reads.  The kernel driver tries to use a conservative clock value
which is in the control register.  It should always write the full
control register from the values it saves at initialization (driver
probe).

Yeah so definitely has the BMC kernel and pflash accessing the same
regs and that didn't go well. We don't need to disable the kernel
driver, we simply need to ensure that either always pflash -m or never
use the kernel driver. Obviously the safest option is to disable it. I
must stress here that this is only true and a good idea while the host
is off!

Since we don't have the arbitration between skiboot and BMC for flash
access, we must never use either the kernel driver or pflash while
the host is on! That is more or less ok since pflash on the host works
fine so the host can flash its self.

Our present skeleton kicks off > pflash tune against the host > pnor, and also uses pflash to update the BMC. The boot process calls > pflash to write the base device tree with the BMC discovered inventory items.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/openbmc/linux","title":"openbmc/linux","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/openbmc/linux"}},"updates":{"snippets":[{"icon":"PERSON","message":"@mdmillerii in #51: I was running the pflash binary on the BMC that was in the root fs already on my target system (some rev ~120 commits past 1.0.0 according to /etc/os-release). I did not try booting the host. I know pflash will tune for fast-read while the kernel still does slow reads. The kernel driver tries to use a conservative clock value which is in the control register. It should always write the full control register from the values it saves at initialization (driver probe).\r\n\r\nOur present skeleton kicks off pflash tune against the host pnor, and also uses pflash to update the BMC. The boot process calls pflash to write the base device tree with the BMC discovered inventory items."}],"action":{"name":"View Issue","url":"https://github.com/openbmc/linux/issues/51#issuecomment-247491126"}}}