#asahi-dev · 2023-01-10 00:34
cyrozapsven: Thanks, a panic should be "good enough" for what I'm testing.#asahi-dev · 2023-01-09 08:25
cyrozapQuick question about the IOMMU behavior under macOS--will macOS log IOMMU faults somewhere? Like, if I have a PCIe device plugged in over Thunderbolt, and the PCIe device tries accessing memory it doesn't have permission to access, will that be logged somewhere I can see it?#asahi-dev · 2022-12-13 09:05
cyrozaptime after the original bad write.#asahi-dev · 2022-12-13 09:05
cyrozapa "hardware-controlled" part (for the USB data TX/RX), and even if the quirks of the software-controlled part don't cause issues, the as-yet unknown quirks of the hardware-controlled part could. Which is why I'm writing this analyzer code--to find exactly where the bad write happens, since the POWER9 doesn't send a Completer Abort TLP until the device tries reading again, which could be a very long #asahi-dev · 2022-12-13 09:05
cyrozapmarcan: It very well could be. This controller has a limitation where it will read two 32-bit dwords when you tell it to read 1-8 bytes, and four dwords if you tell it to read 9-16 bytes, and while that shouldn't normally cause any issues, it wouldn't surprise me if there were more such quirks in its design. Especially since the PCIe interface has a "software-controlled" part (for use by the 8051) and #asahi-dev · 2022-12-13 08:42
cyrozapASM1142: https://github.com/cyrozap/asmedia-xhc-re/blob/3817694cb8c47f1a959b91cf0b91749aeac3d5af/data/regs-asm1142.yaml#L561-L648), but it's basically the same register interface at a different offset, so I can provide that info if you need it.#asahi-dev · 2022-12-13 08:42
cyrozapmarcan: btw, if you want to deliberately trigger an IOMMU access fault, you can use the BAR0 registers to access the internal MMIO registers of the host controller's PCIe controller (holy indirection, Batman!) to perform arbitrary DMA reads and writes. I haven't publicly documented that for this generation of the host controller (I have some registers documented for the previous generation, the #asahi-dev · 2022-12-13 08:31
cyrozapYeah, that's the worst part--the bug could be in the hardware (I've already found a couple other hardware bugs), or in the firmware, or in the IOMMU driver, or in how the xhci driver interacts with the IOMMU driver. So, basically _anywhere_.#asahi-dev · 2022-12-13 08:28
cyrozapxHCI spec is _huge_ so progress on that is slow :/#asahi-dev · 2022-12-13 08:28
cyrozapmarcan: I haven't ruled out PCIe as the issue yet, but based on the traces I've analyzed (I have some PCIe analyzer equipment), I don't believe it's an issue at the PCIe layer. I've been working on scripts to parse the trace data (https://github.com/cyrozap/agilent-pcie-analyzer-re), and have some (unpublished, hacked-together) code to do higher-level tracing and analysis of the xHCI traffic, but the #asahi-dev · 2022-12-12 22:13
cyrozapThough it may not be strictly necessary.#asahi-dev · 2022-12-12 22:13
cyrozapAlso, I think it might be easier to trigger the issue with UAS drives.#asahi-dev · 2022-12-12 22:12
cyrozapUSB. This shouldn't be necessary, since I've been able to trigger the fault with mechanical hard drives on 5 Gbps USB, but maybe it's the fact that, even with the newer firmware, the USB link is now getting saturated (I saw ~973 MB/s on the P5 Plus drive before the system detected the bad write).#asahi-dev · 2022-12-12 22:12
cyrozapI just managed to trigger the fault on firmware version 210330_70_02_40. When I tried earlier, I was using a cheap USB 3 thumb drive (64 GB, 100 MB/s max read speed) and a Crucial P5 250 GB SSD in a SuperSpeed Plus Gen 2x1 enclosure. This time, instead of the cheap USB drive, I used a Crucial P5 Plus 2 TB SSD in a SuperSpeed Plus Gen 2x1 enclosure, so now both SSDs plugged into the card support 10 Gbps #asahi-dev · 2022-12-12 21:53
cyrozapI was using the one from macOS 13.0, but I don't think the slight version difference should matter that much since I know the fault can be triggered on even older versions.#asahi-dev · 2022-12-12 21:36
cyrozapjannau: Thanks for your help! It's especially interesting to see that you're running a slightly older version than what I have (191118_70_11_08 vs. 191118_70_11_11), and yet haven't encountered the issue. Meanwhile, I've uploaded a much more recent firmware (210330_70_02_40) to my ASM3142 card and haven't been able to trigger the fault yet.#asahi-dev · 2022-12-12 21:26
cyrozapjannau: Interesting... Would you mind running `dd if=/usr/lib/firmware/asmedia/asm2214a-apple.bin bs=1 skip=128 count=6 2>/dev/null | xxd -ps`? That will print the version of the firmware you have installed.#asahi-dev · 2022-12-12 21:01
cyrozapIt'll be interesting to see if this issue happens on Apple hardware, and whether it happens in just Linux or in macOS, too.#asahi-dev · 2022-12-12 20:51
cyrozapASM3142 firmware version: 191118_70_11_11 (the one I extracted from a kernelcache)#asahi-dev · 2022-12-12 20:49
cyrozapI've confirmed that the issue still exists even when running the firmware Apple ships. Tested with Linux 6.0.11 on POWER9.#asahi-dev · 2022-12-12 20:00
cyrozapzzywysm_: Yeah, that's why I'm testing on my ASM3142 card the firmware that Apple ships--I want to see if Apple had them fix the issue in firmware.#asahi-dev · 2022-12-12 19:41
cyrozapAnd if you _can't_ trigger the fault, I'll be really interested to know about that, since that would mean that the issue likely has something to do specifically with the POWER9 IOMMU or its kernel driver (though this issue doesn't occur with any of the TI or Fresco Logic USB host controllers I've tried).#asahi-dev · 2022-12-12 19:35
cyrozapI'm not sure if there's a way to enable tracing the DMA buffers (or whatever they're called) allocated by the xHCI driver, but assuming you can successfully trigger the fault, it would be helpful to know what buffer is being overrun (assuming that's what the issue is and the xHC isn't just trying to write to some random address).#asahi-dev · 2022-12-12 19:25
cyrozapbtw, thanks for helping me look into this!#asahi-dev · 2022-12-12 19:21
cyrozapdrives into the same host controller, then start two reads from them (`sudo dd if=/dev/sdN of=/dev/null bs=1G iflag=direct status=progress`) as simultaneously as possible. Doing this triggers the issue almost immediately. It also works with USB hard drives, though it may trigger slightly less than immediately.#asahi-dev · 2022-12-12 19:21
cyrozapmarcan: Firmware version shouldn't matter--I've seen the issue with every ASMedia USB host controller I've tested, from the original ASM1042 through the ASM3142 (it seems I haven't tried the ASM3242 yet). But I'll load the Apple ASM3142 firmware on an ASM3142 card I have here and see if I can trigger the fault, just to be sure. The most reliable way I've found to trigger it is to plug in two USB NVMe #asahi-dev · 2022-12-12 06:29
cyrozapI first wrote about this issue a couple of years ago, but have yet to discover exactly what's going wrong, and where. You can find the original thread where I discussed it here: https://lore.kernel.org/all/CAO3ALPyB1JDvvC27JGgAoTuHh0w+897tPhmTKX9PQWBFCrrnbQ@mail.gmail.com/#asahi-dev · 2022-12-12 06:28
cyrozapoccurs.#asahi-dev · 2022-12-12 06:28
cyrozapreproduce the error on POWER9 systems, which have a very strict IOMMU, so I was curious if the same issue affects Apple Silicon systems, which I assume also have a strict IOMMU. Even if the platform doesn't support any kind of capability like EEH (Enhanced Error Handling), where the IOMMU reports misbehaving devices to the OS, it would be interesting to see if any data gets lost or memory corruption #asahi-dev · 2022-12-12 06:28
cyrozapmarcan, sven: Heads-up, there's an issue with ASMedia's USB host controllers where if you try reading from multiple USB devices simultaneously that are connected to the same ASMedia host controller (doesn't matter whether it's through a hub or they're plugged directly into the HC's USB ports), the host controller will attempt a DMA write to an address it doesn't have permission to access. I can easily #asahi-dev · 2022-11-30 14:41
cyrozapdocumentation from the manufacturer, I'd be really wary of making any extra optimizations.#asahi-dev · 2022-11-30 14:41
cyrozapmarcan: No, you should definitely also add the checks for the SRAM address, both before and after writing the SRAM data. Otherwise, data might get written to the wrong address, or the write rate will otherwise just be too fast for the chip to handle. If it works without those checks, then I guess you can skip them, but I'd be worried about the process failing randomly for users. Without proper #asahi-dev · 2022-11-30 02:17
cyrozapLike, you just can't assume this part of the hardware will always work properly. Since as you've already noted, it was probably originally intended as a debug interface.#asahi-dev · 2022-11-30 02:13
cyrozapbut regardless of that I think it serves as a good example of how incredibly janky this process is and why it's absolutely necessary to do so many checks at each step.#asahi-dev · 2022-11-30 02:13
cyrozapcorrupted. More specifically, the highest bit of the highest byte in each affected 16-bit word would be flipped from either 0 to 1 or 1 to 0--all other bits were unaffected. I'm not sure if this is an issue with only this specific chip, and other ASM1142's are fine, or if it's a more widespread issue with this series of controllers (and maybe the ASM1042A as well, since it also supports the feature), #asahi-dev · 2022-11-30 02:13
cyrozapAlso, a small note about the reliability of this process: I was also experimenting with the firmware loading process for an earlier host controller, the ASM1142, and as part of that wrote a small firmware that would simply read itself back to the host. I was very surprised to find that pretty much every time I uploaded firmware to that chip, about 50-60 bytes between 0x0E40 and 0x1090 would get #asahi-dev · 2022-11-30 02:00
cyrozapHopefully these suggestions will help you fix the issues you were seeing with the firmware load process!#asahi-dev · 2022-11-30 01:58
cyrozap100 ms or so (just a guess).#asahi-dev · 2022-11-30 01:58
cyrozap5) Immediately after disabling SRAM write access (`asmedia_write_reg(hcd, 0x500e, 0);`), release the controller from reset with `asmedia_write_reg(hcd, 0x5042, 0);`. Then wait for the controller to initialize--it will reset itself once during this process in order to increase its clock speed from 78.125 MHz to 156.25 MHz, so don't try to talk to it before it's finished. Maybe add a delay of 10 ms or #asahi-dev · 2022-11-30 01:50
cyrozapregister 0xEF needs to be set to enable readback (set it before doing the readback, but after writing has finished).#asahi-dev · 2022-11-30 01:49
cyrozap4) As a safety check, after all the SRAM has been written to, the SRAM should be read back and compared with the firmware. Note that unlike when writing to SRAM, reading it back is done in 0x4000-byte/0x2000-word interleaved blocks instead of 0x8000-byte/0x4000-word blocks, and uses two dword registers (0x3018 and 0x301c) for that data instead of just one. And of course bit 6 in PCI config space #asahi-dev · 2022-11-30 01:42
cyrozap3) Immediately after writing the code to SRAM (https://github.com/AsahiLinux/linux/blob/602fee88eb0603f4d7576797397aff6b33b41765/drivers/usb/host/xhci-pci-asmedia.c#L246), read the SRAM _address_ back in a loop to make sure that it's changed (it should have increased by two).#asahi-dev · 2022-11-30 01:40
cyrozap2) Immediately after writing the SRAM address (https://github.com/AsahiLinux/linux/blob/602fee88eb0603f4d7576797397aff6b33b41765/drivers/usb/host/xhci-pci-asmedia.c#L243-L244), read it back in a loop until the value read matches the value that was written.#asahi-dev · 2022-11-30 01:39
cyrozap1) Skip the controller reset (https://github.com/AsahiLinux/linux/blob/602fee88eb0603f4d7576797397aff6b33b41765/drivers/usb/host/xhci-pci-asmedia.c#L222-L229), and just do `asmedia_write_reg(hcd, 0x5042, 2);` to halt the controller and hold it in reset (note the "2" to halt instead of "1" for reset).#asahi-dev · 2022-11-30 01:37
cyrozapfirmware is in fact running and that the code was all loaded correctly. Based on this research, I have several suggestions for the kernel code:#asahi-dev · 2022-11-30 01:37
cyrozapmarcan: I've finished looking into the ASM3142 firmware loading process, and now have an implementation working in Python (https://github.com/cyrozap/asmedia-xhc-re/blob/836870de1d2584a5716d392b9cce4dfb5134d05f/tools/asm_tool.py). I haven't seen any issues uploading code into the host controller using this program--it seems to just work every time for me, and I've done testing to confirm that the #asahi-dev · 2022-11-23 04:10
cyrozapMMIO registers) that the code ROM would have done, so having the code ROM do that too is redundant.#asahi-dev · 2022-11-23 04:10
cyrozapThis write (https://github.com/AsahiLinux/linux/blob/602fee88eb0603f4d7576797397aff6b33b41765/drivers/usb/host/xhci-pci-asmedia.c#L233-L234) halts the CPU (I think by holding it in reset), so anything the code ROM does is just going to get thrown away when that write happens. And when it comes out of reset executing from code RAM, the loaded firmware will do the same init stuff (clearing/initializing #asahi-dev · 2022-11-23 04:05
cyrozapbtw, I don't think you need to do the reset to code ROM here at all: https://github.com/AsahiLinux/linux/blob/602fee88eb0603f4d7576797397aff6b33b41765/drivers/usb/host/xhci-pci-asmedia.c#L222-L230#asahi-dev · 2022-11-23 04:03
cyrozapI think I'm going to try some stuff myself on an ASM3142 I have here, and report back with my findings.