#asahi-dev · 2022-02-16 18:30
svenyeah, I remember trying to implement it and then just leaving it for another day#asahi-dev · 2022-02-16 18:29
svenand hopefully once we figure out this weird missed irqs nvme as well#asahi-dev · 2022-02-16 18:28
svensounds good. We should figure out how to submit rtkit then. #asahi-dev · 2022-02-16 18:26
svenbut then again.. it doesn’t even have the display drawing lots of power and the rest is very efficient #asahi-dev · 2022-02-16 18:26
sventhat’s quite a long time #asahi-dev · 2022-02-16 18:22
svenyeah, just need to add some time stamp whenever the command was issued #asahi-dev · 2022-02-16 18:22
svenbut it wouldn’t surprise me and would also fit that I only rarely saw flush commands in the HV#asahi-dev · 2022-02-16 18:21
svenI’ve tried looking for their flush logic but ugh compiled iokit c++ code#asahi-dev · 2022-02-16 18:18
svenit wouldn’t be the first nasty hack we need for these machines :(#asahi-dev · 2022-02-16 18:10
svenI think openbsd takes that “just never issue flushes” approach though#asahi-dev · 2022-02-16 18:08
sven*controller enable#asahi-dev · 2022-02-16 18:08
sventhe firmware will also complain about an unclean shutdown if you don’t clear that control layer enable but before rebooting but I haven’t seen any data loss from that#asahi-dev · 2022-02-16 18:07
svenand you can lose data if you just hard reset the machine before a flush #asahi-dev · 2022-02-16 18:06
svenmacOS does issues flushes but with a much lower frequency compared to linux when running the same fio command #asahi-dev · 2022-02-16 17:21
svenI only looked at the commands it issued, but I can see if I can also get get the flush timing #asahi-dev · 2022-02-16 17:21
sventhe same thing happens on the T2 Macs with pci.c fwiw#asahi-dev · 2022-02-16 17:20
svenThat’s already more than I figured out#asahi-dev · 2022-02-16 16:40
svenI think to actually serialize the writes we’d need an additional dummy read after that write due to https://www.kernel.org/doc/html/latest/driver-api/io_ordering.html#asahi-dev · 2022-02-16 16:40
svenit smells like we are racing something in the irq handler that makes us miss the next interrupt for the completion :/#asahi-dev · 2022-02-16 15:46
svenso i'm just even more confused now :D#asahi-dev · 2022-02-16 15:46
svenyeah.. i've been reading some irq code and i haven't even found a different code path with the M1 interrupt controller yet#asahi-dev · 2022-02-16 14:32
svenit seems to fix the random lockup but i don't think it improves flush/randwrite performance#asahi-dev · 2022-02-16 14:31
sveni think i just messed up there fwiw#asahi-dev · 2022-02-16 13:38
svenaxboe: so i had some explanation why this might be the issue in the shower this morning but i can't put it together again right now :/#asahi-dev · 2022-02-16 13:37
svenmaz: ah, true. multi-tasking with $work right now and my brain doesn't seem to like that#asahi-dev · 2022-02-16 13:27
svenjust IRQ_TYPE_EDGE_RISING. also not totally convinced anymore at this point it will do anything different#asahi-dev · 2022-02-16 13:13
svenif i understand the memory model correctly (very big if) two stores to device memory from two separate cores will always issue both stores but their order isn't guaranteed #asahi-dev · 2022-02-16 13:10
sveni think those two take a different path in the core irq code though#asahi-dev · 2022-02-16 12:59
sven*increase#asahi-dev · 2022-02-16 12:59
svenerm..#asahi-dev · 2022-02-16 12:57
sven(also seems to use randwrite+flush performance for me)#asahi-dev · 2022-02-16 12:55
svenjannau: / axboe: so... uh.. can you guys try to set the nvme irq to IRQ_TYPE_EDGE_RISING and see if you can still reproduce the timeouts?#asahi-dev · 2022-02-15 21:14
svenand that irq handling indeed looks better :)#asahi-dev · 2022-02-15 21:13
sveni think the enabled originally comes from NVMEQ_ENABLED in pci.c. i convinced myself at some point that i didn't need it but then went back a few days later because i convinced myself that i do need it and then decided to keep it for now just-in-case IIRC#asahi-dev · 2022-02-15 20:36
svenI just set index = tag so that I don’t have to deal with a tcb index and a queue index #asahi-dev · 2022-02-15 20:36
sventhe hw first looks into the tcb at the specified tag, then reads the actual index into the “queue” (which is really more like an array here) and then reads the queue at that index#asahi-dev · 2022-02-15 20:34
sventhe memcpys should be to two different tags though #asahi-dev · 2022-02-15 20:33
sveni still wonder what exactly messes up the submission if it’s not inside a lock though #asahi-dev · 2022-02-15 20:32
sventrue, that’s even better#asahi-dev · 2022-02-15 20:32
svenonce should be enough #asahi-dev · 2022-02-15 20:32
svenah, no#asahi-dev · 2022-02-15 20:31
svenyeah, I’m pretty sure most of the irq handling is inside the lock#asahi-dev · 2022-02-15 20:27
svenhmmm… I wonder if there’s anything in the irq handler that could mess something up in that case#asahi-dev · 2022-02-15 20:24
svenI assumed that writel to MMIO should be enough. It’s dma_wmb which should order it against the previous writes to normal memory and then just a single STR#asahi-dev · 2022-02-15 20:22
svenyeah, same here.. time to watch will deacon’s io ordering talk again ;)#asahi-dev · 2022-02-15 20:20
svenvery. that writel has a dma barrier though#asahi-dev · 2022-02-15 20:17
sveni just don’t understand why#asahi-dev · 2022-02-15 20:17
svenand adding that lock seems to fix it too