#
✕ nick:sven
#asahi-dev · 2022-02-17 16:36
svenI saw part of those optimization on twitter. Very impressive how much it improved
#asahi-dev · 2022-02-17 16:34
svennice!
#asahi-dev · 2022-02-17 16:33
svencould be worth a try
#asahi-dev · 2022-02-17 16:33
svenmacOS does a readl_relaxed after each inval fwiw. I’m not sure if that’s required ofc.
#asahi-dev · 2022-02-17 16:16
svenyup, that was pretty quick :)
#asahi-dev · 2022-02-17 16:16
svenyeah, absolutely
#asahi-dev · 2022-02-17 16:15
svenyup :D
#asahi-dev · 2022-02-17 16:14
svenyeah, i just hate when a mystery lock solves something that smells like some kind of race
#asahi-dev · 2022-02-17 16:13
svenquite a few people have been using it on the max/pro as well before cpufreq as well and never saw it either.
#asahi-dev · 2022-02-17 16:12
svenif it's actually a race between nvmmu/cq_db and that writel maybe the higher cpufreq just makes it more likely to lose (or win, depending how we look at it :D) that one
#asahi-dev · 2022-02-17 16:11
svenmakes sense
#asahi-dev · 2022-02-17 16:09
svenyeah. i'm just trying to understand why we need that lock around the writel
#asahi-dev · 2022-02-17 16:08
svenso maybe if something like "start nvmmu invalidation, start new command, write cq doorbell" happens things break. no idea why though.
#asahi-dev · 2022-02-17 16:08
svenokay, so from looking at the hypervisor logs macos seems to never interleave "ring cq doorbell + nvmmu invalidation" with "write new tag to sq"
#asahi-dev · 2022-02-17 16:00
sveni can't seem to reproduce the issue at all even without the lock right now :/
#asahi-dev · 2022-02-17 15:59
svenaxboe: remind me again, is taking the anv->lock around the writel itself enough to prevent those timeouts or did the memcpy also have to be inside the lock?
#asahi-dev · 2022-02-17 14:49
sven:D
#asahi-dev · 2022-02-17 14:13
svenevery team needs someone to do the mediawhoring ;)
#asahi-dev · 2022-02-17 14:12
svenaaand now I can't reproduce the missing interrupts at all anymore with the original code *sigh*
#asahi-dev · 2022-02-17 11:42
svenmacos uses INTMS and INTMC in its interrupt handler apparently, which linux doesn't because it relies on the interrupt controller itself
#asahi-dev · 2022-02-17 11:42
svenwhat exactly does "number of openings" do exactly?
#asahi-dev · 2022-02-17 11:41
sventhat sounds very similar actually. some commands never generate the completed interrupt but they do appear in the completion queue and can be polled but others don't ever seem to be triggered
#asahi-dev · 2022-02-17 11:16
svenyeah, that's what I thought after spending some time with the interrupt code yesterday as well
#asahi-dev · 2022-02-17 11:01
svenso that just leaves that weird issue where we sometimes miss an interrupt now
#asahi-dev · 2022-02-17 07:37
sventhere's force unit access for write commands but that would probably only order writes where it's set (if it enforces the ordering at all)
#asahi-dev · 2022-02-17 07:34
svenI don't remember seeing anything about write barriers in the nvme spec but I only read that one for the first time a few weeks ago
#asahi-dev · 2022-02-17 07:33
svenit's not impossible to do with multiqueues. it's just very challenging.
#asahi-dev · 2022-02-17 06:51
svenat least that explains all the weird behavior
#asahi-dev · 2022-02-17 06:50
svenhah
#asahi-dev · 2022-02-16 21:01
svencan't see any obvious pattern and it's getting late here. i'll take a closer look tomorrow
#asahi-dev · 2022-02-16 20:53
svenand if only macos would now let me ssh to it again I could run fio with randwrite once more to see if there's a pattern for the issued flushes
#asahi-dev · 2022-02-16 20:49
svenso it looks like macos doesn't have any kind of magic to make them faster
#asahi-dev · 2022-02-16 20:49
svenyeah
#asahi-dev · 2022-02-16 20:48
svenok, also found a ~20x write -> flush -> 1x write -> flush sequence where both flushs are slow as well
#asahi-dev · 2022-02-16 20:40
svenlet's see if i can find a fast flush. but so far i've only seen the slow ones
#asahi-dev · 2022-02-16 20:39
sventhe python+usb delay is already bigger than 20 usec afaict
#asahi-dev · 2022-02-16 20:37
sven~15-30 msec on macos for flushes (with all the additional delays introduced due to usb and python)
#asahi-dev · 2022-02-16 20:16
svenfair :D
#asahi-dev · 2022-02-16 20:16
svenmeasuring flush time there is a bit tricky because by the time I can figure out the command is a flush there's already usb and python in the way but i'll see what i can do
#asahi-dev · 2022-02-16 20:15
sveni'll still try to confirm if macos does the same thing but it certainly looks like it does
#asahi-dev · 2022-02-16 20:14
sveni'm in favor of that patch fwiw
#asahi-dev · 2022-02-16 20:14
svenaxboe: yeah, if fio (or anything else in userland) was slow on macos we could probably get it fixed. but with the way it currently is I don't have high hopes
#asahi-dev · 2022-02-16 20:12
sventhere are some apple employees lurking in this channel but I think the issue with this problem is that it only happens under linux
#asahi-dev · 2022-02-16 20:09
svenwe can always file a radar with apple and have it disappear and get no feedback for years! ;)
#asahi-dev · 2022-02-16 19:38
svendefinitely!
#asahi-dev · 2022-02-16 19:36
svenI enjoy the occasional hack very much ;)
#asahi-dev · 2022-02-16 19:34
sven:D
#asahi-dev · 2022-02-16 18:49
svenI think jannau volunteered to take over anyway :>
#asahi-dev · 2022-02-16 18:30
svenI’ll take a closer look tomorrow but it looked reasonable at first glance
#asahi-dev · 2022-02-16 18:30
svenglad you took a shot at it ;)

plus ›