#
…

2022-02-15

00:41user982492 has joined #asahi-dev
01:33phire_ has joined #asahi-dev
01:33phire is now known as Guest407
01:33phire_ is now known as phire
01:36Guest407 has quit [Ping timeout: 480 seconds]
02:55yuyichao has quit [Ping timeout: 480 seconds]
03:00skipwich has quit [Quit: DISCONNECT]
03:01skipwich has joined #asahi-dev
03:31PhilippvK has joined #asahi-dev
03:34phiologe has quit [Ping timeout: 480 seconds]
03:43kov has quit [Quit: Coyote finally caught me]
03:48kov has joined #asahi-dev
05:00c10l3 has quit []
05:00c10l3 has joined #asahi-dev
05:49the_lanetly_052___ has joined #asahi-dev
06:42Bey0ndB1nary has joined #asahi-dev
07:45Bey0ndB1nary has quit []
08:00MajorBiscuit has joined #asahi-dev
08:18Bey0ndB1nary has joined #asahi-dev
08:18Bey0ndB1nary has quit []
08:47user982492 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
08:55gladiac has quit [Quit: k thx bye]
08:56gladiac has joined #asahi-dev
09:24Major_Biscuit has joined #asahi-dev
09:26MajorBiscuit has quit [Ping timeout: 480 seconds]
09:32user982492 has joined #asahi-dev
09:32user982492 has quit []
10:27n1c has joined #asahi-dev
10:34n1c has quit [Quit: ZNC 1.8.2+deb1+focal2 - https://znc.in]
10:43n1c has joined #asahi-dev
11:34the_lanetly_052___ has quit [Remote host closed the connection]
11:34the_lanetly_052___ has joined #asahi-dev
11:48the_lanetly_052___ has quit [Ping timeout: 480 seconds]
12:20yuyichao has joined #asahi-dev
13:40Major_Biscuit has quit [Ping timeout: 480 seconds]
13:41Major_Biscuit has joined #asahi-dev
13:49refi64 has quit [Remote host closed the connection]
13:50alyssa has joined #asahi-dev
13:50refi64 has joined #asahi-dev
14:11riker77 has quit [Quit: Quitting IRC - gone for good...]
14:29riker77 has joined #asahi-dev
14:34riker77 has quit [Quit: Quitting IRC - gone for good...]
14:35MajorBiscuit has joined #asahi-dev
14:37yuyichao has quit [Ping timeout: 480 seconds]
14:37Major_Biscuit has quit [Ping timeout: 480 seconds]
15:01yuyichao has joined #asahi-dev
15:23riker77 has joined #asahi-dev
15:38marcanok, too tired to do more work today, I'm going to sleep early for once. pushed things as is to spmi/work.
15:39marcant6000 should work; t8103 needs devicetree stuff (the nvmem/spmi bits)
15:39j`eymarcan: yes get some rest!
15:39marcanthanks :)
15:40j`eymarcan: i guess the t8103 stuff is pretty simple, if copied mostly from the previous attempt..
15:41marcanyeah, mostly
15:41marcanjust needs the nvmem offsets, which I think I pasted here previously
15:42marcanalso the RTC stuff is, of course, >12.x only
15:42j`eyoh right.. that might be the push I need to get off 11.x :P
16:14axboe has joined #asahi-dev
16:14axboemarcan: tested the new branch, and fwiw I see the same issue as the old one
16:15axboeit works for the E cores, and first cluster of the P cores
16:15axboesecond cluster is stuck at whatever the boot frequency is
16:15axboesecond cluster also has cpu_capacity == 0 rather than 1024
16:15axboeI feel like this is something silly, but it eluded me yesterday
16:18j`eydoes it try to change it? ie does it even get to apple_soc_cpufreq_set_target for the 2nd cluster
16:21axboeI poked a bit yesterday, and seems like it was using the wrong clk base for pcluster2, it was the pcluster1 address
16:21axboewhich again made little sense, I saw all three get registered
16:22axboeand the fact that capacity isn't set either just made me assume that something is wrong with the setup of that second pcluster
16:23axboenew one does seem to set the right clock if I just force performance as the governor
16:23axboeso might be as simple as schedutil being borken with capacity == 0
16:23axboegiven that the other clusters have it set right
16:27jannauthe readq_poll_timeout in apple_soc_cpufreq_set_target needs to be atomic
16:29axboein the devicetree in sysfs, capacity looks correct for all CPUs
16:29jannauotherwise it appears to work expected on a m1 max with all cores
16:32axboejannau: curious on the m1 max, do you get the cpu capacity set for both p clusters?
16:34jannauaxboe: yes, 456 for the two e-cores and 1024 for all 8 p-cores
16:34axboejannau: hmm
16:35axboe456 here for E cluster, 1024 for first P cluster, 0 for second P cluster
16:38jannauI suspect we need to remove the disabled core also in cpu-map in the devicetree
16:38axboedid try that yesterday but didn't make a difference
16:39axboelet me try on the new base, just in case I messed it up
16:41jannaureproduced the missing capacity by skipping cpu 5 and 9
16:42axboebooting now with those 2 removed
16:42axboeI think m1n1 needs a fixup for that
16:43axboenow it finds last cpu in p cluster 1 "not alive", and then hits a MPIDR mismatch on first cpu in cluster 2
16:46jannauyes, I'm working on it
17:12axboejannau: I can give it a whirl
17:12axboejannau: with or without editing the dtsi to remove the non-existent cores?
17:14jannauaxboe: should work with an unmodified dtsi
17:14axboejannau: ok, will try
17:16kettenisto what extent does m1n1 trim the device tree?
17:18j`eymarcan: spmi/work Kconfig has MFD_APPLE_SPMI_PMU, which seems to be leftover
17:20axboejannau: gives me a MPIDR mismatch
17:20axboeDT CPU 1 MPIDR mishatc; 0x1 ~= 0x10100
17:21jannaukettenis: quite bit if nodes are missing from the ADT, required by the hypervisor
17:22jannauI'm only aware of the missing cpu cores for direct boot
17:23kettenisI mean the FDT not the ADT
17:25jannauit removes nodes from the FDT based on the ADT
17:25kettenisright, and it should do so for the cpu nodes as well
17:26user982492 has joined #asahi-dev
17:29jannauit does, it doesn't however remove disabled cores from the cpu-map added for cpufreq driver
17:32axboeif I just hack smp.c to keep a separate cpu index rather than use i it works for me, on top of plain m1n1
17:32svenjannau: it should’ve been clairvoyant and done that already ;)
17:33axboefixes capacity, and cpufreq then works on 2nd pcluster too
17:37jannauaxboe: which "cpu-id"s do you have in the apple device tree? 0-7 or 0-9 with gaps
17:37jannaum1n1 prints "Starting CPU x ..."
17:38axboejannau: this is with the dtsi trimmed, didn't try without for that hack
17:38axboejannau: so e(0 1) p(0 1 2) p(0 1 2)
17:39axboejannau: I tried your patch with stock dtsi
17:44MajorBiscuit has quit [Quit: WeeChat 3.4]
17:48jannauaxboe: we have to solve this in m1n1 as we haf to use a single dtb for M1 Pro machines with 8 and 10 cpu cores
17:49axboejannau: yep
17:50axboefor reference, this is the m1n1 hack
17:50axboehaven't tried it with un-edited dtsi
17:51axboethe sparseness ends up being an issue for the spin_table
17:51axboewhich the hack works around
17:51axboeso might work with stock dtb
17:56jannauI thought you tried with the unmodified dtb. I use https://paste.debian.net/1231019/ to simulate the sparseness and that works with my patch and the stock dtb
17:59axboejannau: for your patch I used unmodified
17:59axboejannau: like you asked :)
18:00axboejannau: see the debug output I badly typed
18:01jannauok. I saw that but I can't explain how the change could cause that
18:04user982492 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
18:04Guest119 has left #asahi-dev [WeeChat 3.3]
18:07svenaxboe: do you want to submit that tipd interrupt mask fix or should I do it?
18:15axboesven: either way is fine with me, just let me know what you prefer
18:16svenif you already have a patch that’s working it’s probably the easiest to just submit it
18:17axboeok, will do so right now
18:18axboesven: adding an actual changelog :)
18:19axboesven: do you want a suggested-by in there?
18:19svenuh, sure
18:19axboecredit where credit is due
18:20axboeSven Peter <sven@svenpeter.dev>
18:20axboethat your preferred email?
18:22axboeshipped
18:22svennice :)
18:28user982492 has joined #asahi-dev
18:29user982492 has quit []
18:33axboeI'm all for flushing out patches, hate carrying them
18:33alyssahere here
18:33j`eyso with the tps issue, it was a DEFER_PROBE thing or?
18:33alyssaoh that tps
18:33axboeI know you guys all know this, but I'm still blown away by the perf here. game changing imho
18:35j`eyaxboe: we're a bit late on the carrying patches part :P
18:37svenyeah… we’re carrying way too many patches right now.
18:39axboej`ey: I know! but can always get done ;)
18:40j`eysven: 157 patches now
18:40svenAt least I’m only responsible for maybe 10
18:40j`eysven: Im hopeful!
18:45axboewith a big backlog like that, useful to just try and flush out some of the independent ones. at least that makes you feel like you made progress ;)
18:47jannausven: t6000-dart could use a ping for robin
18:47svenI already pinged him off-list last week or so
18:48svenI guess I could also re-submit that ugly usb hack
18:49axboesven: oh, reminds me, didn't find any time to fiddle with shared tags yet
18:49axboesven: as a temporary improvement, adjusting AQ to 2 would make sense imho
18:50axboeI've only once seen higher AQ depth be useful, and that was under provisioning and with discard sizes being smaller where then having a higher admin queue depth made a difference
18:51axboesven: speaking of patches that could be pushed out... ;)
18:51axboesven: upstream would look kindly at the apple-nvme driver, as it can be combined with a "remove various weird apple workarounds or quirks" in the generic one
18:52svenoh, I agree. But nvme requires this rtkit protocol that’s also required for smc. I’m just waiting for marcan to finish that part
18:52axboeah ok, dependencies
18:52svenapple-nvme right now only works for platform devices fwiw
18:52axboeok, so we can't drop the generic quirks
18:53axboe(not a huge issue imho, it's not very invasive)
18:54axboeit's more of an hch'ism
18:54axboeas long as they stay out of the fast path, then I don't really care
18:55svenI considered adding a pcie backend to apple nvme but that driver is imho already big enough for the initial submission
18:55j`ey.. did I just realise that the first 'h' stands for Herr?
18:56axboesven: I think the platform one is a good start
18:58axboej`ey: if we ever end up at the same conference, I'll tell you the story of the time hch took my kids to watch the lego movie
18:59j`eyhaha ok :D
19:09svenyeah, it’s also been tested by a few people for at least weeks (and sometimes even months) now without any major issues
19:10jannaumarcan: spmi t8103 dts and a small fix for for the cpufreq driver https://github.com/jannau/linux/commits/spmi/work
19:10svenjust waiting for marcan and smc
19:11j`eyand smc is mostly about the bindings
19:11jannauworking poweroff is nice
19:12svenwe’ll have to figure out how to deal with the shared rtkit dependency but I guess in the worst case we just create an immutable branch somewhere so that both trees can merge it after review
19:12j`eyjannau: for the atomic thing did you actually get the sleep-during-atomic BUG, or just from manual inspection?
19:13jannauI hit a sleep during atomic BUG
19:16kettenisI hope the bindings we came up with will be fairly uncontroversial since they largely just use generic stuff
19:29axboesven: two subsequent boots I've run into aq timeouts
19:29axboehaven't seen that before
19:29svenhuh, don’t think I ever saw that one either
19:29axboeoh sorry, io queue
19:30axboehmm weird
19:30jannauthe same I saw with the old cpufreq patches?
19:31axboetimes out tag 10, then 2, then 3 and resets controller
19:31axboe30 seconds between each
19:31axboeboots fine, log into X and start a bunch of things
19:32svenguess I jinxed it by claiming no one had any major issues ;)
19:33svenso it doesn’t even completion poll the commands but they actually time out?
19:34axboeit does
19:34axboehang on, took a pic
19:34axboecompletioned polled for all 3, on last one it resets the controller, reset fails, IO errors ensue
19:34axboelet me know if you want to see the pic
19:36svenit should only reset when completion polling didn’t find that command
19:38axboeloading up the device with just reads doesn't lock it up
19:38svenlooks like the reset even fails
19:39axboedidn't reproduce now on starting tbird + ff
19:39axboeok it did
19:39sven-62 is timeout… so I guess the controller firmware doesn’t come up anymore? Weird
19:39axboesame thing again
19:40axboeok so tag 2 this time, it reset the controller
19:40axboeand it did come up fine
19:40axboe(after reset)
19:42axboedoes look like it's correlated with some IO load and CPU usage
19:42axboeeg loading firefox with bunch of tabs
19:46svenlooks like I can’t reproduce it on the M1. Let me try with the cpufreq driver
19:55axboesven: never saw it before adding that
19:55axboesven: question, what serializes writes to q->sq_db?
19:57svenI don’t think I need to serialize writes there. it’s not really a queue pointer but just “please start command with tag x”
19:57axboesven: yeah I guess as long as the writel is atomic it should be fine
19:59svenyeah. nvmmu_inval might need a lock though to serialize the write/read sequence
20:00jannaustill reproducible with setting the cpufreq governor to performance
20:01jannauseeing following messages after nvme reset https://paste.debian.net/1231040/
20:02axboegrabbing anv->lock around the issue does seem to fix it for me
20:04axboecould just be timing and I'll still hit it, but seemed to be trivial to hit before and seems solid now
20:17svenstrange, I can reproduce it with the cpufreq driver as well
20:17svenand adding that lock seems to fix it too
20:17sveni just don’t understand why
20:18timokrgr has quit [Quit: User left the chat]
20:19axboesven: trying to come up with a theory :)
20:19axboebut having unserialized issue seems like asking for trouble
20:20axboethough not quite sure why it needs to be serialized against irq
20:20axboehow relaxed is the arm memory model?
20:20kettenisvery relaxed
20:20svenvery. that writel has a dma barrier though
20:22axboewe _might_ just need an appropriate barrier so that it's visible on a different CPU
20:22axboebut not sure how that's even easily doable without serialization anyway
20:22axboethen again, not an ordering expert, I can always convince myself either way...
20:22axboeI'd sugggest just going with the lock + irq disable around the issue for now
20:22svenyeah, same here.. time to watch will deacon’s io ordering talk again ;)
20:23timokrgr has joined #asahi-dev
20:23axboesven: cuddle up with a cup of your favorite beverage and open Documentation/memory-barrier.txt ;)
20:24svenI assumed that writel to MMIO should be enough. It’s dma_wmb which should order it against the previous writes to normal memory and then just a single STR
20:25axboesven: my worry was between CPUs
20:25axboeissue on N, irq right after on N+1 - seems far fetched though
20:27svenhmmm… I wonder if there’s anything in the irq handler that could mess something up in that case
20:28axboethe spin unlock is a barrier, and as irq grabs the lock too first, I think it'd be safe
20:28axboeiirc you grab the lock around the whole irq handling
20:31svenyeah, I’m pretty sure most of the irq handling is inside the lock
20:31axboeanother thing I just noticed, there seems to be excessive tcb clearing in there
20:32axboesince we memset at alloc, do we really need to clear in both inval and submit?
20:32svenah, no
20:32svenonce should be enough
20:32axboeseems we can kill both of those, and just set flags to zero
20:32sventrue, that’s even better
20:32axboeI've got a patch for that too :)
20:33axboeand now booted
20:33sveni still wonder what exactly messes up the submission if it’s not inside a lock though
20:33axboesven: one idea
20:34axboeyou have two issues going on, one is half way through memcpy of command, other happens faster and submits the tag
20:34sventhe memcpys should be to two different tags though
20:34axboenot sure how the nvme hw side works if first tag isn't written yet
20:35axboebut only explanation I can find is that we trigger an interrupt for completion and don't find it, then it gets polled later
20:36sventhe hw first looks into the tcb at the specified tag, then reads the actual index into the “queue” (which is really more like an array here) and then reads the queue at that index
20:36svenI just set index = tag so that I don’t have to deal with a tcb index and a queue index
21:01axboecleanup of irq handling
21:01axboeunrelated
21:02axboeno new ideas on the other front, yet
21:05axboesven: assuming you're keeping the ->enabled state for adding suspend, because I don't think it's needed right now
21:13sveni think the enabled originally comes from NVMEQ_ENABLED in pci.c. i convinced myself at some point that i didn't need it but then went back a few days later because i convinced myself that i do need it and then decided to keep it for now just-in-case IIRC
21:14svenand that irq handling indeed looks better :)
21:29axboeyeah I don't think you need it right now, but for suspend you probably will. so I left it alone
21:29axboeI'll toss you that tcb patch too in a bit
21:32axboeyou can just fold in or however you prefer, I don't need attribution since it's not in tree yet
21:36axboetop 3 commits there
21:37axboe(and what a mess of patches...)
21:44j`eyaxboe: 'force' in apple_nvme_handle_cq is unused now
21:44axboej`ey: heh, it's funny I even thought of that while doing it
21:44axboethen forgot
21:45axboewill fix
21:46j`eythe code after /* last chance to complete any requests before nvme_cancel_request */ wont be useful if its removed
21:47c10l39 has joined #asahi-dev
21:48axboeyeah I know, plan was to refactor around it but then I forgot the last bit
21:54c10l3 has quit [Ping timeout: 480 seconds]
21:57jeffmiw has quit [Ping timeout: 480 seconds]
21:58jeffmiw has joined #asahi-dev
21:59axboe has quit [Ping timeout: 480 seconds]
23:00marcanjannau, axboe: I expected the cpu-map issue; also that fix is probably incorrect
23:00marcanwe actually need to renumber the nodes in cpu-map, since the code expects them to be consecutive
23:01marcanthe fix will only work if only core #3 is missing from each cluster
…