#
…

2023-04-25

00:59nsklaus has quit [Ping timeout: 480 seconds]
01:08nsklaus has joined #asahi-dev
01:11korreckj328 has quit []
01:17nsklaus has quit [Ping timeout: 480 seconds]
01:20nsklaus has joined #asahi-dev
01:29nsklaus has quit [Ping timeout: 480 seconds]
01:38nsklaus has joined #asahi-dev
01:46nsklaus has quit [Ping timeout: 480 seconds]
01:55nsklaus has joined #asahi-dev
01:57kesslerd1pont has quit [Ping timeout: 480 seconds]
01:58rhysmdnz has joined #asahi-dev
02:00gabuscus has quit []
02:03nsklaus has quit [Ping timeout: 480 seconds]
02:06rhysmdnz has quit [Quit: Bridge terminating on SIGTERM]
02:06rhysmdnz has joined #asahi-dev
02:12nsklaus has joined #asahi-dev
02:14gabuscus has joined #asahi-dev
02:15Jamie has joined #asahi-dev
02:16kesslerdupont has joined #asahi-dev
02:16Jamie is now known as Guest12081
02:17SalimTer- has joined #asahi-dev
02:18salimterryli has quit [Ping timeout: 480 seconds]
02:20nsklaus has quit [Ping timeout: 480 seconds]
02:24kesslerdupont has quit [Ping timeout: 480 seconds]
02:28nsklaus has joined #asahi-dev
02:41kesslerdupont has joined #asahi-dev
02:41nsklaus has quit [Ping timeout: 480 seconds]
02:49kesslerdupont has quit [Ping timeout: 480 seconds]
02:50nsklaus has joined #asahi-dev
02:59nsklaus has quit [Ping timeout: 480 seconds]
03:08nsklaus has joined #asahi-dev
03:17nsklaus has quit [Ping timeout: 480 seconds]
03:25nsklaus has joined #asahi-dev
03:35nsklaus has quit [Ping timeout: 480 seconds]
03:44nsklaus has joined #asahi-dev
03:50salimterryli has joined #asahi-dev
03:51SalimTer- has quit [Ping timeout: 480 seconds]
03:52nsklaus has quit [Ping timeout: 480 seconds]
04:01nsklaus has joined #asahi-dev
04:09nsklaus has quit [Ping timeout: 480 seconds]
04:17nsklaus has joined #asahi-dev
04:26nsklaus has quit [Ping timeout: 480 seconds]
04:34nsklaus has joined #asahi-dev
04:42nsklaus has quit [Ping timeout: 480 seconds]
04:52nsklaus has joined #asahi-dev
05:01nsklaus has quit [Ping timeout: 480 seconds]
05:03marcan320x240 maybe, definitely nothing lower
05:05marcanaxboe: none of those look like a problem, need a full log
05:08nsklaus has joined #asahi-dev
05:17nsklaus has quit [Ping timeout: 480 seconds]
05:22salimterryli has quit [Remote host closed the connection]
05:24salimterryli has joined #asahi-dev
05:26nsklaus has joined #asahi-dev
05:35nsklaus has quit [Ping timeout: 480 seconds]
05:42nsklaus has joined #asahi-dev
05:51nsklaus has quit [Ping timeout: 480 seconds]
05:55kesslerd1pont has joined #asahi-dev
06:01nsklaus has joined #asahi-dev
06:03kesslerd1pont has quit [Ping timeout: 480 seconds]
06:10nsklaus has quit [Ping timeout: 480 seconds]
06:18nsklaus has joined #asahi-dev
06:26nsklaus has quit [Ping timeout: 480 seconds]
06:33nsklaus has joined #asahi-dev
06:42nsklaus has quit [Ping timeout: 480 seconds]
06:43bps has quit [Ping timeout: 480 seconds]
06:48MajorBiscuit has joined #asahi-dev
06:50nsklaus has joined #asahi-dev
06:58nsklaus has quit [Ping timeout: 480 seconds]
06:59MajorBiscuit has quit [Quit: WeeChat 3.6]
07:08nsklaus has joined #asahi-dev
07:11jannaudoes pmgr-misc exist on die 1? I'm getting transalation faults on 0x228e20c000 and 0x228e20c800 on m1 ultra
07:11bps has joined #asahi-dev
07:15MajorBiscuit has joined #asahi-dev
07:16nsklaus has quit [Ping timeout: 480 seconds]
07:18jannauI see dart translation faults for 583008000.iommu on s2idle and the nvme didn't boot on the second resume
07:18jannauno problems with dcp though
07:20marcanno idea, haven't tested on ultra. I *assumed* it exists but who knows :p
07:20marcanyou're getting... translation faults?
07:20marcannot SErrors?
07:25nsklaus has joined #asahi-dev
07:25chadmed has quit [Remote host closed the connection]
07:27marcanuhh that looks like a hypervisor bug
07:27marcanthat FAR doesn't make any sense
07:29marcancould even be a CPU bug? like somehow ending up with the IPA instead of the VA in FAR?
07:33nsklaus has quit [Ping timeout: 480 seconds]
07:34kesslerdupont has joined #asahi-dev
07:34marcanjannau: anyway yeah, doesn't seem to exist on die1 so let's just drop it
07:36jannauif you move it, it has a t6020-pmgr-misc in .compatible
07:37marcanthanks, fixing
07:38marcanI'm guessing that is a SoC-global block and those clocks/pstates also forward to die1 automatically then
07:39jannaudid the sleep/shutdown tracing made the system slower under the hypervisor? boot timing seems to be much slower than I remember and macos felt slow as well
07:39marcanit shouldn't
07:39marcanbut I could have screwed something up :)
07:40marcanI did try to speed things up on M2+ though
07:42kesslerdupont has quit [Ping timeout: 480 seconds]
07:42nsklaus has joined #asahi-dev
07:43marcanjannau: should be fixed
07:43marcan(the die1 thing, not the HV thing)
07:43marcanIIRC the Ultra was always kind of slow under the HV though, but there may be some pathological issue there
07:51nsklaus has quit [Ping timeout: 480 seconds]
07:52jannaumight be a little bit faster with cbd9b7b9e1724244 but not much so I might misremember
07:58Z750 has quit [Quit: Ping timeout (120 seconds)]
07:59Z750 has joined #asahi-dev
07:59nsklaus has joined #asahi-dev
08:11nsklaus has quit [Ping timeout: 480 seconds]
08:13marcanhm, it could be cpufreq is broken in the guest with clpc=0 and we don't bump the pcores, yeah
08:13marcanfeel free to call cpufreq_init() in the hv init path if it helps
08:14marcanalso I haven't tested on ultra but at least on M2 Max macos seemed to just... not use the pcores?
08:15marcannot sure why or if that's a regression
08:21nsklaus has joined #asahi-dev
08:29nsklaus has quit [Ping timeout: 480 seconds]
08:30nsklaus has joined #asahi-dev
08:35cylm has joined #asahi-dev
10:05kesslerdupont has joined #asahi-dev
10:13kesslerdupont has quit [Ping timeout: 480 seconds]
10:13chadmed has joined #asahi-dev
10:17chadmed has quit [Remote host closed the connection]
10:22chadmed has joined #asahi-dev
10:25c10l has quit [Quit: Bye o/]
10:26c10l has joined #asahi-dev
10:43jannaucpuidle makes linux boot under HV slow. with cpuidle.off=1 in the commandline it's 2-3 times faster
10:47nsklaus has quit [Quit: WeeChat 3.8]
10:48marcanok, that makes sense because it'll cause a ton of HV traps
10:48marcanand on M1, the HV ends up interrupting the CPUs at 1000Hz anyway for dumb reasons
10:49marcanthose normally don't serialize, but they do via the cpuidle stuff
10:49marcanI can probably fastpath the cpuidle stuff a bit so at least it doesn't serialize
10:55kesslerdupont has joined #asahi-dev
11:09kesslerdupont has quit [Ping timeout: 480 seconds]
11:09marcanjannau: pushed a thing, let me know if it helps
11:14kesslerdupont has joined #asahi-dev
11:14jannaumarcan: it does, might be even a little faster than cbd9b7b9e1724244 with cpuidle disabled. thanks
11:31kesslerdupont has quit [Read error: Connection timed out]
11:36nsklaus has joined #asahi-dev
11:36jannauI see the same on ultra, macos (12.3) doesn't use the p-cores with clpc=0. without it does
11:40ChaosPrincess has quit [Quit: WeeChat 3.8]
11:41ChaosPrincess has joined #asahi-dev
11:42kesslerdupont has joined #asahi-dev
11:59kesslerdupont has quit [Quit: leaving]
12:17i509vcb has quit [Quit: Connection closed for inactivity]
12:19c10l has quit [Quit: Ping timeout (120 seconds)]
12:20c10l has joined #asahi-dev
13:17axboemarcan: X restarting on resume, is that expected in current asahi-wip?
13:24_jannau_axboe: I couldn't reproduce it (using HV with console_no_suspend and sddm as X client) so quite a few things which could explain different behavior
13:31axboethis is gdm + X
13:31axboesorry lightdm
13:32axboelooks like X is still running, but my session is just gone and I get a lightdm login screen
13:33axboeI did update m1n1 and u-boot at the same time, I can try my previous kernel and see if it happens there too. was just assuming it's a kernel side thing with dcp
13:46_jannau_could be still the X server of your session which was killed. DCP shouldn't have changed much for existing installs
13:47axboeyeah, what I mean is that X itself didn't restart, it's still there and same pid, but something made it exit the session
13:47_jannau_main change was supporting the 13.3 firmware for m2 devices but is mostly separate from the existing support
13:48axboemight be xfce4 triggering it. I don't recall seeing those "display went away" on suspend before, but it could just be that I never looked
13:48axboelet me boot the old kernel and give it a spin
13:48_jannau_there should be be 2 X sessions after you logged in. the lightdm one and you user session
13:49_jannau_how long was the system suspended? did you enable the new apple cpuidle driver?
13:50axboeold kernel doesn't boot anymore, so can't test that
13:50axboeI did enable the new cpuidle
13:51axboeseems to happen regardless of how long it was suspended
13:51axboejust eg 10 min will do it
13:51axboeso just did a suspend and resume after 10 seconds
13:52axboeand that works fine
13:52_jannau_I didn't keep the system suspended that long. I'd guess 30 seconds maximal
13:53axboethat may be why
13:53axboeusually if I suspend it's because I'm heading to the office, so probably 20-30 minutes min suspend
13:53axboeor overnight
14:04tobheaxboe: I think I have seen something similar before, even on wayland. out of curiosity: does your wifi still work after wakeup?
14:10axboewifi has always been a bit hit or miss for me. I'd say that 9/10 wakeups wifi is fine, 1/10 it doesn't come back and it needs a reboot
14:10axboebut this one is different, I've never had my X session go away
14:13tobhethe few times this has happened to me it was usually both at the same time so I assumed it might be gnome related
14:16Knedlik has joined #asahi-dev
14:17cylm has quit [Ping timeout: 480 seconds]
14:20cylm has joined #asahi-dev
14:40KnedlikHey guys, I would like to use C++ for the touchbar Fxx buttons I was tasked with earlier. What should I use? I assume the DRM has some headers, but what should I use for touch input?
14:46enron has quit [Quit: Ping timeout (120 seconds)]
14:46nopeslide13 has quit []
14:46j`eylibevdev I think
14:46enron has joined #asahi-dev
14:46nafod0 has joined #asahi-dev
14:48nafod has quit [Read error: Connection reset by peer]
14:48nafod0 is now known as nafod
14:49nopeslide13 has joined #asahi-dev
14:55KnedlikOkay I'm getting a bit confused reading through all the DRM-related stuff on the internet... how do I use it to draw on the touchbar?
15:05Knedlik has quit [Quit: Konversation terminated!]
15:06kesslerdupont has joined #asahi-dev
15:07marcanaxboe: we've long had that thing with systemd-journald crashing on long suspend due to some watchdog thing, but that's exactly what cpuidle should have fixed...
15:07marcanKnedlik: that is a question with many answers...
15:08_jannau_it didn't, systemd-journal still crashed for me after waking from 15 minutes of suspend
15:08_jannau_I'll look at it tonight. X session survived though
15:09marcan_jannau_: does the smc power log stuff log during suspend?
15:09marcanbecause if it doesn't timekeeping is *definitely* stopping now
15:09_jannau_wasn't enabled
15:09marcanalso does the kernel log timestamp jump during suspend, or does it skip the time?
15:10_jannau_timestamps jump but I'd expect those to use boot time
15:10marcanno, they don't jump for me
15:10marcanso something is wrong
15:11bps has quit [Ping timeout: 480 seconds]
15:13kesslerdupontchadmed: I'll take a look today at speakersafetyd and see what I can do
15:13_jannau_hypervisor + no_console_suspend?
15:14_jannau_log_power does nothing on the studio (desktop systems)
15:16marcanoh wait, this probably isn't going to ever work in the hypervisor
15:16marcanbecause it basically triggers forever spurious wakeups, which will cause the kernel to repeatedly re-enable and re-disable timekeeping
15:17marcanI might be able to cheat it
15:23_jannau_no timestamp jump on bare metal
15:24marcan_jannau_: pushed a thing to fix this on the *kernel* side under the HV
15:24marcanyou also probably want hv.p.hv_set_time_stealing(0, 1) to prove that it isn't just time stealing warping time
15:24marcanwith both of those no timestamp jump under the HV either, as expected
15:25marcanshould also give a nice performance improvement on top even
15:25marcan(plus it actually tests deep sleep under the HV which is always nice)
15:26marcanso... if timekeeping is actually stopping, why the hell is journald crashing?
15:26marcanor did you only see that under the HV?
15:27_jannau_that was only under HV with time clearly progressing
15:28marcanyeah, just tested the userspace view, seems right now under the HV
15:28marcanso. not an issue but not what axboe is seeing either (assuming he wasn't running under the HV)
15:29MajorBiscuit has quit [Ping timeout: 480 seconds]
15:30_jannau_I doubt that very much for the running to the office use case
15:32rosefromthedead has joined #asahi-dev
15:35axboejust a regular joe user, no HV
15:40marcanaxboe: I think we're going to need more info/logs then... I don't have any good ideas of what to look for right now :/
15:41marcanI wonder if I can cheat it with the HV by adding time instead of subtracting time...
15:41marcanthat might be a good test
15:43axboelet me boot the latest, and will be happy to grab whatever you'd want in terms of logs or messages
15:44_jannau_I've seen a dcp call ordering issue on wakeup but everything still worked. so probably some kind of race where at least one call is not needed
15:44axboeback in a few min
15:45bps has joined #asahi-dev
15:47axboeok back up - I'll try and suspend for 5 min and see what happens
15:49marcanI'm adding an explicit timewarp op to the HV so I can test this, but if it really has to do with time elapsed I can only guess it's a userland bug with the calendar time jumping somehow
15:53axboe5 min is up, maybe I should do 5 more to be sure
15:54axboewhat do you want to see if it fails?
15:55marcanX log, general syslog etc?
15:55marcanwe don't know what we're looking for, anything that might be relevant :)
15:56axboeI hear ya, I'll try and poke a bit and see if I can figure out what's crashing
15:57axboeI already checked the X log previously and found nothing interesting
15:57axboeok 9 min, let's try and bring it back
15:57axboeyep, it happened
15:57marcanjust tested under the HV on the M1 Studio, warped time 10 hours forwards and nothing bad happened (kde on x11)
15:58axboeinterestingly, seemed like it was already resumed when I opened the lid
15:58axboehuh maybe it rebooted
15:58marcanis this with the GPU stuff?
15:58axboe0 min uptime
15:59axboethis is with current asahi-wip, running mesa from a month ago or so (updated whenever the api changed)
15:59marcancould be the GPU stuff is also breaking due to its power domain shutting down. can you add apple,always-on; to the ps_gfx power domain and see if that helps?
16:00axboealways forget what model this is...
16:00marcancat /proc/device-tree/compatible
16:00axboet6001, thanks
16:01axboelooks like it's in t6002.dtsi
16:01j`eyaxboe: also says at the top of dmesg: 'Machine model: '
16:01marcanps_gfx should be in t600x-pmgr.dtsi
16:01axboeso add to the DIE_NODE() there?
16:01marcanyeah, next to the other properties
16:02marcanthough that might actually kind of break for die1, but that should only cause some warnings at most
16:02marcansince I don't think gfx for die1 is used
16:02axboedone, will reboot in a sec
16:05marcanI wonder if it's something like gfx-asc goes down unexpectedly, it has a backdoor pipe to SMC, SMC is sending it notifications for some reason, the FIFO fills up and SMC panics and reboots the machine
16:06axboesuspended, let's try 30 sec first
16:06axboebriefly showed the screen on resume, then rebooted
16:07marcanokay, we need any logs you managed to get from that then
16:07marcanthat looks like something goes very wrong with SMC or similar, on resume
16:07marcan(or kernel panic if you have it set to reboot on panic)
16:07axboedon't enable that
16:08axboelast thing syslog has before a bit of garbage is "entering sleep state suspend"
16:08axboeso not very useful
16:09marcanah wait, this could be the pmgr-misc thing
16:09marcanaxboe: in t600x-die0.dtsi, comment out the whole pmgr_misc block
16:10nyx_o has quit [Ping timeout: 480 seconds]
16:10axboeremove the ps_gfx always on or leave it?
16:10marcanleave it just in case
16:11axboewill do, updating it now
16:12axboeI'm still doing the whole osx recovery thing for dtb updates like probably a moron (assuming there's a better way now)
16:12marcanumm... you know we've been shipping u-boot for that for a year now right? :p
16:13marcan(and m1n1 chainloading)
16:13axboeI haven't changed my ways :)
16:13axboenever updated that side, will do once the m2 can get used
16:14marcanI hope at least you're on the right stub version :p
16:14axboewell...
16:14axboeit's up, suspending
16:14_jannau_dcp would complain if not
16:15axboeit came back, I entered password to unlock, and then 1 sec later it rebooted
16:15marcanI hope journald saved a log from that
16:15marcancan you see if anything is in there?
16:16axboeI got something
16:17marcanso IIRC the SMC reboot watchdog is like 30 seconds or something, and if this is along those lines it might be crashing directly on suspend
16:18marcanif so, try 15 seconds and you should have at least as much time to sync to disk and such
16:18bps has quit [Ping timeout: 480 seconds]
16:18axboe2023-04-25T10:14:57.380543-06:00 m1max kernel: [ 33.664422] nvme-apple 393cc0000.nvme: RTKit: syslog message: cmd.c:7861: NVMe shutdown start seg->lba: 0, seg->size: 0
16:18axboe2023-04-25T10:14:57.380548-06:00 m1max kernel: [ 33.801967] nvme-apple 393cc0000.nvme: RTKit: syslog message: cmd.c:7874: seg->lba 0 saveCtx 1 took 137 ms
16:18marcanthat's normal
16:18axboe2023-04-25T10:14:57.380550-06:00 m1max kernel: [ 33.834543] macsmc-rtkit 290400000.smc: RTKit: syslog message: apComms.cpp:373: SMC HID Event: 03 00 01
16:18axboe2023-04-25T10:14:57.380552-06:00 m1max kernel: [ 33.834560] macsmc-hid macsmc-hid: Lid wakeup
16:18axboe2023-04-25T10:14:57.380554-06:00 m1max kernel: [ 33.835102] apple-pmgr-pwrstate 28e580000.power-management:power-controller@288: PS mca0: Failed to reach power state 0xf (now: 0x24f)
16:18axboe2023-04-25T10:14:57.380557-06:00 m1max kernel: [ 33.835223] apple-pmgr-pwrstate 28e580000.power-management:power-controller@290: PS mca1: Failed to reach power state 0xf (now: 0x24f)
16:18axboe2023-04-25T10:14:57.380559-06:00 m1max kernel: [ 33.835339] apple-pmgr-pwrstate
16:19marcanyeah that's all normal, also pastebin?
16:19axboewell shoot, that's all I get before the next line is from the next boot
16:19axboeyeah sorry, will pastebin
16:19marcandid it stop right there mid line?
16:19axboelet me put it in a pastebin
16:19marcanso it can't be SMC crashing before the wakeup because (duh) then we wouldn't wake up
16:20axboeplanned maintenance
16:21axboethat's from starting suspend, and then last line is new boot
16:21marcanok yeah that's... not useful :(
16:22marcancan you try again and see if you can sneak in a sync or something?
16:22marcanobviously that didn't flush to disk everything that happened while it was alive
16:22axboesure, let me find the resume script and add a sync
16:22marcanyou could just do `echo mem > /sys/power/state ; sync` probably
16:22axboelet's try
16:23marcanmaybe sync in a loop :)
16:23axboessh in with dmesg in a loop too :)
16:23axboeok here goes
16:24axboethat resumed fine
16:24marcan`dmesg -w` is your friend :)
16:24axboewow, after all these years
16:24axboetrying again
16:25axboeworked again, damn
16:25axboelet me try the lid
16:26marcanheisenbug \o/
16:27axboetried lid again and it just insta booted when it came out
16:28axboenothing in syslog, last one is the entering sleep state
16:29axboeonce more, got X back for half a second, then reboot
16:29axboelet's see if syslog got anything
16:29axboeI miss my servers with serial console :/
16:29marcanyou can get a serial console on these but..
16:30axboethere's something, let me pastebin it again
16:34axboethat's from start resume, to again last line being from the next boot
16:35marcanok, so the borked i2c isn't great but that doesn't excuse a reboot
16:37marcanI don't see anything fatal in there :(
16:37axboeah crap
16:37axboeI do have panic on oops
16:37axboeoh no scratch that, it's just the timeouts
16:37axboeactual PANIC_ON_OOPS isn't set
16:38marcanjust to confirm, CONFIG_BACKLIGHT_GPIO isn't set?
16:40axboeit's not
16:40axboeBACKLIGHT_CLASS_DEVICE=y, BACKLIGHT_PWM=y
16:41axboethat's it for backlight
16:41axboeoh it's too big
16:42marcanok, here's a random idea: make ps_uart0 apple,always-on;
16:43* axboedoes the osx dance
16:44marcanwhy do you like hurting yourself so much :p
16:45Jamie has joined #asahi-dev
16:45axboeyeah...
16:45rhysmdnz1 has joined #asahi-dev
16:45Jamie is now known as Guest12142
16:45marcanis this the 14" or the 16"?
16:46axboelet's try again
16:47axboecame back fine
16:47axboelet me give it 5 cycles
16:49rhysmdnz has quit [Ping timeout: 480 seconds]
16:49marcannext move that always-on from ps_uart0 to ps_sio
16:49axboe4 so far, it's fine
16:49axboegot some hci0 complaints on the last one, timeouts
16:49axboebut it's up
16:49marcanyeah bluetooth breaks, we know that
16:49Guest12081 has quit [Ping timeout: 480 seconds]
16:50marcanI think some coprocessor needs ps_sio and is exploding when we shut that down
16:50axboewill do a few min for the last one, but seems promising so far
16:51axboeI'll move it to ps_sio after this one and re-run
16:51axboereboot on the slightly longer one...
16:51axboeand again looks like suspend was the culprit, because it was sitting instantly in lightdm login when I opened the lid
16:52marcanoh, it crashed again even with that change?
16:52axboeit did...
16:52axboelast line in syslog is the entering suspend again
16:53marcanI'm running out of ideas... AFAIK the coprocessors that can nuke the system like that are SMC and the CIO stuff (cc sven), but the latter shouldn't be enabled yet...
16:53marcanand SMC is clearly alive since wakeup works
16:53marcanDCP also clearly works
16:54axboewas running asahi-wip based on 6.3-rc4 before, with mainline merged
16:54axboeso that one was fine
16:54marcanyeah but I don't think I made any changes that could cause this other than the cpuidle stuff :(
16:54marcanyou could try disabling cpuidle
16:54marcancpuidle.off=1
16:55axboelet's try
16:55marcancan you put a #define DEBUG at the top of drivers/soc/apple/apple-pmgr-pwrstate.c too?
16:55axboewant that first, or disable?
16:55marcanyou can do both
16:55axboewill do
16:56marcanif cpuidle fixes it, it's probably unrelated. if it doesn't, try to get me another log where you get some stuff from resume (so at least we have a complete log of the suspend process)
16:57axboewill try, I'll try the manual suspend rather than lid and see if I can get something
16:57marcanyeah the result should be the same either way, I don't care if it doesn't crash at that point
16:57marcanI just want a trace of what your system is powering down
16:57axboeup with it disabled and with DEBUG, will let it sit suspended for a minute
17:00axboe3 so far, fine
17:03axboe5 now, anywhere from 30 seconds to 2 min
17:03axboeall good
17:05marcanjust to be sure, which version of m1n1 are you using?
17:05axboecurrent -git
17:05marcanas of at least a couple days ago I take it
17:05marcanwhat's your boot chain right now?
17:06axboelet me resume and give you the sha
17:06axboe9e...ff
17:06axboethat one
17:06marcanif it's recent you have it in /proc/device-tree/chosen/asahi,m1n1-stage2-version :p
17:07marcangot a log of the pmgr stuff?
17:07axboeI still have the same bootup as from summer last year
17:07axboegenerate the blob with m1n1, u-boot, dtb
17:07axboeosx recovery, etc
17:07axboedmesg | grep pmgr?
17:08marcanfrom a suspend cycle, yeah
17:08marcanor just paste the whole dmesg so I can see what when
17:08axboethat's it from boot and the suspends I did so far
17:10marcanI don't see the debugs?
17:10marcanmaybe you need to boot with `debug`?
17:10axboewas assuming it is because it is off, will try with 'debug'
17:14marcananother random guess: make ps_msg apple,always-on
17:15axboenope, will just change them to dev_info
17:15axboeso far it doesn't reproduce with cpuidle.off=1, I can try that and kill the cpuidle off?
17:16kesslerdupont has quit [Ping timeout: 480 seconds]
17:16axboeok got a bunch of pmgr stuff now
17:16marcanI want to see the pmgr stuff first
17:17marcanit should be the same regardless of what cpuidle is doing
17:17axboesuspended it, resume and will paste it
17:17axboefrozen on resume, X still on
17:17kesslerdupont has joined #asahi-dev
17:18marcanthat sounds like a separate issue..
17:18axboeI'll dump what I got
17:18axboeit's ssh'able
17:19axboeah log too small, damn
17:19axboelikely not interesting, sorry
17:20marcanyeah I need the suspend cycle
17:20marcanspi is going to spam (that's the keyboard/trackpad)
17:20axboewant me to add always_on for ps_msg first or retry with larger dmesg?
17:21marcanadd that and remove the cpuidle.off and see if it repros, if it does either catch enough of a log or disable cpuidle again and catch it that way
17:23marcanthere's also one interesting thing we can try. set macOS as the default boot OS, do a one-time boot into Linux, repro, and see if macOS gives you any panic logs. I think it can catch certain system engine/management errors.
17:23marcan(if it finds something it will pop it up on startup)
17:26axboebuilding with larger dmesg and not disabling cpuidle and doing those first, then we can try that if it still craps out without being able to capture anything
17:26axboehave ps_msg always on too
17:29axboe2nd resume came back with it rebooted, by the time the lid was opened
17:31axboenothing in syslog
17:31axboesee t entering suspend at 11:28:31.30 and booting a new kernel at 11:19:26.14
17:31axboeand already up when I opened the lid
17:34marcanyou mean 29?
17:34svenyeah, cio won’t be enabled yet so it shouldn’t be able to reset the machine
17:34axboesorry yes 29
17:34svenI’ve never seen a reset with just atcphy without cio
17:34svenor, well, I have seen that but only when I messed up the init sequence which shouldn’t happen with the Linux driver
17:35kesslerdupont has quit [Ping timeout: 480 seconds]
17:37axboeit certainly looks like it's suspend somehow
17:37axboe55 seconds is probably roughly the time it takes to power cycle, and go through all of boot
17:38axboetriple fault during suspend?
17:38marcanthat's not a thing on arm :p
17:38marcanso, pmgr log?
17:38axboethe osx default thing
17:40axboethere's a ton of stuff before the entering suspend, let me pastebin it in case there's anything interesting
17:40marcanno, I mean some dmesg of pmgr changes through a suspend cycle
17:40marcanas long as I can see the whole suspend cycle (i.e. there should be one line of wakeup, otherwise you probably didn't get anything) we should be good
17:40marcan*didn't get everything
17:41axboebut there is no full cycle, it looks like it's power cycling right after having entered suspend
17:41marcandidn't you get one good cycle at least?
17:41axboeI can try one now, see if it's different
17:41marcanif you got one good cycle it should be in the journal too
17:44axboerebooted again
17:44axboelet's try again...
17:45axboeshorter suspend worked, paste coming up
17:46lebakassemmerl has joined #asahi-dev
17:46lebakassemmerl has quit [Remote host closed the connection]
17:47axboethat's around the suspend + resume
17:49marcanok, so you have a fairly short list of power domains that shut down between `Suspending console(s)` and `Lid wakeup`
17:50axboewthis is with pmgr_misc removed, and always-on for ps_uart0, ps_gfx, and ps_msg
17:50marcanI really need to get some sleep, but if you want to continue debugging on your own: make all of those always-on, and see if that fixes things. if it does, manually bisect (though really gpio is the only one there that I'd even slightly suspect)
17:50axboesleep well!
17:50marcanI also pushed a thing to the linux.git:pmgr-stuff branch that you might want to cherry pick and see if it helps
17:50axboeI'll poke at it tonight, I should also try and get some work done...
17:51marcanif making everything on that list always-on doesn't fix it either, then we have some weirder problem where cpuidle is broken, but it somehow only triggers during sleep for unknown reasons (e.g. because it sleeps more then, or because other stuff in the system is going lower power anyway)
17:51marcanalso do try the macos default thing, that might point at something useful
17:51jannauI'll try to reproduce with serial on j314c (14" m1 max)
17:52marcanbasically my theory right now is that the CPUs going into deep idle is triggering some machinery that we otherwise haven't triggered yet, and it hates something we're doing, and something critical crashes and takes down the machine
17:55marcanthis *may* be related to how I couldn't get the CPUs properly fully shut down (with loss of MMIO) but could when I let macOS do it
17:55marcanit's entirely possible there is some init we're missing
17:56marcanaxboe: so basically if making everything always-on doesn't help, do the macos thing and whatever the result of that is, I think that's as much as you can do at this point
17:56marcanin that case I will continue poking at the sleep magic thing to see if it's related
17:57marcangood night :)
18:01axboesounds good!
18:05Cyrinux9 has quit []
18:08Cyrinux9 has joined #asahi-dev
18:16WindowPain has joined #asahi-dev
18:21WindowPa- has quit [Ping timeout: 480 seconds]
18:23MajorBiscuit has joined #asahi-dev
18:31cylm has quit [Ping timeout: 480 seconds]
19:34jannauaxboe: I haven't been able to reproduce yet :(
19:41jannaucan you think of anything out of the ordinary? you're using xfce4 if I remember correctly and I gather you suspend/resume by closing/opening the lid
20:06deteg1337 has joined #asahi-dev
20:06deteg1337 has quit []
20:22pharonix71 has quit [Read error: No route to host]
20:27flying_sausages has quit []
20:38flying_sausages has joined #asahi-dev
20:41kesslerdupont has joined #asahi-dev
21:05bluetail has quit [Quit: The Lounge - https://thelounge.chat]
21:22Z750 has quit [Quit: Ping timeout (120 seconds)]
21:22Z750 has joined #asahi-dev
21:38zzywysm has joined #asahi-dev
22:03kesslerdupont has quit [Ping timeout: 480 seconds]
22:16WindowPain has quit [Quit: ZNC 1.8.2 - https://znc.in]
22:17WindowPain has joined #asahi-dev
22:33kesslerdupont has joined #asahi-dev
22:43psykose_ has joined #asahi-dev
22:46psykose has quit [Ping timeout: 480 seconds]
22:54kesslerdupont has quit [Ping timeout: 480 seconds]
23:23bluetail422 has joined #asahi-dev
23:24bluetail422 has quit []
23:24bluetail422 has joined #asahi-dev
23:25bluetail422 has quit []
23:25bluetail422 has joined #asahi-dev
23:26bluetail422 has quit []
23:26bluetail422 has joined #asahi-dev
23:26bluetail422 has quit []
23:27bluetail422 has joined #asahi-dev
23:27bluetail422 has quit []
23:28bluetail422 has joined #asahi-dev
23:28bluetail422 has quit []
23:29bluetail422 has joined #asahi-dev
23:29bluetail422 has quit []
23:30bluetail422 has joined #asahi-dev
23:31kesslerdupont has joined #asahi-dev
23:31abd has joined #asahi-dev
23:32bluetail422 has quit []
23:33bluetail422 has joined #asahi-dev
23:34bluetail422 has quit []
23:35bluetail422 has joined #asahi-dev
23:35bluetail422 has quit []
23:35bluetail422 has joined #asahi-dev
23:36bluetail422 has quit []
23:36bluetail has joined #asahi-dev
23:38bluetail has quit []
23:38bluetail has joined #asahi-dev
23:38bluetail has quit []
23:39bluetail has joined #asahi-dev
23:39bluetail has quit []
23:40bluetail has joined #asahi-dev
23:41bluetail has quit []
23:51abd has quit [Ping timeout: 480 seconds]
23:53bluetail has joined #asahi-dev
23:57bluetail has quit [Remote host closed the connection]
23:58bluetail has joined #asahi-dev
…