#
…

2021-05-09

00:05raster has quit [Quit: Gettin' stinky!]
00:06odmir has quit [Remote host closed the connection]
00:07odmir has joined #asahi
00:12odmir has quit [Ping timeout: 276 seconds]
00:18DrNapster has joined #asahi
00:20DrNapster has quit [Client Quit]
00:38odmir has joined #asahi
00:44odmir has quit [Ping timeout: 250 seconds]
00:51agnem has quit [Ping timeout: 240 seconds]
00:54agnem has joined #asahi
00:57odmir has joined #asahi
00:57odmir has quit [Remote host closed the connection]
00:58odmir has joined #asahi
00:59agnem has quit [Ping timeout: 246 seconds]
01:04odmir has quit [Ping timeout: 250 seconds]
01:22agnem has joined #asahi
02:23marcanpipcet[m]: what I meant is that what I'm doing should work; if it doesn't it means I'm doing something wrong, and I want to find out what it is
02:23marcansince i'm getting faults I can just trace the cause of the problem with symbols/etc
02:24marcan"known good" bisection debugging is a valid strategy, but not worth it here, where I already have better visibility
02:34phiologe has quit [Ping timeout: 276 seconds]
02:38phiologe has joined #asahi
03:14amwIf you are buried deep in exception traps perhaps it's work building support to trap (breakpoint?) earlier and ealier in the trap heap?
03:23awesomebing1 has quit [Quit: ZNC 1.8.2+deb2+b1 - https://znc.in]
03:24awesomebing1 has joined #asahi
03:59marvin24_ has quit [Ping timeout: 250 seconds]
04:00marvin24 has joined #asahi
04:05niv has quit [Quit: Free ZNC ~ Powered by LunarBNC: https://LunarBNC.net]
04:06niv has joined #asahi
04:55marcanamw: that's already the case
04:56marcaneach EL1 abort ends up in python anyway right now, it's just that initially I was passing through page faults on the assumption that they were normal (they aren't, not this early)
04:57marcanso in fact the state right now is data abort on bad address -> hypercall hook -> m1n1 -> python
04:57marcansince I just made it break to the debugger on everything
04:57marcanwas just changing one line of python :)
04:57marcanthat's the beauty of this all, the silly fast test cycle
04:57marcanhonestly the slowest part right now is just sending over the xnu kernel
04:58marcan(which is like a 100+MB image)
05:15VinDuv has joined #asahi
06:08roxfan2 has joined #asahi
06:11roxfan has quit [Ping timeout: 246 seconds]
06:16jeffmiw has joined #asahi
06:21jeffmiw has quit [Ping timeout: 260 seconds]
07:28svenpeterSo you’re saying we actually need to bring up the usb3 phy? :P
08:11jeffmiw has joined #asahi
08:16jeffmiw_ has joined #asahi
08:17amwSorry, I meant it's worth building support..., definitely like the idea that you can start trapping the page faults and only break on the very specific faults you are interested in
08:21amwPerhaps it's worth adding a tracing to help zero in on where it hangs. e.g. record the PC every certain time period or watch points?
08:22jeffmiw_ has quit [Remote host closed the connection]
08:23jeffmiw_ has joined #asahi
08:31jeffmiw_ has quit [Remote host closed the connection]
08:39amwIt's probably too much work but perhaps remote gdb protocol could allow easy stack dumps, breakpoints, watch points, but I don't know much about the effort required.
08:49sventhe protocol itself is simple
08:49sventhe limitation is the hv currently
08:50svenonce marcan can do everything that gdb requires from python all that's left is a few 100 lines of glue code to make gdbserver.py
08:50marcansven: actually compressing is about 1/3 of the time and calculating the checksum in python is about 1/3 of the time... and that's after switching from lzma to gzip :D
08:51marcanso it's pretty much fail all around
08:51j`eycan you just cache the compression for now..
08:51marcanand yeah the main thing missing is actually using the debug features so you can set breakpoints and such
08:52marcanwell I could stash the image somewhere in RAM that doesn't get touched, but right now I reboot the system after runs anyway to clean up properly so that wouldn't really help
08:52marcanI could append it as a m1n1 payload just to have it lying around there but... eh
08:52marcanit's not *that* bad waiting a few seconds for it to load
08:52marcan(it's still faster than the old linux kernels over serial)
08:52j`eyit just "looks slow"
08:52j`eyall those dots
08:52svenphew, no usb3 phy then :)
08:52eta(use zstd maybe? >_<)
08:52marcanmost of the time is *before* and *after* the dots!
08:52marcanand yeah but then the checksum is the bottleneck
08:53marcanand uh I really don't feel like implementing a cython module just for that :-)
08:53j`eyjust upload the first 10mb of macos :P
08:53marcanwonder if I can somehow use numpy or something to implement the checksum... or just disable checksums over USB anyway
08:53marcanI'm not sure chopping a kernel at the 10MB mark is a good idea :P
08:53j`eymarcan: are you patching the macOS image still?
08:54etamarcan: are you compressing in Python directly?
08:54svenpeterMaybe try pypy?
08:54svenpeterThat should speed up the checksum nicely
08:54* etacan attest to just giving up and piping the data through a `pigz` subprocess
08:54marcaneta: yeah I was thinking of shelling out to pigz for the gzip part too
08:55marcanthen the checksum is left. if it's dumb enough maybe I can numpy it
08:55marcanj`ey: not right now
08:55etathat sped up my gzip code like 6x vs the common lisp library I was using for this particular project
08:57marcanI think the checksum is linear? can probably get away with precomputing some powers of 31337 mod 2**32 and parallelizing this in numpy
08:57j`eywhats the checksum actually checking?
08:58marcanit's just a checksum over all uart messages
08:58j`eyI mean, what bit are you worried will go wrong?
08:58marcanwhich is completely redundant for USB because USB already has checksums
08:58marcanbut, well, it's the same protocol
08:58marcanoh also, there's ZLP issue with USB. there is *always* a ZLP issue with USB
08:58svenmarcan: just be glad it's not a crc :-P
08:58marcanI think I haven't had a USB project that didn't screw that up
08:58marcanneed to fix it
08:59marcan(if you read a multiple of 512 bytes of memory it hangs)
09:06VinDuvThe builtin zlib module seems to have support for adler32 and crc32 (implemented in C) but I suppose these wouldn’t be compatible with the current checksum you’re using
09:07marcanyeah, I try to keep backwards compat to avoid forcing people to reinstall over old versions during development
09:08marcanbut I think I can implement this with a few numpy primitives (which might be even faster than plain C if they vectorize)
09:18PendulumSwinger has quit [Ping timeout: 260 seconds]
09:22jeffmiw_ has joined #asahi
09:23vimal has joined #asahi
09:26jeffmiw_ has quit [Ping timeout: 260 seconds]
09:35vimal has quit [Quit: Leaving]
09:36amwIf the image isn't changed much after a run, you could implement rsync like update in place only the blocks with a changed checksum?
09:38matt6 has quit [Ping timeout: 252 seconds]
09:42matt6 has joined #asahi
09:56linkmauve has joined #asahi
10:04adamcstephens has quit [Read error: Connection reset by peer]
10:05adamcstephens has joined #asahi
10:14raster has joined #asahi
10:20marcanamw: I reboot the thing after each boot, and I would kind of expect RAM to be cleared or shuffled after a reboot
10:20linkmauve has quit [Ping timeout: 240 seconds]
11:06marcan< amw> Perhaps it's worth adding a tracing to help zero in on where it hangs. e.g. record the PC every certain time period or watch points? <- sorry, missed this, but that's how I found the PAN thing
11:06odmir has joined #asahi
11:07marcanI was setting up a timer to raise a FIQ after a certain time
11:07marcansetting it to 1 cycle was the closest I got to single stepping and found me the msr pan, #1
11:07marcanI still want to do proper single stepping etc, but so far this quick hack has worked well enough
11:11odmir has quit [Ping timeout: 268 seconds]
11:19choozy has joined #asahi
11:32amwOn the checksum backwards compatibility you can always add a new command extension, partially checksummed commands supported, if yes, send new style, command + unprotected data otherwise use old style fully protected command.
11:33amwI am thinking proxyclient/proxy.py checksum() used to protect commands is the problem?
11:35amwThe two part would have only the command not the bulk data protected, or perhaps just a unprotected blob operation or a switch to cheaper builtin checksum
11:36amwSorry just thinking out loud ideas not complete solutions :-)
12:04linkmauve has joined #asahi
12:04maknho has quit [Quit: WeeChat 2.3]
12:04maknho has joined #asahi
12:05maknho has quit [Client Quit]
12:05maknho has joined #asahi
12:18linkmauve has quit [Ping timeout: 250 seconds]
13:15VinDuv has quit [Quit: Leaving.]
13:29ephe_meral1 has joined #asahi
13:46marcanyeah, I thought of a flag like that for bulk data :)
13:46marcanbut really if I speed it up enough not to matter, who cares :-)
13:46marcanalso, there is another use case (which in practice I have hit more often than actual line noise): concurrency issues interleaving data and messages
13:46marcanwith SMP etc
13:47marcanso maybe I'll add checksum disabling once all that is properly SMP-safe :-)
13:47choozy has quit [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.]
14:30VinDuv has joined #asahi
15:06klaus has quit [Ping timeout: 252 seconds]
15:09klaus has joined #asahi
15:16arndI've finally gotten to the point of trying install the Mac mini, going through the newbie setup at https://github.com/AsahiLinux/docs/wiki/Developer-Quickstart, but I'm failing at the basic MacoOS install. I got an external TB3 drive so I can leave the working installation on the internal drive alone
15:16Bublik has quit [Ping timeout: 260 seconds]
15:17arndPartitioning and installing macos once works great, but after I install macos a second time, the first installation on the same drive fails
15:18Bublik has joined #asahi
15:20arndI tried erasing the first install on the external drive and reinstalling to it. The reinstall also worked, but after rebooting, it still fails with "The version of macOS on the selected disk needs to be reinstalled. Use Recovery to ereinstall macOS or select another startup disk."
15:22arndThe internal drive still boots, and the "Linux" partition on the external drive (which contains the unmodified MacOS 11.3.1) also boots, but the "Macintosh HD" on the external drive does not
15:23VinDuvThe page you linked indicates that even if you install macOS on external storage, the macOS kernel will be installed on internal storage (search for “M1 machines cannot boot from external storage.”)
15:23VinDuvMaybe that does some weird things if you have multiple external installs?
15:29arndCould be. I'm trying a third macos partition now to see which ones and how many I can boot after that
15:32Bublik has quit [Ping timeout: 252 seconds]
15:34Bublik has joined #asahi
15:53stemnic has quit [Quit: The Lounge - https://thelounge.chat]
15:56stemnic has joined #asahi
16:00Augur[m] has quit [Quit: Idle for 30+ days]
16:03klaus has quit [Ping timeout: 240 seconds]
16:05jeffmiw_ has joined #asahi
16:06klaus has joined #asahi
16:10jeffmiw_ has quit [Ping timeout: 260 seconds]
16:50balrogroxfan2: the instructions say that they do
16:50balrog(support two-machine debugging)
16:55roxfan2cool if so
16:56roxfan2 is now known as roxfan
17:25jeffmiw has quit [Ping timeout: 246 seconds]
17:28jeffmiw has joined #asahi
17:34jeffmiw has quit [Ping timeout: 260 seconds]
17:38jeffmiw has joined #asahi
17:40jeffmiw has quit [Read error: Connection reset by peer]
17:43balrogroxfan: the biggest caveat I see is that the instructions claim you have to use built-in ethernet or a very small list of tbt ethernet interfaces
17:44roxfanyeah probably the kbd code uses a built-in driver
17:44roxfan*kdb (kernel debugger)
17:45balrogwell, to be clear, the readme claims that only built-in ethernet works on Apple Silicon Macs
17:45balrogbut laptops don't have that, and doesn't the Mac Mini use a third party chipset anyway?
17:45balrogSo I'd expect a TBT adapter with the same or similar chip to work
17:46balrogAlso I've heard that the 10GbE Mac Mini is now available
17:46balrog(that's Aquantia AQC107/113 AIUI)
18:05odmir has joined #asahi
18:05marvin24_ has joined #asahi
18:06marvin24 has quit [Ping timeout: 276 seconds]
18:09odmir has quit [Ping timeout: 240 seconds]
18:31marvin24_ has quit [Ping timeout: 250 seconds]
18:31marvin24 has joined #asahi
19:09VinDuv1 has joined #asahi
19:10VinDuv has quit [Ping timeout: 240 seconds]
19:49Chainsaw has quit [Remote host closed the connection]
20:00VinDuv1 has quit [Quit: Leaving.]
20:24ephe_meral1 has quit [Ping timeout: 265 seconds]
20:39Bublik has quit [Ping timeout: 252 seconds]
20:41Bublik has joined #asahi
20:52TheJollyRoger has quit [Ping timeout: 240 seconds]
20:53TheJollyRoger has joined #asahi
21:05jeffmiw has joined #asahi
21:09jeffmiw has quit [Ping timeout: 252 seconds]
21:18matt6 has quit [Quit: Connection closed]
21:37raster has quit [Quit: Gettin' stinky!]
21:43Bublik has quit [Quit: Bublik]
22:00yrlf has quit [Quit: The Lounge - https://thelounge.chat]
22:01yrlf has joined #asahi
22:42m0drobert has joined #asahi
22:43marcanarnd: I know they still have some, um, breakage when moving external installs between devices; I wouldn't be surprised if multiple OSes on one external drive is broken
22:43marcanI don't think anyone has tested that so far
22:43marcanthe problem is all of this has to be provisioned on the internal boot policy and the kernels go in the iSC partition, and... there's a lot of surface area for bugs here
22:44modrobert has quit [Ping timeout: 265 seconds]
22:45marcanmaybe you can just stick to one OS externally (the "Linux" thing); are you planning to do anything with the Mac OS install on the external drive?
22:45marcanFWIW for mmiotrace/etc I plan to use the "Linux" partition; I haven't documented this yet but I expect people who want to play along with that to leave the Mac OS install under Linux and boot that when needed (probably best think of it as a throwaway OS)
22:46marcannot the other partition (though you could, if you install m1n1 there too, but it seems kind of redundant at that point)
22:46marcan(I don't want to cross-boot from one partition to another, that is bound to cause issues)
22:56arndmarcan: the MacOS on the internal disk isn't mine, it has to stay functional and I don't have root access on it. I have no need for MacOS on the external drive other than to try to follow your how-to document.
22:57arndFor some reason I assumed I had to build m1n1 under MacOS, but that is clearly not the case
22:58marcannot at all, I've never even tested that
22:58marcanm1n1 builds under linux
22:58marcanI assume it builds under macos if you have an ELF toolchain installed :)
22:59marcanthe point of the dual-boot in the guide is exactly what you want, to leave a functional macOS for normal tasks (or normal macOS dev)
23:00marcan(and also for upgrades and such, so you can upgrade 1TR that way)
23:02arndMy current theory of what went wrong with my external drive is that this all broke down because of nonmatching credentials: I had to provide an Admin password for the internal drive when installing to the external drive, but then after booting into that, the new Admin account lacked permissions to change the boot setup back to another external partition
23:03marcanah, but have you tried changing boot partition via 1TR?
23:03marcanI think that is supposed to do the provisioning properly
23:03marcanand yes the big issue with all of this is the credentials; each OS's credentials also end up in the SEP data store, which is how 1TR authorizes things like replacing the kernel...
23:04marcanthat's actually something I need apple to fix for our end-user install flow, at least with my current plan
23:04marcan(the drive adoption thing)
23:04arndAfter a partition gets messed up, 1TR also complains about missing permissions, in various ways
23:04marcanhm, interesting
23:04marcandid you first-boot into those OSes?
23:04marcan(to set up the first user)
23:05marcanI wouldn't be surprised if booting off of an external drive also forbids internal boot mode changes per policy in some way
23:06arnd1TR says something like "this partition has no privileged user, pick one now" when trying to boot back into a partition that stopped working
23:06marcan(this all sounds like legit bugs that could be reported to apple fwiw)
23:06marcanand then you can't pick?
23:06marcan(and ahh yeah, this sounds like related to machine owner stuff)
23:06arndI always did the First-Boot setup, and that worked, but just couldn't get back to the other partition after that
23:06marcanfwiw I don't fully understand this myself but I know there are dragons all around... :-)
23:06marcanwish apple documented it more clearly
23:07marcanalso, from 1TR, have you tried both the direct boot picker, and the startup disk selection under options?
23:07marcanI suspect they are not the same
23:08arndI suspect they wouldn't really provide support for users on external drives, afaik it only works by accident on TB3 disks but should be disabled on removable drives otherwise
23:09marcanI'm pretty sure booting macos from an external drive is supposed to work in general
23:09marcaneven USB or whatever
23:09marcanactually let me try that, I just found my USB SSD...
23:09marcan... it's just kind of broken in various ways because they rushed this M1 thing a *lot*
23:10marcanbut also, I *want* them to fix this stuff as it overlaps with feature I need for our installer :-)
23:10arndI did try various boot pickers (from running OS, from 1tr, from long pressing power button without 1TR and from the failed-boot screen)
23:10marcanthe basic idea for our installer I have is to make it look like a foreign external drive (just internally), which means credential adoption needs to work
23:11marcanfwiw long-press is still 1TR AIUI, the boot picker is just a fullscreen macos app
23:11arndI think I got into different scenarios there, but could never boot more than one of the external partitions
23:11marcanI think "options" just does a UI/session restart into the desktop
23:11marcanso I call that menu 1TR too
23:12marcan(just boot picker)
23:12marcanlet me try to reproduce what you see
23:13arndThere was clearly a difference between the long-press and the 1TR menu, in that the latter does not show USB drives, only TB3/nvme
23:14marcanhuh, interesting
23:14marcannow I'm wondering if it thinks nvme is internal and that's what causes it to be really confused
23:15marcanobviously iBoot can't boot from TB3, there's no way it initializes that
23:15marcanso those drives still need to be treated as external for boot policy purposes
23:15arndIt shows a picture of an external drive, but a different color compared to the USB one
23:19arndAnyway, I should go sleep now, will try more tomorrow
23:20marcangood night :)
…