#
…

2022-02-14

00:12chengsun has joined #asahi-gpu
00:19chengsun_ has quit [Ping timeout: 480 seconds]
01:07chadmedthe russians arent content with just invading ukraine, they want our IRC channels too
03:36PhilippvK has quit [Ping timeout: 480 seconds]
03:37phiologe has joined #asahi-gpu
04:27user982492 has joined #asahi-gpu
06:01chadmed has quit [Remote host closed the connection]
06:29user982492 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
06:34chadmed has joined #asahi-gpu
07:05mrkajetanp has quit [Ping timeout: 480 seconds]
09:22phireI've got the launch command: 00001002 00018000 00000020 00000001 00000001 00000020 00000001 00000001 6000217c
09:23bisko has joined #asahi-gpu
09:23phirebut there is nothing actually mapped at 0x18000 in the gpu VA
09:25phirehowever, userspace tracing shows a region is mapped with the the VA of 0x18000
09:26phireso there is this weird, extra address space? that apparently only pipeline defintions live in?
09:37mrkajetanp has joined #asahi-gpu
09:38phireit's only the pipeline defintion. That references straight back to regular gpu VAs
10:01marcanphire: interesting. maybe it's some kind of control block that is mapped separately?
10:02marcanalso, are we sure AGX firmware can alter the page tables? I'm going to have to include that info in my tech doc since I've been saying proprietary blobs don't have full RAM access and that seems to be an exception
10:02marcanbut I also wouldn't be surprised if Apple did *something* and in fact it's more restricted
10:03marcane.g. it could be that the reason for that special VA range is because that's the one thing AGX firmware *can* access
10:03marcan(just a random guess)
10:03phireWell, it populates it's own pagetables for the 0xf8000000000 range when you boot it
10:04marcanand those just point to regular RAM?
10:04marcanor to the GFX special regions?
10:04marcane.g. it could be that the UAT/GART filters the high range/TTBR to only allow translation to stuff within a specific carveout region
10:05phireThey cover the agx-asc firmware blob that iboot put in main ram, and the IO range for agx-asc's registers
10:06marcanwhere are the page tables themselves?
10:07phiremain ram. The L0 page is at gpu-region-base, kernel points L0[1] at gfx-shared-region-base
10:08phirethe gpu uses gfx-shared-region-base for it's page tables
10:09phireI haven't fully investiated the permission bits. It's possible the pagetables are setup in such a way that the firmware can't modify them once booted
10:09marcanwhat region is the agx init struct in, do you know?
10:09marcan(the one passed to the first/whatever command)
10:09phireit's a virtual address, after the pagetables are already initilized
10:10marcanyes, but what vaddr range is it in, and what paddr does it point to?
10:11phireit's in the 0xfa0_00000000 vaddr range (everything the macos kernel passes into it is in that range)
10:11phireand I think it's just a regular physical memory address
10:11marcanso that's in L0[1]?
10:12phireL0[1][3]
10:15phireinitdata is at 0xfa0000b0000
10:22MajorBiscuit has joined #asahi-gpu
10:29marcanhm, yeah, I don't see how that would work based on VA ranges, but it could still be what you say about the PTs getting locked
10:30marcanor rather, the PTs themselves are mapped with flags that might mean "RO from ASC"
10:30marcanso only the initial mechanism to populate the PTs would have to be locked off
10:32phiresome of the log messages I've seen talk about a "microPPL"
10:32marcanthere's also the existing VMSA lockdown stuff
10:32marcantheir cores already have the ability to lock translation on
10:32phireinteresting. I get the feeling we are going to have to RE this blob just so we know it's security model
10:33phireAh... found the Pipeline data I was missing
10:33phireit lives at VA 0x1100018000
10:34marcanpossibly a per-context VA prefix is involved?
10:34marcanthat would make sense
10:34phireyeah, there is an L0 table for every context
10:34phirethey all share the same L0[1] ptr
10:35marcanbut yeah, if the AGX firmware maps its own PTs read-only, and then VMSA locks itself, early in init, that's all the code we need to audit to say "no backdoor here"
10:35marcanand doing it that way is quite Apple-like
10:35phireeach L0 is 16 bytes, so they sit next to eachother at gpu-region-base
10:35marcanah, right
10:37phireat this point I'm going to have to start on a m1n1 agx driver
10:38phireI understand close to enough to launch a compute kernel
10:40marcanin python right?
10:40marcanthat would be awesome
10:40marcanbtw, stringsing the kexts, they seem to call the PTs UAT and GART somewhat interchangeably
10:40marcanI'm guessing UAT is the current generation of GART or so
10:41phireyes. UAT seems to mean Unified Address Translater, So I'm guessing there used to be a seperate GART and DART
10:41phirebut they got combined
10:41marcanmakes sense
10:42marcanalso I wonder how this works together with the ARM's actual page tables
10:42marcani.e. does this sit entirely outside of the ARM's own MMU?
10:42phireI think it might be a modified ARM MMU
10:43marcanmaybe they run in parallel?
10:43phireponting at the same data?
10:43phirewell, same pagetables
10:43marcanyeah, like maybe the ARM's actual sysreg TTBR1 gets pointed to the same L1[0] and that all uses the standard VMSA lockdown stuff
10:44marcanand then it's possible the ARM never actually has to R/W data in the lower half of the address space directly
10:44marcanwe have rtkit-private-vm-region-base which points to what very much looks like a typical ARM top half VM address space base
10:45phireEverything I've seen points to a unified address space
10:45phireyeah, rtkit-private-vm-region-base is 0xf80_00000000 to 0xf8f_ffffffff
10:45marcan0xffffff8000000000
10:45phireit gets truncated to 40 bits
10:46marcanthat could be PAC?
10:46phireit just seems to be sign extention of bit 39
10:46marcanalso I see it going to f8..fa
10:46marcanand yes, that's how the PAC truncation works
10:46marcanyou sign extend at some bit
11:14povik11:38 < phire> I understand close to enough to launch a compute kernel
11:14povikexciting
11:25marcanphire: yeah, the ARM's TTBR gets pointed straight at gfx-shared-region-base
11:25marcanlooking at the firmware init a bit
11:25marcanso these should be ARM format page tables
11:25phireit doesn't seem to be far off, from what I can tell. But I don't know the ARM MMU very well
11:28phireIt gets pointed at gfx-shared-region-base, and not gpu-region-base?
11:34marcangpu-region-base is the TTBRs, right?
11:34marcanthe ARM does not have that PT level, TTBRs are sysregs
11:35marcanthe firmware calls gpu-region-base "UAT GPTBAT" fwiw
11:36phireso it just copies the two entires from gpu-region-base into into TTBR0 and TTBR1?
11:36marcannot from gpu-region-base; those are patch args to the firmware
11:38phireI guess if the ARM core never touches the lower address range, it doesn't need to care about the gpu contexts and handle updates to gpu-region-base
11:59marcanphire: it does have a function to switch TTBR0 around to different contexts
11:59marcanthe cool thing is the GXF stuff somehow has symbols here
12:00marcanand by the nature of it, if it can guarantee those PTs never point to crazyland, then all we need to do is audit the GXF code (which is one page) to convince ourselves that there is no backdoor
12:00marcanI think that's good enough for the "no, really, Apple don't have a backdoor here" argument
12:01phireFrom what I can see, context switching is controlled by second field in work submission commands.
12:02phiresometimes the 3rd field, for the compute work submission
12:02sveniirc GXF also has (had?) some symbols in some XNU kernel
12:03svenbut then they were removed the next released again
12:03sven*release
12:04svenand then there was ofc that one kernel with llvm bitcode released under the open source license. that should contain full symbols and then some for GXF as well :D
12:08phireif GXF is so small, could we rewrite it, and then graft the rest of apple's firmware onto it?
12:09sveni'm not sure i understand that question
12:09svenGXF is what apple called "guarded exception level". it's a lateral level to EL2 called GL2 with a different interpretation of pagetable permissions and usually the only code that gets to modify pagetables
12:11phireWhat I'm wondering is could we take the bulk of apple's gfx-asc firmware, but replace the GFX level code with our own code?
12:11phireas an alternative to auditing
12:11svenso we can't easily replaced the firmware
12:11sven*replace
12:11svenit's in a region of RAM that's write protected
12:11Jamie[m]1and it's loaded early in boot, right?
12:12phireoh, that region is write protected?
12:12sveni think apple calls that CTRR. iBoot loads the firmware of all co-procs into a memory region and setups CTRR to write protect it
12:13sventhey do the same for XNU but that protection can be disabled in 1TR
12:13phiredidn't we have to disable CTRR to boot m1n1, or am I thinking of something else?
12:13svenalmost!
12:13sventhey have two CTRR regions
12:13svenone for the kernel and one for the co-processor firmware
12:13svenwe can only disable the first one
12:16phireDoes that mean since we have control over the pagetables, we could hijack all of the agx-asc firmware except for the GFX level?
12:18sveni didn't follow the pagetable discussion, but I think that if you control whatever TTBR0/1 points to you can even hijack the GL2 code
12:18svenyeah. if we can modify the full pagetable we can just map the firmware (or parts of it) somewhere else
12:19sven(including the code running in GL2)
12:20svenhrm... unless they also limit the physical addresses that are allowed to contain code like they do for the XNU kernel
12:21phirethough, that might not enough to satisfy the people conserned about backdoors. It would be a racecondition to replace the pagetables before too much of the agx-asc firmware runs
13:30marcanhmm... I don't think this MiniPPL is very well done
13:30marcanas far as I can tell it reads the GPTBAT base from a random global that is not in a PPL-only write protected page?
13:30marcanand then reads the TTBR0 from that for context switches
13:31marcanso you can just redirect TTBR0 to any page table you want by changing the GPTBAT base
13:33marcanunless I'm confused about the MMU maps
13:36marcanthe page tables are indeed readonly though, AFAICT
13:36marcanbut yeah, without protecting PPL data, this is a bit pointless isn't it
13:44phireawkward
13:52svenand i guess that GPTBAT pointer is not inside the CTRR firmware region either?
14:09balrog has quit [Quit: Bye]
14:17balrog has joined #asahi-gpu
14:38yuyichao has quit [Ping timeout: 480 seconds]
14:55yuyichao has joined #asahi-gpu
14:55phirewell, I just found what might be the gptbat pointer being passed in with the initdata
14:55phirethis initdata is a mess
14:59phireI zeroed out the region I had labled as "heap", and I got an "Instruction Abort" crashlog
15:13marcansven: it's data
15:14marcanphire: probably jumping to a 0 function pointer
15:14marcanwe know the firmwares are quite pwnable, the bigger question is what is the backdoor surface and whether Apple are going to harden the PPL stuff to the point where we can say we don't need to audit the whole firmware
16:54user982492 has joined #asahi-gpu
17:15nafod has quit [Quit: Ping timeout (120 seconds)]
17:15nafod has joined #asahi-gpu
17:28balrog has quit [Remote host closed the connection]
17:28balrog has joined #asahi-gpu
17:38MajorBiscuit has quit [Ping timeout: 480 seconds]
18:04user982492 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
19:01user982492 has joined #asahi-gpu
19:43Gaspare has joined #asahi-gpu
20:47boardwalk has quit [Quit: Ping timeout (120 seconds)]
20:48boardwalk has joined #asahi-gpu
21:11Gaspare has quit [Read error: Connection reset by peer]
23:24user982492 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
23:57VinDuv has quit [Remote host closed the connection]
23:57VinDuv has joined #asahi-gpu
…