#
…

2023-09-08

00:35jeisom_ has quit [Quit: Leaving]
00:44jeisom has joined #asahi-gpu
01:18ourdumbfuture has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
01:27ourdumbfuture has joined #asahi-gpu
01:44Armlin has joined #asahi-gpu
01:44Armlin has quit [Read error: Connection reset by peer]
01:45Armlin has joined #asahi-gpu
02:01hightower3 has joined #asahi-gpu
02:02ourdumbfuture has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
02:07ourdumbfuture has joined #asahi-gpu
02:08hightower4 has quit [Ping timeout: 480 seconds]
02:21crabbedhaloablut has joined #asahi-gpu
02:28iaguis_ has joined #asahi-gpu
02:29jeisom has quit [Ping timeout: 480 seconds]
02:32ourdumbfuture has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
02:33iaguis__ has joined #asahi-gpu
02:33iaguis has quit [Ping timeout: 480 seconds]
02:36iaguis_ has quit [Ping timeout: 480 seconds]
03:32Armlin has quit []
07:28homura has joined #asahi-gpu
07:53homura has quit []
08:00JTL has quit []
08:08JTL has joined #asahi-gpu
08:17JTL has quit []
08:18JTL has joined #asahi-gpu
08:19JTL has quit []
08:51pg12 has quit [Remote host closed the connection]
08:53linaalyssa: Soooo.... there's no mailbox ^^;;
08:54linaIt's all firmware and shared memory, there is no mechanism to interrupt a shader or anything like that.
08:55linaInstead what they do is they have a compute shader calculate how much memory is needed, and write that out to some shared memory, then when the next shader runs that shared memory location is passed to the firmware as an alloc request, and then the firmware allocates memory before running it.
08:56linaEach job has an alloc request and a free request, and writes out the alloc/free request counts of the next job
08:56pg12 has joined #asahi-gpu
08:57linaFor mesh, there are 3 compute jobs: the first one calculates the first allocation, the second one writes to it and calculates the second allocation, then the third one writes out the mesh data/stuff into the second allocation, then the final 3d/ta job uses that.
08:58linaSo the "just in time" allocation mechanism is just there so the jobs don't have to pingpong through the kernel for allocs, but rather they can be queued at the firmware eagerly and only when the firmware is about to run a job does it poke the kernel to allocate its memory
08:58linaIn typical Apple fashion, it's overcomplicated with two layers of accounting for buffer size/usage...
08:59linaAlso everything is done in discrete "block count" units but I think the firmware doesn't care about where we alloc or how big a block is, since that's just up to the shaders/kernel
09:00linaThe only interesting thing in the shaders is those shared memory counts are written to like this:
09:00lina 1ca: 4591280510f83200 device_store.TODO 0, i32, xy, r18_r19, r12_r13, 2, signed, lsl 2, 1, 1, 0, 3, 0
09:00linaMy guess is that's an uncached store or something like that, since the firmware needs to be coherent with this stuff.
09:06linaOhh but I think the firmware may have a 64-bit free blockmap, so max 64 blocks. Blocks are 64MiB on macOS, which means 4GiB max buffer size, which makes sense as a max.
09:06linaSo we could use a different blocksize, but we'd lower our max usage if we do that.
09:23iaguis__ is now known as iaguis
09:48mairacanal has quit [Remote host closed the connection]
10:05chadmed has joined #asahi-gpu
10:06linaI think for us, the ABI for this would look essentially the same as the firmware ABI, except we use a BO and offsets for the count/shared buffers with the firmware (then we can map it in firmware land without making it world writable like Apple does, they're so bad at this...)
10:07linaAnd the requirement is that everything is submitted as one ioctl, from alloc to free
10:08linaOur ioctl could pass in a max required blocks size, and then at submit time the kernel ensures the global shared allocation has at least that much space. Then we mutex job execution using the global buffers (considering this another resource, blocking via a fence), but lazily drag buffer blocks between VMs so we avoid remapping and zeroing out when the jobs actually use less than the max memory.
10:28chadmed has quit [Quit: Konversation terminated!]
10:30chadmed has joined #asahi-gpu
11:29ourdumbfuture has joined #asahi-gpu
11:30ourdumbfuture has quit []
11:44ourdumbfuture has joined #asahi-gpu
12:08ourdumbfuture has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
12:18alyssa has joined #asahi-gpu
12:19alyssa 09:00 <lina> 1ca: 4591280510f83200 device_store.TODO 0, i32, xy, r18_r19, r12_r13, 2, signed, lsl 2, 1, 1, 0, 3, 0
12:19alyssaYeah, we're assuming the .TODO variants have different cache bits but we're not sure what any of the bits mean yet
12:19alyssaso that tracks
12:20alyssa 08:57 <lina> For mesh, there are 3 compute jobs: the first one calculates the first allocation, the second one writes to it and calculates the second allocation, then the third one writes out the mesh data/stuff into the second allocation, then the final 3d/ta job uses that.
12:20alyssaThis seems deeply cursed
12:20alyssaTo be clear, "job" in this context is a thing kicked off by a RunCompute op?
12:21alyssa(rather than an individual dispatch or a whole ioctl?)
12:22alyssa10:08 <lina> Our ioctl could pass in a max required blocks size
12:22alyssaI don't know that information
12:22alyssaSo it would be set unconditionally to the size of the entire global heap
12:23alyssaThe entire point of this mechanism is that I /don't/ know any reasonable bounds ahead-of-time
12:23alyssa(If I did I'd just static allocate in userspace)
12:25ourdumbfuture has joined #asahi-gpu
12:34jeisom has joined #asahi-gpu
12:44dylanchapell has quit [Remote host closed the connection]
12:58cylm has joined #asahi-gpu
13:17linaalyssa: Yes, "job" is a RunCompute
13:17linaalyssa: I know we don't know the max required blocks, but it seems more useful to allow that to be set from userspace (e.g. via env variable) than a single global kernel module parameter?
13:17linaThen we could even do driconf overrides for known problem apps
13:18ourdumbfuture has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
13:27dylanchapell has joined #asahi-gpu
13:36mairacanal has joined #asahi-gpu
13:37linaalyssa: Do you want me to look at this more or is this enough to make up a plan? I *think* I can easily test this whole mechanism from the m1n1 python stuff, since it doesn't really have to involve shaders (I can just pre-populate the buffers with whatever I want and dump whatever the firmware did to them)
13:38lina(And since as far as I can tell the mechanism is uniform, it doesn't even have to be compute, I can just have a render pass request allocatinon)
13:39commandoline has quit [Quit: Bye!]
13:40commandoline has joined #asahi-gpu
13:44i509vcblina: Is the mentioned theoretical 4GiB maximum buffer size related to the usage of the buffer or is it more of a strict limit? (Want to understand this before blindly updating the limit in agxv)
13:45linaI think macOS just restricts it to 4GiB of address space
13:45linaI don't think the GPU/firmware care about that, but there does seem to be what looks like a 64-bit bitmask of blocks involved, so whatever blocksize we pick we can only have up to 64 of them.
13:46linaAnd using blocks larger than 64MiB seems... kind of wasteful
13:48linai509vcb: What limit are you thinking about? This is mostly for internal mesh/geom/etc emulation, not a buffer exposed to users
13:48commandoline has quit [Quit: Bye!]
13:48ourdumbfuture has joined #asahi-gpu
13:49commandoline has joined #asahi-gpu
13:49alyssa13:17 lina | Then we could even do driconf overrides for known problem apps
13:49alyssaMaybe? shrug
13:49linaI mean if we need to allocate at ioctl time it's not much harder to have it allocate-or-expand and take the buffer size from userspace
13:50os has quit [Ping timeout: 480 seconds]
13:50linaThat has nothing to do with this
13:50linaThis is a very specific mesh/geom emu buffer thing
13:50linaNot "any buffer"
13:50i509vcbOkay then I guess I just ignore the above
13:51linaWhat's the value right now?
13:51i509vcbNo idea, I think it's either max memory allocation size or UINT64_MAX
13:53linaUser address space right now is 256GiB (half of the 512GiB half since the rest is reserved for other purposes) and Apple don't sell machines with that much RAM, so something based on RAM size probably makes more sense if you want a "useful" value
13:54commandoline has quit []
13:54linaFrom what I heard we want to try to de-hardcode most of this address space stuff, so eventually it might be closer to the full 512GiB lower half (full AS is 40 bits / 1 TiB but half is kernel land)
13:54commandoline has joined #asahi-gpu
13:54alyssalina: relatedly, did we decide that SVM is hard-impossible on current-gen AGX?
13:55linaNo, that's what this is for. Apparently Nvidia has the same problem and somehow they make it work?
13:55lina(with GPU AS < CPU AS)
13:55alyssa(I didn't really follow that conversation)
13:55lina(I donÂ't know how they deal with that but if they do, no raeason not to try here too)
13:55lina*reason
13:57linaThis will be another UAPI break once I flip it around
13:58linaWe probably want some code in mesa to rand() the carveouts to an extent, to prove this actually works
13:59linaThe globals that mention these addresses annoy me, but for all we know the firmware never uses them...
13:59linaI guess I should grep the firmware a bit, see if I can confirm or deny any of it
14:06alyssaapple doesn't do SVM on macOS do they?
14:16linaOK, the "magic page" address at 0x6fffff8000 is written to a hardware register, so that one does something...
14:16linaI don't think any of the others are used for anything, so we can probably ignore those globals and let userspace define those addresses
14:16linaI guess we should move the magic page to the top at 0x7fffff8000 and then just say we "almost" have 40 bits of address space for userspace to allocate however it wants
14:17linaWhich is the best we can do for SVM
14:51ourdumbfuture has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
15:41compassion1785 has quit [Quit: lounge quit]
15:43cylm has quit [Ping timeout: 480 seconds]
15:54cylm has joined #asahi-gpu
15:56compassion1785 has joined #asahi-gpu
16:17karolherbstare you managing the VM in userspace or kernel space?
16:18karolherbstand anyway, as long as you don't do full system SVM the address space size doesn't matter
16:18karolherbstmemory used for SVM needs to be allocated manually and you can just use mmap instead of malloc and restrict the area from where to allocate
16:19karolherbstI still have to write a proper protototype for all of this, but I'm kinda having a plan
16:39ourdumbfuture has joined #asahi-gpu
17:18rappet has quit [Quit: https://quassel-irc.org - Chat comfortably. Anywhere.]
17:20rappet has joined #asahi-gpu
17:20cylm has quit [Ping timeout: 480 seconds]
17:23linakarolherbst: Userspace, except the carveout for kernel BOs
17:24linaIt's the VM_BIND thing
17:24linaBut I mean, we could have an SVM mode which syncs the CPU page tables like I think nouveau has?
17:25karolherbstyeah, the carveout needs to happen, but as long as you don't use mmu_notifiers to mirror arbitrary CPU memory, having the address spaces be a different size isn't a problem
17:25karolherbstlina: yeah....
17:25karolherbstand then you'd have to careveout all invalid regions
17:25linaYeah...
17:25linaAnyway I don't know how this really works, I just want to make sure I understand what we can do ^^
17:25karolherbstI don't have a solution for that, but luckly nothing yet requires it
17:25linaI should sleep ^^
17:26karolherbstyeah.. me neither :D
17:26karolherbsthave a good rest!
17:26karolherbstbut anyway.. userspace needs a way to map arbitrary host memory into the GPU AS, which is generally called "userptr"
17:26karolherbstand that's all you need for basic SVM
17:27karolherbstI have a prototype for that against iris, I just have to fix it
17:27linaYeah, userptr... as long as you're on a 16K kernel ^^
17:27lina(That's never going to work on 4K...)
17:28karolherbstnah, that's fine
17:28karolherbstiris can map arbitrary pointers, you just offset it
17:28linaI mean on AGX
17:28linaThe GPU pages are 16K so good luck mapping arbitrary CPU 4K pages...
17:28karolherbstso you map the entire page, but the driver deals with the alignment
17:28karolherbstohh right...
17:28karolherbstthat would be 4 pages...
17:28karolherbstthough it _might_ just work anyway
17:28linaYeah, which won't be contiguous
17:29karolherbstyeah if that's required that you are screwed :D
17:29linaAnd can't be made contiguous reliably even if you have weird migration stuff
17:29linaSo yeah that's never going to work
17:29karolherbstin which case no SVM...
17:29karolherbstbut whatever
17:29linaYeah, no SVM on 4K
17:29linaAnyway, nn Â^^
17:29karolherbstit's such a niche feature, just that SyCL/HIP require it :')
17:30lina4K is just for gamers anyway ^^
17:30karolherbstuserspace could always try though and the kernel either accepts a mapping or not....
17:30karolherbstbut yeah...
17:31lina(That's already how it works on 4K kernels if you try to give it unaligned BOs ^^)
17:32linaIt would work with hugepages w
17:32karolherbstI wonder if there is mmap magic userspace could do
17:32linaThere's no general solution to this problem
17:32linaThere are probably a bunch of weird specific solutions ^^
17:32karolherbstjust `MAP_HUGE_2MB` I guess...
17:33linaLike my broken shm hack to get order-2 allocs
17:33jannauI can send an subscriber link
17:33linaAnyway really, nn ^^
17:33linajannau: I'll read it later but that would be nice ^^
17:33karolherbstuhh.. I can access lwn via RH for free, but I might have to fix some VPN stuff for it.. let's see...
17:36karolherbstohh.. I can also generate links.. handy
17:37jannauit would allow using multi page folios for mmap but it's far off from being merged and might not be application controllable
17:59alyssalina: I assume the answer is yes, but 4K VMs on a 16K system are also affected (no SVM inside the VM?)
18:38delroth has quit [Remote host closed the connection]
18:39delroth has joined #asahi-gpu
18:56chadmed has quit [Ping timeout: 480 seconds]
19:21nightstrike has joined #asahi-gpu
20:11maximbaz has quit [Quit: bye]
20:11maximbaz has joined #asahi-gpu
20:29ave36309 has quit [Ping timeout: 480 seconds]
20:39ave36309 has joined #asahi-gpu
21:22crabbedhaloablut has quit []
21:29nightstrike has quit [Quit: Connection closed for inactivity]
22:01nela has quit [Quit: bye!]
22:01nela has joined #asahi-gpu
22:05Misthios has quit [Quit: Misthios]
22:08ourdumbfuture has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
22:10ourdumbfuture has joined #asahi-gpu
22:16Cyrinux9474 has quit []
22:16Cyrinux9474 has joined #asahi-gpu
23:13i509vcbalyssa: any pointers for implementing load_layer_id on agxv? I noticed a few drivers involve creating an nir_variable but I suspect this might not be right for agxv
23:13i509vcb(trying to get blits to at least try to run)
23:17i509vcbI'll peek there I guess
23:18alyssathat MR needs to be merged into agxv/main
23:18alyssaand then if you're really getting load_layer_id and not VARYING_SLOT_LAYER load_inputs, then use nir_lower_sysvals_to_varyings
23:18alyssaand the code in that MR will take care of it
23:19alyssayou'll need to extend lower_sysvals_to_varyings in the obvious way in that case
23:27ourdumbfuture has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
23:33ourdumbfuture has joined #asahi-gpu
23:45jeisom has quit [Ping timeout: 480 seconds]
23:47darkapex1 has joined #asahi-gpu
23:48darkapex has quit [Ping timeout: 480 seconds]
…