#asahi-dev · 2023-04-11 08:44
marcanaaaaargh, it's the power gate#asahi-dev · 2023-04-11 08:31
marcanok, config space is fully dead so something is wrong here#asahi-dev · 2023-04-11 08:28
marcanpushed a bunch of stuff to linux/m1n1 on t602x/bringup for both, but pcie still doesn't work :/#asahi-dev · 2023-04-11 06:46
marcanlooks like MSIs changed too, there's now a table thing#asahi-dev · 2023-04-11 06:36
marcanthis is probably Mac Pro related, they're preparing for real general purpose PCIe slots#asahi-dev · 2023-04-11 06:36
marcanrid2sid got moved out, presumably to make a lot more space there#asahi-dev · 2023-04-11 06:36
marcanper port changed too#asahi-dev · 2023-04-11 06:19
marcanok, re pcie, more registers moved individually, so this is going to be a proper device variant now#asahi-dev · 2023-04-11 02:05
marcanjannau: m1n1 gets the links up at least I think, so linux probably takes them down during its init (and then fails to do the right dance to bring them up again)#asahi-dev · 2023-04-10 17:53
marcannow I really need to sleep :p#asahi-dev · 2023-04-10 17:53
marcanI hope I didn't break anything#asahi-dev · 2023-04-10 17:53
marcanjannau: pushed#asahi-dev · 2023-04-10 17:53
marcan*finally* it works with that#asahi-dev · 2023-04-10 17:51
marcanack#asahi-dev · 2023-04-10 17:50
marcanyou do (with a warning), but for 64bit it overflows the size and breaks#asahi-dev · 2023-04-10 17:48
marcan*working around#asahi-dev · 2023-04-10 17:48
marcanwe shouldn't be working Linux silliness in the device tree#asahi-dev · 2023-04-10 17:48
marcandma-ranges = <0 0 0 0 0xffffffff 0xfffffffe>; I guess? This feels silly.#asahi-dev · 2023-04-10 17:48
marcan(since it gets adjusted)#asahi-dev · 2023-04-10 17:48
marcanand dma-ranges = <0 0 0 0 0xffffffff 0xffffffff>; does not work because it overflows then fails the zero check#asahi-dev · 2023-04-10 17:47
marcanheh it's actually impossible to declare all 64-bit AS as dma-ranges because 1<<64 does not fit in a u64#asahi-dev · 2023-04-10 17:44
marcanand that breaks because those ops see that the device is asking for 32 bits and the IOMMU *cannot support that* (because its range is restricted to only a higher aperture than the low 32 bits)#asahi-dev · 2023-04-10 17:44
marcanyes, but that isn't working because that OF codepath is calling arch_setup_dma_ops/iommu_setup_dma_ops with a 32bit mask *before* that can happen#asahi-dev · 2023-04-10 17:41
marcanbut also isn't that kind of backwards? what I want is drivers to be able to set their own DMA mask and OF not get in their way by clamping it further#asahi-dev · 2023-04-10 17:41
marcanhow do you get "all 64bit"?#asahi-dev · 2023-04-10 17:39
marcanIMO the default size in that function should be 64-bit on 64-bit platforms period; the actual dma_mask can remain 32-bit by default of course#asahi-dev · 2023-04-10 17:39
marcanput another way: of_dma_configure_id() is doing something fairly silly, taking the device DMA mask as a default size (or 32bit if not set), then clamping the DMA mask to that again#asahi-dev · 2023-04-10 17:37
marcan(only, that's all the max is used for, which is why this hasn't actually broken everyone by *forcing* 32bit DMA for everything)#asahi-dev · 2023-04-10 17:37
marcanthe problem is not the default, it's that the OF code is running before that default can be overridden and then eventually that codepath ends up in a sanity check that we fail#asahi-dev · 2023-04-10 17:37
marcanrobher: ^^#asahi-dev · 2023-04-10 17:37
marcanalso I think the reason nobody noticed this yet is that the subsequent arch_setup_dma_ops -> iommu_setup_dma_ops codepath only uses the size for a sanity check, it doesn't keep it anywhere. so it will pass as long as IOMMUs will at least *support* the 32-bit DMA range, and we're probably the first one that doesn't when we specify a restricted high aperture.#asahi-dev · 2023-04-10 17:32
marcanrobher: the call chain here is really_probe -> platform_dma_configure -> of_dma_configure_id. that ends up pulling a default 32-bit DMA mask since the device driver probe() hasn't run yet. that seems wrong.#asahi-dev · 2023-04-10 17:30
marcangoing to do this for now pending robher's response#asahi-dev · 2023-04-10 17:30
marcanyeah, that'd make sense logically, which is why I think this is a workaround and this kernel codepath is just broken#asahi-dev · 2023-04-10 17:27
marcanrobher: this codepath is getting called before the driver probes, so the DMA mask is unset, which means we end up limiting everything to 32bit. I see that if we set dma-ranges to a dummy single covering the entire 64bit AS, we'll probably get what we want... but is this what we're supposed to do?#asahi-dev · 2023-04-10 17:26
marcanyeah, though now I'm wondering if this really makes sense#asahi-dev · 2023-04-10 17:18
marcansame check, happening in the IOMMU core, similarly way too early#asahi-dev · 2023-04-10 17:17
marcanlovely...#asahi-dev · 2023-04-10 17:17
marcan[ 0.630506] specified DMA range outside IOMMU capability#asahi-dev · 2023-04-10 17:14
marcanyeah, I just dropped it and fixed the conflict#asahi-dev · 2023-04-10 17:10
marcanI don't think there's a solution for this, let's just drop that idea... setting the DMA range at the DART will already do the right thing wrt clamping IAS anyway.#asahi-dev · 2023-04-10 17:10
marcanjannau: the problem is the device DMA mask is not set yet, so we get 32bit. the iommu stuff happens too early.#asahi-dev · 2023-04-10 17:02
marcanwait no I think I messed this up#asahi-dev · 2023-04-10 16:58
marcanthat wouldn't work properly if we fully decoupled it from the IOMMU driver#asahi-dev · 2023-04-10 16:58
marcanI think it also makes sense to put it in the IOMMU because we have a special case where that range can exceed the IOMMU's own input range, and in that case it is assumed the high bits are dropped#asahi-dev · 2023-04-10 16:57
marcanthere's the dma-ranges stuff that can also limit this, but that's for buses and this isn't a bus...#asahi-dev · 2023-04-10 16:57
marcanApple puts it in the IOMMU and on Linux the IOMMU is the only easy place for this to be set, since that driver controls the aperture fully (devices can only set a DMA mask)#asahi-dev · 2023-04-10 16:56
marcanit's somewhat questionable whether this is a property of the IOMMU or the device, since it's really a property of how the IOMMU interacts with the device's view of memory (and we don't even know exactly how that is implemented)#asahi-dev · 2023-04-10 16:56
marcannot sure if it makes sense to make it generic, in principle other IOMMU drivers could easily support this but I'm not sure if they care#asahi-dev · 2023-04-10 16:55
marcanthis is a property for IOMMU device nodes to specify the allowable DMA address range, I have it as two u64s <base size> right now.