2022-02-17
00:09kenzie35 has quit []
00:09kenzie35 has joined #asahi-dev
00:29refi64 has quit [Remote host closed the connection]
00:30refi64 has joined #asahi-dev
00:31phire_ has joined #asahi-dev
00:31phire is now known as Guest12
00:31phire_ is now known as phire
00:33Guest12 has quit [Ping timeout: 480 seconds]
00:34axboe has joined #asahi-dev
00:52milek7_are modern ssd really providing any guarantees on power loss? with opaque FTLs and hundreds of megabytes of cache
01:15yuyichao has quit [Ping timeout: 480 seconds]
01:38axboebut it's one of the 4 core primitives, really should get that wrong for any half-way decent device and above
01:44yuyichao has joined #asahi-dev
02:15rkjnsn has quit [Quit: Reconnecting]
02:15rkjnsn has joined #asahi-dev
03:29PhilippvK has joined #asahi-dev
03:32phiologe has quit [Ping timeout: 480 seconds]
03:39axboe has quit [Quit: leaving]
03:40kov has quit [Quit: Coyote finally caught me]
03:40kov has joined #asahi-dev
04:53marcan< sven> we can always file a radar with apple and have it disappear and get no feedback for years! ;)
04:53marcanif it violates barrier guarantees in a way that can cause proper corruption when you yank the plug on the mac mini, we repro it then file a bug then I blog about it so it hits hacker news ;)
05:00marcanmilek7_: yes, some SSDs (even some consumer lines) have enough capacitors to flush the cache on plug pulls
05:00marcanthough these days they're mostly given up onthat, but Micron/Crucial ones used to be like that
05:01marcanI think at the lower end they only flush enough to make sure the FTL is consistent, but not necessarily all the data
05:03marcansee e.g https://www.micron.com/-/media/client/global/documents/products/white-paper/ssd_power_loss_protection_white_paper_lo.pdf
05:04marcanthe consumer line has enough caps to avoid corruption on power yank; the enterprise line has enough to save the entire cache
06:16XeR has quit [Ping timeout: 480 seconds]
06:17VinDuvI may have missed some part of the discussion, but on macOS fsync does not flushes the disk cache; it only writes the data to disk. You have to use fcntl(fd, F_FULLFSYNC) to flush the disk cache.
06:26rkjnsnSounds like it matches how kettenis was describing OpenBSD as working, then, which I suppose is not terribly surprising.
06:42marcansimple python script writing to a file (using raw os. calls) and doing fsync() on it, then sleeping
06:43marcanin fact I even managed to trigger an inconsistency in an Apple app by accident: I had GarageBand open, and closed it before the test, not saving data. on reboot, it tried to restore the now-nonexistent file, and threw up an error about corruption/invalid project
06:43marcanso I guess it deleted the "unsaved" project but didn't commit the "last open" state to restore on boot
06:50marcanand indeed with a dumb repeated write test I get ~46 IOPS with that, vs. 40000 with plain fsync
06:56marcangiven the default dirty writeback on Linux is 5 seconds and macOS is losing more than 5 seconds worth of data with this, it *effectively* is like doing nothing on Linux, modulo writeback pressure
06:59marcanso on a Samsung SSD 860 EVO mSATA on my laptop, I get 10K IOPS with the same write cache hack, and ~330 without (and fsync)
07:01the_lanetly_052__ has joined #asahi-dev
07:04rkjnsnDo we know how much of an issue it would is outside of apt and synthetic benchmarks? I'm curious if it would make sense just to make apt less aggressive with its syncs, given that in can still be a significant slowdown (albeit less of one) on other drives. Presumably most software isn't trying to sync more than 50 times a second…
07:09rkjnsnOut of curiosity (since I don't know too much about this topic), if folks do want to ignore/postpone flushes on Linux, does the SSD support any other kind of ordering operation that could ensure data written after a flush doesn't hit the disk before data written prior to it?
07:09rkjnsnI know it could still eat data that was supposed to be safely stored, but it'd be nice to avoid broken internal consistency.
07:16VinDuvI’m pretty sure there isn’t a way to properly order the writes since basically the whole chain is allowed to reorder them
07:30Glanzmannmarcan: axboe said yesterday that there is no such thing as 'barrieres' for 15 years. And that it is impossible due to multiqueue support IIUC.
07:31rkjnsnI think they were talking about barriers in Linux, as opposed to barriers in the drive, though?
07:31rkjnsnEven without support for write ordering in the kernel, it seems like (IIUC) translating fsyncs to write barriers (assuming the drive supports them) rather than flushes could at least help avoid corruption from the drive reordering things, even if it doesn't protect against writes that were supposedly committed being lost.
07:34svenI don't remember seeing anything about write barriers in the nvme spec but I only read that one for the first time a few weeks ago
07:34marcanand yeah, translating fsync()s to write barriers would likely avoid corruption, if the drive can do that
07:37sventhere's force unit access for write commands but that would probably only order writes where it's set (if it enforces the ordering at all)
07:45_jannau_marcan: I don't think dcp in its current state is ready to be integrated in asahi. preserved regions need a coordinated m1n1/kernel changes with a not fully agreed on dt-binding. My current code to remap in the locked dcp dart apparently only works when iboot doesn't initialize dcp (it works only on the mac mini)
07:46marcanwould it be fair to say that if I make I release within this month I should keep simpledrmfb for now?
07:47_jannau_yes, I think simpledrm should be preferred unless someone has time to work on dcp before that
07:50_jannau_I don't think there is huge amount of work to be done to make dcp useful but I should concentrate on submitting spi-hid
08:01MajorBiscuit has joined #asahi-dev
08:02marcanrepro'd on the Mac Mini pulling the plug; there is definitely no (working) last-gasp mechanism
08:19c10l4 has joined #asahi-dev
08:25c10l has quit [Ping timeout: 480 seconds]
08:26arinterestingly, https://sqlite.org/atomiccommit.html#sect_9_2 > Setting fullfsync on a Mac will guarantee that data really does get pushed out to the disk platter on a flush. But the implementation of fullfsync involves resetting the disk controller. And so not only is it profoundly slow, it also slows down other unrelated disk I/O. So its use is not recommended.
08:31dottedmagIs there a way to quickly detect power loss on mini, to cobble last-gasp signal purely in sw?
08:37marcandottedmag: not seeing anything in SMC, quick look at the MBA schematic (they often have hints as to other variants) doesn't show any place that signal would go
08:38marcandottedmag: polling the main primary voltage rail voltage, it updates once a second, but I caught it updating immediately before shutdown after pulling the plug and it wasn't drooping
09:12tanty has joined #asahi-dev
10:09marcanwhile running the python loop, python says 46 ops/s, powermetrics reports 915 disk ops per second / 4.7MB/s (seems a bit much? APFS write amplification? and 20MB/s of ANS2 memory bandwidth in both directions)
10:10marcandoing the same thing, but with an artificial delay to make it run at the same speed sans the flush, I get 44 disk ops/s / 180 KB/s, and ~200 KB/s of ANS2 memory bandwidth
10:11marcanso that sounds like two problems here... FULLFSYNC does some horrible APFS amplification nonsense, *and* ANS2 blows it up even more
10:14marcanon FAT32 it's actually slower (34 IOPS), but no serious amplification: 78.37 ops/s 321.02 KBytes/s
10:15marcanso yeah, ANS2 is definitely doing something somewhat dodgy if it's doing 9MB/s of memory write traffic and 3MB/s of memory read traffic to serve 320KB/s of data traffic
10:27XeR has joined #asahi-dev
10:44refi64 has quit [Read error: Connection reset by peer]
10:45refi64 has joined #asahi-dev
11:11rkjnsnI saw a reference to a F_BARRIERFSYNC. Not sure if that's a macOS thing or just an iOS thing, but if it's available on macOS, it might be worth seeing if it works as advertised and results in less performance loss.
11:11mazsven: contrary to what I said, the AIC doesn't seem to have a configuration for edge/level, and "knows" which line in which. and the fasteoi flow doesn't distinguish them either.
11:16svenyeah, that's what I thought after spending some time with the interrupt code yesterday as well
11:37kettenissven: so on OpenBSD I still set the "number of openings" to 1 to avoid the NVMe from locking up
11:41sventhat sounds very similar actually. some commands never generate the completed interrupt but they do appear in the completion queue and can be polled but others don't ever seem to be triggered
11:42svenmacos uses INTMS and INTMC in its interrupt handler apparently, which linux doesn't because it relies on the interrupt controller itself
11:43kettenisI think "number of openings" is classic SCSI terminology for the number of commands that can be in flight simultaniously
12:01MajorBiscuit has quit [Ping timeout: 480 seconds]
12:04gladiac has quit [Quit: k thx bye]
12:05gladiac has joined #asahi-dev
12:17MajorBiscuit has joined #asahi-dev
13:01Gaspare has joined #asahi-dev
13:43marcanfor shits and giggles (and because HN commenters are tiring): full fsync on a shitty USB3 flash drive on macOS/M1: 223 IOPS. internal NVMe, 58 IOPS. Both FAT32.
13:44marcan(of course, the flash drive has no cache at all, so it's equally slow with a vanilla fsync :p)
13:44marcanalyssa: yes, you get better database transaction latencies on a shitty flash drive than on internal NVMe on these machines :-)
14:12svenaaand now I can't reproduce the missing interrupts at all anymore with the original code *sigh*
14:14marcanthe only reason I haven't deleted twitter is those 50k followers are *sometimes* useful :p
14:27alyssamarcan: unfollowing everyone and setting all privacy settings to "following only" has made my twitter quiet! :-p
14:32Major_Biscuit has joined #asahi-dev
14:33MajorBiscuit has quit [Ping timeout: 480 seconds]
14:41yuyichao has quit [Ping timeout: 480 seconds]
14:59yuyichao has joined #asahi-dev
15:06hays has quit []
15:10hays has joined #asahi-dev
15:26hays has quit []
15:28hays has joined #asahi-dev
15:29axboe has joined #asahi-dev
15:37Gaspare has quit [Ping timeout: 480 seconds]
15:44Jamie[m]1is radar priority decided directly based on HN rank, or is twitter likes the bigger factor? :P
15:59svenaxboe: remind me again, is taking the anv->lock around the writel itself enough to prevent those timeouts or did the memcpy also have to be inside the lock?
16:06axboesven: just around writel is enough, that's what I've been running and it's been rock solid
16:07axboesven: reproduces trivially for me, was a bit harder with issues serialized, but could still hit it. consistently just doing a make -j8 kernel compile on it
16:08svenokay, so from looking at the hypervisor logs macos seems to never interleave "ring cq doorbell + nvmmu invalidation" with "write new tag to sq"
16:08svenso maybe if something like "start nvmmu invalidation, start new command, write cq doorbell" happens things break. no idea why though.
16:09axboeI generally didn't feel comfortable without that lock to begin with, only part I'm a bit puzzled on is why the cpu freq changes makes this so much more likely to trigger
16:10axboeit'd be better if we didn't need to share this lock (and particularly since it's irq disabling on the submit path), but at the rates of these drives and the fact that it only has a single queue anyway, it's not going to be an iops monster anyway and hence it doesn't really matter
16:12svenif it's actually a race between nvmmu/cq_db and that writel maybe the higher cpufreq just makes it more likely to lose (or win, depending how we look at it :D) that one
16:13svenquite a few people have been using it on the max/pro as well before cpufreq as well and never saw it either.
16:31axboesven: though for the batching, we could optimize the nvmmu_inval by writing each tag, then one readl_relaxed at the end...
16:34axboeI've got this funky series for io_uring that allows registered buffers to retain dma and iommu mappings, instead of doing them for each io submit and complete
16:35axboewas planning on adding support for apple nvme just to test out what it can actually do, if I do that, then I can try the nvmmu inval optimization too as it'd likely make a difference at that point
16:35axboewas part of the "lets see how many iops we can do on a core" experiments, 1 of 2 series that hasn't been posted anywhere yet as it's a bit of a hack
16:53axboe has quit [Quit: reboot]
17:00the_lanetly_052___ has joined #asahi-dev
17:02skipwich has quit [Quit: DISCONNECT]
17:02skipwich has joined #asahi-dev
17:06the_lanetly_052__ has quit [Ping timeout: 480 seconds]
17:06axboe has joined #asahi-dev
17:11axboe has quit [Quit: Lost terminal]
17:14axboe has joined #asahi-dev
17:16bpye3 has joined #asahi-dev
17:17al3xtjames2 has joined #asahi-dev
17:18skipwich has quit [Ping timeout: 480 seconds]
17:19al3xtjames has quit [Quit: Ping timeout (120 seconds)]
17:19al3xtjames2 is now known as al3xtjames
17:23skipwich has joined #asahi-dev
17:23bpye has quit [Ping timeout: 480 seconds]
17:23bpye3 is now known as bpye
17:32axboesven: ok queue_rqs is hard to support, since the sq door bell is writing the specific tag...
17:35rkjnsnmarcan, would it be easy to test F_BARRIERFSYNC using your python script to see what it does and how it performs on these machines?
17:44Major_Biscuit has quit [Ping timeout: 480 seconds]
17:44axboe has quit [Quit: Lost terminal]
17:59the_lanetly_052___ has quit [Ping timeout: 480 seconds]
18:05svenalyssa: yeah, I need to take a look at what marcan did to rtkit and we need to discuss how we will upstream that together with smc
18:28axboe has joined #asahi-dev
18:29axboesven: got queue_rqs setup, and it's just cpu reduction at this point, iops are capped / controller limited
18:38sirn- has joined #asahi-dev
18:45sirn has quit [Ping timeout: 480 seconds]
18:45sirn has joined #asahi-dev
18:52sirn- has quit [Ping timeout: 480 seconds]
18:52sirn- has joined #asahi-dev
18:53sirn| has joined #asahi-dev
18:56sirn^ has joined #asahi-dev
18:57sirn has quit [Ping timeout: 480 seconds]
18:57sirn^ is now known as sirn
18:58axboe has quit [Quit: Lost terminal]
18:59axboe has joined #asahi-dev
19:00sirn- has quit [Ping timeout: 480 seconds]
19:03sirn| has quit [Ping timeout: 480 seconds]
19:10axboemarcan: https://git.kernel.dk/cgit/fio/commit/?id=a04e0665cb5d3a545ab1dbe2d2b7c150b404735d
19:19___nick___ has joined #asahi-dev
19:31Telvana has joined #asahi-dev
19:45___nick___ has quit []
19:47___nick___ has joined #asahi-dev
19:49___nick___ has quit []
19:51___nick___ has joined #asahi-dev
20:26axboekettenis: yeah looks like the bsds took the "make it a system tunable" instead, which does make a lot more sense than a separate command
20:46axboethis is the perf-wip patches for persistent maps cherry-picked, then the apple-nvme bits needed on top
20:46axboeidea there is based on how linux manages IO in an on-stack plug for submitting more than 1
20:47axboethat part works a bit better on "normal" nvme, as we just have to do that one doorbell write with the sq tail
20:54svenah, true. Just copy all commands to the queue and advance it by N elements with a single write
21:08___nick___ has quit [Ping timeout: 480 seconds]
21:33robher has quit [Read error: Connection reset by peer]
21:33kendfinger has quit [Read error: Connection reset by peer]
21:33weems_ has quit [Read error: Connection reset by peer]
21:33daniels has quit [Read error: Connection reset by peer]
21:33arnd has quit [Read error: Connection reset by peer]
21:33weems_ has joined #asahi-dev
21:33daniels has joined #asahi-dev
21:33kendfinger has joined #asahi-dev
21:33arnd has joined #asahi-dev
21:33tardyp has quit [Read error: Connection reset by peer]
21:33tardyp has joined #asahi-dev
21:33robher has joined #asahi-dev
21:58axboe has quit [Ping timeout: 480 seconds]
22:06axboe has joined #asahi-dev
22:08thermoblue[m] has left #asahi-dev [#asahi-dev]
23:34axboe has quit [Ping timeout: 480 seconds]
23:57axboe has joined #asahi-dev