#
…

2022-02-17

00:09kenzie35 has quit []
00:09kenzie35 has joined #asahi-dev
00:29refi64 has quit [Remote host closed the connection]
00:30refi64 has joined #asahi-dev
00:31phire_ has joined #asahi-dev
00:31phire is now known as Guest12
00:31phire_ is now known as phire
00:33Guest12 has quit [Ping timeout: 480 seconds]
00:34axboe has joined #asahi-dev
00:35axboekettenis: seriously? like 90s style?
00:39axboewell I guess any kind of fs perf numbers on openbsd needs to come with that caveat ;)
00:40kettenisI don't think i've ever lost a filesystem though
00:40axboeI mean, all's well unless it crashes or you lose power
00:40axboeI used to use openbsd for a ppoatm router, when that was my inet connection
00:52milek7_are modern ssd really providing any guarantees on power loss? with opaque FTLs and hundreds of megabytes of cache
00:52axboethat's what the flush cache command is for
00:53axboeif previously acked writes aren't power loss stable after that, then the hw is defective
01:15yuyichao has quit [Ping timeout: 480 seconds]
01:19alyssadefective hardware? impossible.
01:38axboeheh I know
01:38axboebut it's one of the 4 core primitives, really should get that wrong for any half-way decent device and above
01:39axboeI'm sure there's tons of shitty ones that just nop it :/
01:40axboes/wrong/right, obviously
01:44yuyichao has joined #asahi-dev
02:15rkjnsn has quit [Quit: Reconnecting]
02:15rkjnsn has joined #asahi-dev
03:29PhilippvK has joined #asahi-dev
03:32phiologe has quit [Ping timeout: 480 seconds]
03:39axboe has quit [Quit: leaving]
03:40kov has quit [Quit: Coyote finally caught me]
03:40kov has joined #asahi-dev
04:53marcan< sven> we can always file a radar with apple and have it disappear and get no feedback for years! ;)
04:53marcanthe question is what does macOS do
04:53marcanif it violates barrier guarantees in a way that can cause proper corruption when you yank the plug on the mac mini, we repro it then file a bug then I blog about it so it hits hacker news ;)
04:53marcanthen apple listens :D
05:00marcanmilek7_: yes, some SSDs (even some consumer lines) have enough capacitors to flush the cache on plug pulls
05:00marcanit's not that hard with how fast SSDs are
05:00marcanthough these days they're mostly given up onthat, but Micron/Crucial ones used to be like that
05:00marcanand enterprise lines certainly like to advertise that feature
05:01marcanI think at the lower end they only flush enough to make sure the FTL is consistent, but not necessarily all the data
05:04marcanthe consumer line has enough caps to avoid corruption on power yank; the enterprise line has enough to save the entire cache
06:16XeR has quit [Ping timeout: 480 seconds]
06:17VinDuvI may have missed some part of the discussion, but on macOS fsync does not flushes the disk cache; it only writes the data to disk. You have to use fcntl(fd, F_FULLFSYNC) to flush the disk cache.
06:18VinDuvI though fsync on Linux worked similarly but maybe not?
06:24marcanif that's how it works, maybe fio should be changed on macos so it stops lying... :p
06:26rkjnsnSounds like it matches how kettenis was describing OpenBSD as working, then, which I suppose is not terribly surprising.
06:42marcanok, can confirm macOS absolutely loses data if you do a plain fsync
06:42marcansimple python script writing to a file (using raw os. calls) and doing fsync() on it, then sleeping
06:42marcanI wait 5 seconds after the write/sync, then reboot via USB-PD command
06:42marcanwrite is gone
06:43marcanin fact I even managed to trigger an inconsistency in an Apple app by accident: I had GarageBand open, and closed it before the test, not saving data. on reboot, it tried to restore the now-nonexistent file, and threw up an error about corruption/invalid project
06:43marcanso I guess it deleted the "unsaved" project but didn't commit the "last open" state to restore on boot
06:47marcanF_FULLFSYNC indeed works
06:50marcanand indeed with a dumb repeated write test I get ~46 IOPS with that, vs. 40000 with plain fsync
06:50marcanooookay then, time to make some noise on twitter
06:51svenat least that explains all the weird behavior
06:55arso, fsync without f_fullfsync is basically a no-op on macos?
06:55marcanit's like fsync on Linux with the write cache set to write-through
06:55marcan(fake set)
06:55marcanso it's not a no-op but it's not good enough
06:56marcangiven the default dirty writeback on Linux is 5 seconds and macOS is losing more than 5 seconds worth of data with this, it *effectively* is like doing nothing on Linux, modulo writeback pressure
06:59marcanso on a Samsung SSD 860 EVO mSATA on my laptop, I get 10K IOPS with the same write cache hack, and ~330 without (and fsync)
06:59marcanso even this mSATA SSD does better than Apple's NVMe
06:59marcanon my iMac with a WD SSD, I get more like 2000 IOPS with proper flushes
07:00marcanand 20000 without
07:01the_lanetly_052__ has joined #asahi-dev
07:04rkjnsnDo we know how much of an issue it would is outside of apt and synthetic benchmarks? I'm curious if it would make sense just to make apt less aggressive with its syncs, given that in can still be a significant slowdown (albeit less of one) on other drives. Presumably most software isn't trying to sync more than 50 times a second…
07:05marcanit's a problem for e.g. databases
07:09rkjnsnOut of curiosity (since I don't know too much about this topic), if folks do want to ignore/postpone flushes on Linux, does the SSD support any other kind of ordering operation that could ensure data written after a flush doesn't hit the disk before data written prior to it?
07:09rkjnsnI know it could still eat data that was supposed to be safely stored, but it'd be nice to avoid broken internal consistency.
07:16VinDuvI’m pretty sure there isn’t a way to properly order the writes since basically the whole chain is allowed to reorder them
07:25marcanI'm not sure if NVMe itself provides barriers; axboe might know
07:25marcanand this might be a good reason to try to rescue that kind of feature...
07:30Glanzmannmarcan: axboe said yesterday that there is no such thing as 'barrieres' for 15 years. And that it is impossible due to multiqueue support IIUC.
07:31rkjnsnI think they were talking about barriers in Linux, as opposed to barriers in the drive, though?
07:31rkjnsnEven without support for write ordering in the kernel, it seems like (IIUC) translating fsyncs to write barriers (assuming the drive supports them) rather than flushes could at least help avoid corruption from the drive reordering things, even if it doesn't protect against writes that were supposedly committed being lost.
07:33svenit's not impossible to do with multiqueues. it's just very challenging.
07:34svenI don't remember seeing anything about write barriers in the nvme spec but I only read that one for the first time a few weeks ago
07:34marcanand yeah, translating fsync()s to write barriers would likely avoid corruption, if the drive can do that
07:37sventhere's force unit access for write commands but that would probably only order writes where it's set (if it enforces the ordering at all)
07:45_jannau_marcan: I don't think dcp in its current state is ready to be integrated in asahi. preserved regions need a coordinated m1n1/kernel changes with a not fully agreed on dt-binding. My current code to remap in the locked dcp dart apparently only works when iboot doesn't initialize dcp (it works only on the mac mini)
07:45marcanack, I'll defer then. I was on the fence about that
07:46marcanwould it be fair to say that if I make I release within this month I should keep simpledrmfb for now?
07:46_jannau_it might be enough to transition dcp to hibernate in m1n1
07:47_jannau_yes, I think simpledrm should be preferred unless someone has time to work on dcp before that
07:50_jannau_I don't think there is huge amount of work to be done to make dcp useful but I should concentrate on submitting spi-hid
08:01MajorBiscuit has joined #asahi-dev
08:02marcanrepro'd on the Mac Mini pulling the plug; there is definitely no (working) last-gasp mechanism
08:19c10l4 has joined #asahi-dev
08:25c10l has quit [Ping timeout: 480 seconds]
08:26arinterestingly, https://sqlite.org/atomiccommit.html#sect_9_2 > Setting fullfsync on a Mac will guarantee that data really does get pushed out to the disk platter on a flush. But the implementation of fullfsync involves resetting the disk controller. And so not only is it profoundly slow, it also slows down other unrelated disk I/O. So its use is not recommended.
08:27marcanwell, this is #10 on hacker news now, so... who knows, maybe Apple will fix it :-)
08:27arso it's not a new problem
08:27marcanar: indeed, the database folks knew about this for a while
08:27marcanit's just amazingly nonobvious for the rest of us
08:29arpostgres also seems to have some references to `fcntl(fd, F_FULLFSYNC, 0)`
08:31dottedmagIs there a way to quickly detect power loss on mini, to cobble last-gasp signal purely in sw?
08:32marcanvery good question. no idea.
08:32marcanthough I imagine if there were Apple would be using it?
08:32marcanlet me look at the schematics for a bit...
08:32marcanah wait, I don't have the Mini ones, derp
08:33marcanthough there might still be some info
08:37marcandottedmag: not seeing anything in SMC, quick look at the MBA schematic (they often have hints as to other variants) doesn't show any place that signal would go
08:37marcanI suspect there is no such mechanism
08:38dottedmagAlas. I was thinking more about some kind of side effect that betrays the power loss.
08:38marcandottedmag: polling the main primary voltage rail voltage, it updates once a second, but I caught it updating immediately before shutdown after pulling the plug and it wasn't drooping
08:39marcanso I suspect the PSU just keeps it up then immediately kills power...
08:39marcanwonder if there's some PGOOD thing that could still work...
08:39dottedmagOK, so no easy way out.
08:39marcanhard to say without the real schematic
09:12tanty has joined #asahi-dev
10:08marcanwell this is interesting
10:09marcanwhile running the python loop, python says 46 ops/s, powermetrics reports 915 disk ops per second / 4.7MB/s (seems a bit much? APFS write amplification? and 20MB/s of ANS2 memory bandwidth in both directions)
10:10marcandoing the same thing, but with an artificial delay to make it run at the same speed sans the flush, I get 44 disk ops/s / 180 KB/s, and ~200 KB/s of ANS2 memory bandwidth
10:11marcanso that sounds like two problems here... FULLFSYNC does some horrible APFS amplification nonsense, *and* ANS2 blows it up even more
10:11marcanlet me try another filesystem...
10:14marcanon FAT32 it's actually slower (34 IOPS), but no serious amplification: 78.37 ops/s 321.02 KBytes/s
10:14marcanANS2 RD : 3.042 MB/s
10:14marcanANS2 WR : 8.836 MB/s
10:15marcanso yeah, ANS2 is definitely doing something somewhat dodgy if it's doing 9MB/s of memory write traffic and 3MB/s of memory read traffic to serve 320KB/s of data traffic
10:16marcansame test without the full fsync: write: 93.52 ops/s 397.77 KBytes/s
10:16marcanANS2 DCS RD : 0.244 MB/s
10:16marcanANS2 DCS WR : 0.411 MB/s
10:16marcanthat's more like it
10:16marcanso yeah, that NVMe sync command is making ANS2 do a lot of work...
10:16marcanI wonder if it, like, linearly scans some huge cache hashtable?
10:19Dcow[m]is amount of work depends on the storage size?
10:27XeR has joined #asahi-dev
10:44refi64 has quit [Read error: Connection reset by peer]
10:45refi64 has joined #asahi-dev
11:01svenso that just leaves that weird issue where we sometimes miss an interrupt now
11:11rkjnsnI saw a reference to a F_BARRIERFSYNC. Not sure if that's a macOS thing or just an iOS thing, but if it's available on macOS, it might be worth seeing if it works as advertised and results in less performance loss.
11:11mazsven: contrary to what I said, the AIC doesn't seem to have a configuration for edge/level, and "knows" which line in which. and the fasteoi flow doesn't distinguish them either.
11:16svenyeah, that's what I thought after spending some time with the interrupt code yesterday as well
11:21kettenisdoes the NVMe core code implement different code paths for MSI and non-MSI?
11:24mazkettenis: if it does, the latter probably isn't very well tested...
11:37kettenissven: so on OpenBSD I still set the "number of openings" to 1 to avoid the NVMe from locking up
11:38ketteniswhich effectively means command sumission is serialized
11:38kettenisthe issue I ran into sounds somewhat similar to what you're seeing
11:39kettenisat some point the completion interrupt for a command never happens
11:39kettenisalthough in my case the command actually didn't complete as far as I could tell
11:40kettenisnot optimal, but still plenty fast and it has been rock solid since I added that hack
11:41sventhat sounds very similar actually. some commands never generate the completed interrupt but they do appear in the completion queue and can be polled but others don't ever seem to be triggered
11:42svenwhat exactly does "number of openings" do exactly?
11:42svenmacos uses INTMS and INTMC in its interrupt handler apparently, which linux doesn't because it relies on the interrupt controller itself
11:43kettenisI think "number of openings" is classic SCSI terminology for the number of commands that can be in flight simultaniously
12:01MajorBiscuit has quit [Ping timeout: 480 seconds]
12:04gladiac has quit [Quit: k thx bye]
12:05gladiac has joined #asahi-dev
12:17MajorBiscuit has joined #asahi-dev
13:01Gaspare has joined #asahi-dev
13:43marcanfor shits and giggles (and because HN commenters are tiring): full fsync on a shitty USB3 flash drive on macOS/M1: 223 IOPS. internal NVMe, 58 IOPS. Both FAT32.
13:44alyssamarcan: *blinks*
13:44marcan(of course, the flash drive has no cache at all, so it's equally slow with a vanilla fsync :p)
13:44marcanalyssa: yes, you get better database transaction latencies on a shitty flash drive than on internal NVMe on these machines :-)
13:45alyssamarcan: and if you wear out the flash drive you're not fsck()'d :V
14:00phiredid they just not try for decent fsync preformance at all?
14:04marcanI suspect nobody noticed
14:04marcansince it doesn't matter on iOS
14:12svenaaand now I can't reproduce the missing interrupts at all anymore with the original code *sigh*
14:13povikfiling bugs with Apple by getting on HN is an interesting method
14:13svenevery team needs someone to do the mediawhoring ;)
14:14marcanthe only reason I haven't deleted twitter is those 50k followers are *sometimes* useful :p
14:14as400[m]sven: are you suggesting that marcan is a new apple spokesperson ?
14:27alyssamarcan: unfollowing everyone and setting all privacy settings to "following only" has made my twitter quiet! :-p
14:32Major_Biscuit has joined #asahi-dev
14:33MajorBiscuit has quit [Ping timeout: 480 seconds]
14:35marcanI did set that for notifications :p
14:39alyssathat's step 2!
14:41yuyichao has quit [Ping timeout: 480 seconds]
14:48marcanhow's that for a scary demo?
14:49alyssai found the data!
14:49alyssa`-$ echo "very important data" > file.txt` <-- it's right here
14:49j`eymarcan: lul nice
14:56marcanthat said, I'm not even entirely sure if rsync tries to do a normal fsync() here
14:56marcanso it might just be unsafe on any system :p
14:57marcanbut fsync() certainly wouldn't save macOS
14:59yuyichao has joined #asahi-dev
15:06hays has quit []
15:07milek7_rsync doesn't do fsync at all
15:10hays has joined #asahi-dev
15:26hays has quit []
15:28hays has joined #asahi-dev
15:29axboe has joined #asahi-dev
15:30axboeI already had 3 separate people send me marcans flush rant this morning ;)
15:30axboegood to see some noise on this topic
15:37Gaspare has quit [Ping timeout: 480 seconds]
15:44Jamie[m]1is radar priority decided directly based on HN rank, or is twitter likes the bigger factor? :P
15:59svenaxboe: remind me again, is taking the anv->lock around the writel itself enough to prevent those timeouts or did the memcpy also have to be inside the lock?
16:00sveni can't seem to reproduce the issue at all even without the lock right now :/
16:06axboesven: just around writel is enough, that's what I've been running and it's been rock solid
16:07axboesven: reproduces trivially for me, was a bit harder with issues serialized, but could still hit it. consistently just doing a make -j8 kernel compile on it
16:07axboewould always hang
16:07axboeI'm running the three nvme patches now and it's all dandy
16:07axboetcb clear cleanup, lock around writel, and the flush deferral
16:08svenokay, so from looking at the hypervisor logs macos seems to never interleave "ring cq doorbell + nvmmu invalidation" with "write new tag to sq"
16:08svenso maybe if something like "start nvmmu invalidation, start new command, write cq doorbell" happens things break. no idea why though.
16:08axboeso that won't happen with that patch either then
16:09axboeas nvmmu invalidation is already under the anv lock
16:09svenyeah. i'm just trying to understand why we need that lock around the writel
16:09axboeI generally didn't feel comfortable without that lock to begin with, only part I'm a bit puzzled on is why the cpu freq changes makes this so much more likely to trigger
16:10axboeit'd be better if we didn't need to share this lock (and particularly since it's irq disabling on the submit path), but at the rates of these drives and the fact that it only has a single queue anyway, it's not going to be an iops monster anyway and hence it doesn't really matter
16:11axboeso while that could be improved, it doesn't matter in practice imho
16:11svenmakes sense
16:12svenif it's actually a race between nvmmu/cq_db and that writel maybe the higher cpufreq just makes it more likely to lose (or win, depending how we look at it :D) that one
16:13axboedefinitely some correlation there, just not sure what!
16:13axboeI ran quite a bit with the cpus at max freq before that
16:13axboeand didn't see it
16:13axboemaybe differences in clock between simultaneous issuers? though I don't see how...
16:13svenquite a few people have been using it on the max/pro as well before cpufreq as well and never saw it either.
16:14svenyeah, i just hate when a mystery lock solves something that smells like some kind of race
16:14axboeyeah I know
16:14axboeyour osx tracing sounds useful though
16:15axboeif it is indeed inval vs doorbell write
16:15axboeif only there was documentation ;)
16:15svenyup :D
16:15axboeI think we just need a good comment around why we _think_ it's needed
16:16axboein the commit message too, but more importantly in the code
16:16svenyeah, absolutely
16:16axboeirq fix is queued up btw, but I guess you saw that already
16:16svenyup, that was pretty quick :)
16:16axboeone less to worry about :)
16:18axboeoh forgot, also running that "set aq depth to 2" patch
16:31axboesven: though for the batching, we could optimize the nvmmu_inval by writing each tag, then one readl_relaxed at the end...
16:33svenmacOS does a readl_relaxed after each inval fwiw. I’m not sure if that’s required ofc.
16:33svencould be worth a try
16:33axboeprobably not worthwhile to pursue, but something to keep in mind
16:34axboeI've got this funky series for io_uring that allows registered buffers to retain dma and iommu mappings, instead of doing them for each io submit and complete
16:34alyssa"if only there was documentation ;)"
16:34axboeit'd bump the peak 17xK rand iops to substantially more
16:35axboewas planning on adding support for apple nvme just to test out what it can actually do, if I do that, then I can try the nvmmu inval optimization too as it'd likely make a difference at that point
16:35axboewas part of the "lets see how many iops we can do on a core" experiments, 1 of 2 series that hasn't been posted anywhere yet as it's a bit of a hack
16:36svenI saw part of those optimization on twitter. Very impressive how much it improved
16:37axboeit was a fun project
16:38axboethere's no way to test apple-nvme in qemu yet, is there?
16:40axboeah screw it, we'll do it live
16:40axboelet's find out
16:40svenwhat could go wrong ;)
16:51axboedone, let's see if it works...
16:53axboe has quit [Quit: reboot]
17:00the_lanetly_052___ has joined #asahi-dev
17:02skipwich has quit [Quit: DISCONNECT]
17:02skipwich has joined #asahi-dev
17:06the_lanetly_052__ has quit [Ping timeout: 480 seconds]
17:06axboe has joined #asahi-dev
17:06axboeit worked
17:07axboenot sure why I seem to be hitting the segments != 1 path in apple_nvme_map_data() though
17:10axboelet's debug...
17:11axboe has quit [Quit: Lost terminal]
17:14axboe has joined #asahi-dev
17:14axboeoh, it's the 16k page size
17:14axboeI should align my buffers better and not assume 4k page sizes :)
17:14axboeand then apple-nvme needs to support ->queue_rqs too
17:16bpye3 has joined #asahi-dev
17:17al3xtjames2 has joined #asahi-dev
17:18skipwich has quit [Ping timeout: 480 seconds]
17:19alyssasven: so uh what does this mean for apple-nvme upstreaming?
17:19al3xtjames has quit [Quit: Ping timeout (120 seconds)]
17:19alyssai guess we're still super blocked on rtk?
17:19al3xtjames2 is now known as al3xtjames
17:23skipwich has joined #asahi-dev
17:23bpye has quit [Ping timeout: 480 seconds]
17:23bpye3 is now known as bpye
17:32axboesven: ok queue_rqs is hard to support, since the sq door bell is writing the specific tag...
17:32axboeoh well, guess we can't easily do that
17:32axboebut with the changes, we get around the same perf, but at about half the CPU usage
17:33axboeso guessing we're actually controller limited on iops anyway around that point
17:35rkjnsnmarcan, would it be easy to test F_BARRIERFSYNC using your python script to see what it does and how it performs on these machines?
17:44Major_Biscuit has quit [Ping timeout: 480 seconds]
17:44axboe has quit [Quit: Lost terminal]
17:59the_lanetly_052___ has quit [Ping timeout: 480 seconds]
18:03svenaxboe: half the cpu usage already sounds great :)
18:05svenalyssa: yeah, I need to take a look at what marcan did to rtkit and we need to discuss how we will upstream that together with smc
18:14alyssasven: right
18:28axboe has joined #asahi-dev
18:29axboesven: got queue_rqs setup, and it's just cpu reduction at this point, iops are capped / controller limited
18:29axboeso not worth pursuing imho
18:29svenaww… was worth a try though!
18:30axboedoing 32 issues batched, no real gain :/
18:30axboebut yeah, worth an experiment
18:30axboebeats real work, right?
18:30svenyup :)
18:38sirn- has joined #asahi-dev
18:45sirn has quit [Ping timeout: 480 seconds]
18:45sirn has joined #asahi-dev
18:52sirn- has quit [Ping timeout: 480 seconds]
18:52sirn- has joined #asahi-dev
18:53sirn| has joined #asahi-dev
18:56sirn^ has joined #asahi-dev
18:57sirn has quit [Ping timeout: 480 seconds]
18:57sirn^ is now known as sirn
18:58axboe has quit [Quit: Lost terminal]
18:59axboe has joined #asahi-dev
19:00sirn- has quit [Ping timeout: 480 seconds]
19:03sirn| has quit [Ping timeout: 480 seconds]
19:19___nick___ has joined #asahi-dev
19:30alyssaaxboe: i haven't seen a ./configure in a while...
19:31Telvana has joined #asahi-dev
19:33axboealyssa: it's a handrolled one ;)
19:33axboebut don't judge, fio is probably around 20 years old at this point
19:40axboebtw, if someone can test the above on osx, would be great
19:40axboeor I'll have to borrow the wife's laptop later
19:44svenI can give it a try later
19:45___nick___ has quit []
19:46axboesven: thanks!
19:47___nick___ has joined #asahi-dev
19:49___nick___ has quit []
19:51___nick___ has joined #asahi-dev
19:52jannauaxboe: configure checks for F_FULLSYNC instead of F_FULLFSYNC
19:53jannaumissing 'F'
19:53axboejannau: doh
19:53jannausame in code
19:55axboethanks for spotting that, pushed out a fix
19:55* jannauredirects the thanks to the compiler
19:55axboesomeone needs to run the compiler on osx :)
19:56alyssathat would require owning a mac.
19:56axboedefinitely not typing this on the m1
19:56alyssano of course not
20:11jannauaxboe: 51 IOPS for 4k randwrite with --fsync 1 and F_FULLFSYNC
20:12axboejannau: great thanks, seems it's doing its job then
20:13axboeonly about 2x slower than a rotating hard drive, nice
20:13axboeanyone know if F_FULLFSYNC is osx only, or do some of the bsd's use it too?
20:14axboeI guess I can google
20:14axboelooks like freebsd/netbsd at least do it saner and provide a sysctl setting for it
20:21svenalmost the same here, 49 IOPS
20:24kettenisaxboe: looks like osx only; even FreeBSD doesn't have it
20:26axboekettenis: yeah looks like the bsds took the "make it a system tunable" instead, which does make a lot more sense than a separate command
20:41axboesven: pushed the nvme test stuff there, just in case you are curious
20:44svenvery curious actually!
20:46axboethis is the perf-wip patches for persistent maps cherry-picked, then the apple-nvme bits needed on top
20:46axboeand queue_rqs is actually in 5.16, so could be done
20:46axboeidea there is based on how linux manages IO in an on-stack plug for submitting more than 1
20:47axboeso it allows you to submit N ios at the time
20:47axboeammortize the cost of grabbing the submit side lock
20:47axboethat part works a bit better on "normal" nvme, as we just have to do that one doorbell write with the sq tail
20:54svenah, true. Just copy all commands to the queue and advance it by N elements with a single write
21:08___nick___ has quit [Ping timeout: 480 seconds]
21:33robher has quit [Read error: Connection reset by peer]
21:33kendfinger has quit [Read error: Connection reset by peer]
21:33weems_ has quit [Read error: Connection reset by peer]
21:33daniels has quit [Read error: Connection reset by peer]
21:33arnd has quit [Read error: Connection reset by peer]
21:33weems_ has joined #asahi-dev
21:33daniels has joined #asahi-dev
21:33kendfinger has joined #asahi-dev
21:33arnd has joined #asahi-dev
21:33tardyp has quit [Read error: Connection reset by peer]
21:33tardyp has joined #asahi-dev
21:33robher has joined #asahi-dev
21:58axboe has quit [Ping timeout: 480 seconds]
22:06axboe has joined #asahi-dev
22:08thermoblue[m] has left #asahi-dev [#asahi-dev]
23:34axboe has quit [Ping timeout: 480 seconds]
23:57axboe has joined #asahi-dev
…