#soylent | Logs for 2026-03-10
« return
[01:00:13] -!- bender has quit [Remote host closed the connection]
[01:00:28] -!- bender [bender!bot@Soylent/Bot/Bender] has joined #soylent
[01:02:06] -!- Loggie [Loggie!Loggie@Soylent/BotArmy] has joined #soylent
[01:58:53] -!- jje has quit [Quit: ZNC - https://znc.in]
[03:00:23] -!- jje [jje!jje@ddjffrbpjin.info] has joined #soylent
[03:00:25] -!- jje has quit [Read error: -0x7880: SSL - The peer notified us that the connection is going to be closed]
[03:51:58] -!- c0lo [c0lo!~c0lo@124.190.vj.gtm] has joined #soylent
[03:53:21] <c0lo> Perpetual Error 503 Backend fetch failed
[04:43:45] <chromas> Yay, finally a different failure mode
[04:43:56] <chromas> at least, different from the recent failures
[06:47:35] -!- chromas has quit [Ping timeout: 265 seconds]
[06:48:16] -!- chromas [chromas!~chromas@Soylent/Staph/Infector/chromas] has joined #soylent
[06:48:16] -!- mode/#soylent [+v chromas] by Imogen
[08:46:13] <Runaway> Error 503 Backend fetch failed
[08:46:13] <Runaway> Backend fetch failed
[08:46:13] <Runaway> Guru Meditation:
[08:46:13] <Runaway> XID: 2594205
[08:46:13] <Runaway> Varnish cache server
[08:46:44] <Runaway> I guess c0lo already reported it, but there it is.
[09:21:37] <c0lo> We're letting chromas enjoy the change in the scenery for a while.
[09:27:10] <janrinok> I've tried to contact kolie but no response yet. I am trying to manage an unexpected death in my family and I cannot reset the system.
[09:33:11] <c0lo> Sorry about your loss, janrinok. Don't fuss, chromas seems to enjoy it still.
[09:52:37] <c0lo> https://www.youtube.com
[09:54:26] -!- c0lo has quit [Remote host closed the connection]
[09:57:10] -!- Loggie [Loggie!Loggie@Soylent/BotArmy] has joined #soylent
[09:57:40] -!- Runaway1956 [Runaway1956!~OldGuy@the.abyss.stares.back] has joined #soylent
[09:57:48] -!- prg [prg!~prg@ryfhuvb.de] has joined #soylent
[09:57:48] -!- Xyem [Xyem!~xyem@Soylent/Staff/Developer/Xyem] has joined #soylent
[09:57:48] -!- mode/#soylent [+v Xyem] by Imogen
[09:58:03] -!- Bytram [Bytram!~Bytram@Soylent/Staff/Developer/martyb] has joined #soylent
[09:58:03] -!- mode/#soylent [+v Bytram] by Imogen
[09:58:04] -!- progo [progo!~progo@eegc-73-589-96-43.nwrknj.fios.verizon.net] has joined #soylent
[09:58:42] -!- fab23 [fab23!~fabian@ysxbxfyv.wenks.ch] has joined #soylent
[09:58:46] -!- SSLbits [SSLbits!~SSLbits@CanHazVHOST/SSLbits] has joined #soylent
[09:58:54] -!- Deucalion [Deucalion!~Fluff@Soylent/Staff/IRC/juggs] has joined #soylent
[09:58:54] -!- mode/#soylent [+v Deucalion] by Imogen
[09:58:58] -!- chromas [chromas!~chromas@Soylent/Staph/Infector/chromas] has joined #soylent
[09:58:58] -!- mode/#soylent [+v chromas] by Imogen
[09:59:03] -!- Fnord666_ [Fnord666_!~Fnord666@Soylent/Staff/Editor/Fnord666] has joined #soylent
[09:59:03] -!- mode/#soylent [+v Fnord666_] by Imogen
[09:59:24] -!- janrinok [janrinok!~janrinok@Soylent/Staff/Editor/janrinok] has joined #soylent
[09:59:24] -!- mode/#soylent [+v janrinok] by Imogen
[09:59:33] -!- inz [inz!~inz@wbi.fi] has joined #soylent
[10:04:32] <janrinok> I've tried a full reboot but it doesn't seemed to have resolved the problem. We will have to wait for kolie.
[10:11:29] -!- ted-ious [ted-ious!tedious@ted.ious] has joined #soylent
[10:13:32] -!- bender has quit [Remote host closed the connection]
[10:13:40] -!- bender [bender!bot@Soylent/Bot/Bender] has joined #soylent
[10:13:54] -!- devbot has quit [Remote host closed the connection]
[10:14:17] -!- Loggie [Loggie!Loggie@Soylent/BotArmy] has joined #soylent
[10:14:36] -!- lld [lld!~lld@llvm.link.editor] has joined #soylent
[10:14:59] -!- c0lo [c0lo!~c0lo@124.190.vj.gtm] has joined #soylent
[10:24:38] -!- halibut [halibut!~halibut@CanHazVHOST/halibut] has joined #soylent
[10:40:49] -!- Ingar [Ingar!~ingar@2001:bc8:1640:mwzr:pxpv:wq:jjgg:moil] has joined #soylent
[10:41:03] <Ingar> Error 503 Backend fetch failed !
[10:41:09] <Ingar> The Guru is meditating
[10:49:25] <c0lo> Nothing new, Ingar, that's like this for 8h+
[11:22:39] -!- jje [jje!jje@ddjffrbpjin.info] has joined #soylent
[11:28:41] <Ingar> that's a long meditation
[12:18:24] -!- jman [jman!~osxtra@136.49.vyy.ix] has joined #soylent
[12:20:18] <jman> Howdy, in case no one knows, sad to report getting 503's to the site. It says Backend fetch failed, XID 331140
[12:21:27] <jman> FWIW last night's email digest came in though, around 2100 CST...
[12:25:41] -!- cosurgi [cosurgi!~cosurgi@Soylent/Staff/Misc/cosurgi] has joined #soylent
[12:39:21] <Runaway1956> https://www.malone.news
[13:54:52] -!- jman has quit [Quit: WeeChat 4.8.1]
[14:02:30] -!- kolie2 [kolie2!~not_kolie@Soylent/Staff/Management/kolie] has joined #soylent
[14:02:30] -!- mode/#soylent [+o kolie2] by Imogen
[14:02:48] <kolie2> morning
[14:03:22] <kolie2> (Got an error reading communication packets)
[14:03:24] <kolie2> not sure whats causing it but i can't look at it rn, got work to do for the day
[14:03:35] <kolie2> docker compose logs database
[14:03:41] <kolie2> shows that
[14:03:52] <kolie2> so something with the rehash or db container possibly relating to versions or something
[14:09:08] <kolie2> Running a container rebuild dc up rehash --build -d but i can't stick around other then to try that rn.
[14:54:54] -!- jje has quit [Quit: ZNC - https://znc.in]
[14:56:21] -!- jje [jje!jje@ddjffrbpjin.info] has joined #soylent
[15:19:59] -!- AlwaysNever [AlwaysNever!~donaldo@315.38.1.669.dynamic.jazztel.es] has joined #soylent
[15:26:09] <AlwaysNever> error 503 on the site, has a reboot been tried already, of the VM where the Docker containers run?
[15:50:34] -!- jje has quit [Quit: ZNC - https://znc.in]
[15:51:20] <AlwaysNever> If the problem in RAM exhaustion, that should fix it (temporarily), if that does not fix it, then some logs are probably filling some filesystem
[15:58:55] <chromas> kolie gave an instruction to restart the servers, but it looks like he and janrinok aren't able to run it just yet (and I don't haz access to run it), so it may be awhile
[15:59:27] <chromas> oh hah I just saw the kolie2 messages here too
[16:13:03] -!- c0lo has quit [Quit: Ping timeout (120 seconds)]
[16:13:11] -!- jje [jje!jje@ddjffrbpjin.info] has joined #soylent
[16:33:13] <janrinok> AlwaysNever, I have tried several times to restart the servers. There is something preventing it but I cannot identify what it is, nor do I have the knowledge to fix it even if I did know.
[16:36:06] <janrinok> At the same time I have got several real world issues, including the unexpected death of a sibling.
[17:09:17] -!- mode/#soylent [+o chromas] by Imogen
[17:30:09] -!- soylentil41 [soylentil41!~soylentil@Soylent/Staff/Management/kolie] has joined #soylent
[17:30:09] -!- mode/#soylent [+o soylentil41] by Imogen
[17:30:30] <soylentil41> o/
[17:31:11] -!- soylentil41 has quit [Client Quit]
[17:38:44] -!- jje has quit [Quit: ZNC - https://znc.in]
[17:41:37] -!- jje [jje!jje@ddjffrbpjin.info] has joined #soylent
[18:03:49] <AlwaysNever> janrinok: is it that your brother passed away? I'm sorry for your loss.
[18:46:58] <janrinok> Thank you.
[19:53:31] -!- c0lo [c0lo!~c0lo@124.190.vj.gtm] has joined #soylent
[20:11:37] -!- jje has quit [Quit: ZNC - https://znc.in]
[20:30:53] -!- jje [jje!jje@ddjffrbpjin.info] has joined #soylent
[21:02:23] -!- mrpg [mrpg!~mrpg@Soylent/Staff/Editor/mrpg] has joined #soylent
[21:02:23] -!- mode/#soylent [+v mrpg] by Imogen
[21:02:58] <mrpg> Gentlemen. Good day.
[21:17:23] <chromas> g'day
[22:05:43] -!- kolie3 [kolie3!~kolie3@Soylent/Staff/Management/kolie] has joined #soylent
[22:05:43] -!- mode/#soylent [+o kolie3] by Imogen
[22:06:04] <kolie3> Ill start more troubleshooting in a little over 2 hours
[22:06:19] <kolie3> It wasnt quick and I had a full day of work today.
[22:06:50] <kolie3> Looks like the rehash container cant communicate with the database
[22:07:00] <kolie3> It doesnt appear to be a networking issue
[22:07:22] <kolie3> I get some errors on the db side with connections
[22:07:53] -!- undefined [undefined!~undefined@66.171.jo.vll] has joined #soylent
[22:07:53] <kolie3> Why now idk cause shouldn't be updated images ( I did a rebuild this morning though)
[22:08:31] -!- kolie3 has quit [Client Quit]
[22:15:23] <mrpg> Thanks for the update, I blame systemd.
[22:15:27] <mrpg> !blame
[22:15:30] <mrpg> =blame
[22:15:41] <mrpg> $blame
[22:15:53] <mrpg> ?blame
[22:16:00] <mrpg> !"#$%&/blame
[22:16:04] <mrpg> &blame
[22:16:29] <mrpg> .
[22:17:11] <undefined> FYI, I was on when the system went down; it got slower and slower before it finally coughed up the backend error message.
[22:18:16] <undefined> Took about 2 minutes to finally post a reply to a journal, then to preview a story submission,then boom.
[22:28:00] -!- undefined has quit [Quit: Client closed]
[22:28:43] -!- undefined [undefined!~undefined@66.171.jo.vll] has joined #soylent
[22:29:39] -!- undefined has quit [Client Quit]
[22:33:52] -!- undefined [undefined!~undefined@66.171.jo.vll] has joined #soylent
[22:34:32] -!- undefined has quit [Client Quit]
[22:38:02] <AlwaysNever> undefined: those symptoms suggest resource starvation: but it is CPU starvation or RAM starvation?
[22:38:56] <AlwaysNever> kolie3: is all the Docker-Compose "thing" running on a single VM? What's the RAM assigned to that VM?
[22:39:30] <mrpg> Maybe there's a copy of the DB in RAM?
[22:49:45] <mrpg> So RAM starved.
[23:03:47] -!- mrpg has quit [Quit: mrpg]
[23:31:45] -!- undefined [undefined!~undefined@66.171.jo.vll] has joined #soylent
[23:59:28] -!- undefined has quit [Quit: Client closed]