#dev | Logs for 2014-07-30

« return
[23:01:04] <xlefay> paulej72: that's also what I read; that's why I wasn't sure because the file does exists on the master, so I figured, perhaps NC was going to make a multiple-master like scenario if that's even possible with kerberos.
[23:00:13] <paulej72> xlefay: Form what I am reading that file on the master is not used so it should not be an issue
[22:59:37] <xlefay> paulej72: can you let me know how it goes after you've changed that ACL list?
[22:56:20] <xlefay> stuff still worked.. hence why I left it be at least, that was my plan until NC came around so I could inquire)
[22:56:20] <xlefay> @ Boron, it's correct, at helium, it seems incorrect. I mentioned it to NC the other day, when we had the issue with Kerberos & kpropd, but he disappeared, so I could never confirm whether it was correct or not and since I don't have a lot of experience with kerberos, I didn't feel it right to correct it (for all I know, a slight configuration error was made and not corrected and thus the kpropd.acl was correctly formatted as per that error, but
[22:53:49] <xlefay> Notice, the 'li694@' instead of 'li694-22@'
[22:53:35] <xlefay> host/kdc-02.li694@LI694-22
[22:53:35] <xlefay> host/kdc-01.li694@LI694-22
[22:53:35] <xlefay> root@helium:/etc/krb5kdc# cat kpropd.acl
[22:53:34] <xlefay> paulej72:
[22:53:02] <paulej72> xlefay: yes I think that was it. the last part
[22:52:09] <xlefay> Is that what you meant? I'm sorry doing several things at once, so it's hard to keep track
[22:50:27] <xlefay> The hostname ACLs appearing wrong on helium were in the /etc/krb5.../kpropd.acl file, but that's just how it appeared - not sure if it's actually wrong but it does appear to be missing part of our domainname.
[22:49:50] <xlefay> The issue, of ssh'ing? Or, the issue we had with kerberos not updating? Or do you mean, the hostname ACLs appearing wrong on helium?
[22:48:05] <paulej72> do you recall the exact issue that you saw. I want to make sure I fix this properly
[22:48:03] <xlefay> Be careful though, best to make backups before editing stuff, kerberos' is prone to the slightest configuration error as it seems
[22:47:16] <xlefay> Yeah, I don't know the full path, but tab complete will get you there, it starts with /etc/krb5
[22:46:18] <paulej72> can’t remember, but I thought it was the krb5 stuff
[22:45:36] <xlefay> paulej72: domains, as in, resolv.conf domainname, or in the kerberos config files which are in /etc/krb5.../krb5...
[22:45:07] <xlefay> I'm good ;)
[22:45:01] <paulej72> xlefay: back when we were first having kerveros problems you found that the domains were not quit the same on helium and boron. do you remember where those setting were.
[22:44:34] <juggs> ok thanks. And you?
[22:43:45] <xlefay> How are you by the way?
[22:43:41] <xlefay> haha, I'm kidding
[22:42:09] <juggs> sure, sure - I'm getting the round in
[22:40:50] <xlefay> juggs: haha no worries, but while you're at it, can you bring me a cup?
[22:39:14] <juggs> Ah, I see. /me stops being the idiot looking over the shoulders of people while making unhelpful comments and wanders off to get the coffees :D
[22:37:08] <xlefay> juggs: we can access every server, we're not locked out or anything. It's just, using kerberos to access beryllium doesn't work, but direct SSH'ing into it, works just fine
[22:27:21] <juggs> random thought - do you have a virtual console on the troublesome box? Or would that also be knobbled by a kerberos / whatever it is issue?
[22:19:24] juggs|afk is now known as juggs
[22:17:49] juggs is now known as juggs|afk
[22:13:56] * TheMightyBuzzard starts whistling Floyd's Money
[22:13:05] <xlefay> That's good. I gotta go do this swag task force stuff now though... has to be done
[22:11:42] <TheMightyBuzzard> ya, i take weekends off even if live catches on fire
[22:11:27] <xlefay> but if you ever change your mind and want to be on, just say the word
[22:11:14] <xlefay> TheMightyBuzzard: well, I'm sure we'll be happy to have you, but, no need to spread yourself to thin (e.g. be sure to have an actual life)
[22:10:36] <TheMightyBuzzard> xlefay, s'actually hard to keep myself from asking to jump on the admin team too but i get enough of that in my day job.
[22:10:14] <paulej72> xlefay: sounds ok to me
[22:09:49] <paulej72> I think the kernel may have been updated
[22:09:47] <xlefay> paulej72: alright. Do you mind if I leave it in your hands? I have to do some stuff for the Swag Task Force. Feel free to message me though if you need more information but I'm pretty much useless when it comes to kerberos, it's been on my todo list for a long time.. but you know how that goes..
[22:08:55] <TheMightyBuzzard> xlefay, i noticed that too. gave me a chuckle.
[22:08:42] <xlefay> TheMightyBuzzard: ironically.. when I signed into boron, it's motd was complaining it requires a reboot...
[22:08:38] <paulej72> yes I know, I am trying to see what the issue might be
[22:08:22] <TheMightyBuzzard> or rather a few minutes ago
[22:08:22] <xlefay> e.g. not being able to ssh from boron to beryllium (unless, using ssh to proxy)
[22:08:15] <TheMightyBuzzard> paulej72, what all did you reboot just now?
[22:07:57] <xlefay> paulej72: our autossh configs use kerberos to ssh. So, the kerberos issue is what's stopping it from working.
[22:07:19] <paulej72> xlefay: I am looking at autossh config right now to see if I can figure it out
[22:07:09] <xlefay> correct it ourselves* , not correct ourselves, we're fine gents so..
[22:06:49] <xlefay> TheMightyBuzzard: ha. pseudo-random works for me.
[22:06:40] <TheMightyBuzzard> i can fake "random" but you know nothing is ever really random.
[22:06:36] <xlefay> anyway, paulej72 how can we correct the current mistake? Is there something that we can do to correct ourselves or do we have to wait for input from NCommander?
[22:05:55] <TheMightyBuzzard> i can do "worst possible" or "most inconvenient"
[22:05:25] <xlefay> Ignore me.
[22:05:21] <xlefay> (although, granted, most of us could actually code it to do just that..) I'm just messing around, happens when I get tired.
[22:05:15] * TheMightyBuzzard scratches his head
[22:04:56] <xlefay> Thus, it goes right into your alley good sir
[22:04:49] <xlefay> TheMightyBuzzard: I know.. but we sysops can configure it, but we can't actually code it to fail at the appropriate times...
[22:04:27] <xlefay> paulej72: ah I see. I think we all have so much shit to do that it's easy to forget stuff
[22:04:22] <TheMightyBuzzard> <---dev not sysadmin
[22:04:09] <xlefay> Surely, that beats doing paypal crap
[22:04:00] <paulej72> xlefay: most of the problems from kerberos stem form the failed upgrade of helium that NCommander has failed to fix.
[22:03:59] <xlefay> As I just said, Kerberos has been quite inconsiderative of late. Perhaps, you could make it more, considerative?
[22:03:33] <TheMightyBuzzard> shoot
[22:03:28] <xlefay> TheMightyBuzzard: perhaps I could ask a favor of you, good sir?
[22:02:42] <xlefay> (I'm rather getting annoyed by kerberos to be honest. It's awesome, but it's been giving us grieve at random times, which is, quite inconsiderative if you ask me.)
[22:02:08] <xlefay> yeah, my brain's tired anyway, how do we fix kerberos this time
[22:01:40] <paulej72> not it was parsable
[22:01:35] <xlefay> nvm, I get it now... boron is our ssh proxy /feels stupid now
[22:01:20] <xlefay> OOH
[22:01:11] <xlefay> LOL
[22:01:10] <xlefay> eh, that sentences makes no sense
[22:00:59] <xlefay> Unless I've been mistaken all these years, so even though my connection from boron went wrong, it still recorded it as a login?
[22:00:17] <xlefay> paulej72: correct, but I'm refering to the last login time
[21:59:53] <paulej72> that method bypasses kerveros
[21:59:47] <xlefay> A bit odd
[21:59:26] <xlefay> Last login: Wed Jul 30 21:58:11 2014 from carbon.li694-22
[21:59:26] <xlefay> ssh beryllium.li694-22
[21:59:25] <xlefay> I tried to ssh just now from boron, it gave me that error but... from localhost:
[21:59:12] <paulej72> yes it is kerberos, because I was able to ssh from helium to bery
[21:59:08] <xlefay> Well this is odd, paulej72
[21:58:44] * xlefay bets kerberos...
[21:58:27] <xlefay> woa...
[21:58:25] <xlefay> Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[21:58:25] <xlefay> [21:58] xlefay@boron $ ssh beryllium
[21:57:52] <xlefay> Ugh I hate this part where all shit goes up "CRITICAL" || "UNKNOWN"
[21:57:48] <paulej72> NCommander: has oxygen been paid for this month?
[21:57:38] <xlefay> Remote command execution failed: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[21:56:51] <paulej72> I did not go to beryllium, but this is the server outpunagios 32656 1 0 21:38 ? 00:00:00 /usr/lib/autossh/autossh -M 0 -N -T -o ControlMaster=yes -o ControlPath=/var/run/icinga/ssh.icinga@beryllium.li694-22.socket -o ServerAliveInterval 60 -o ServerAliveCountMax 3 icinga@beryllium.li694-22
[21:56:08] <paulej72> He almost did that last month, I hope he paid it this month
[21:55:31] <xlefay> Anyway, ehm, did you reset the autossh stuff for beryllium?
[21:55:21] <xlefay> Perhaps, the server expired and NCommander didn't renew it?
[21:55:09] <paulej72> shit
[21:54:54] <xlefay> paulej72: icinga does report oxygen as being down for at least 4 days now
[21:54:35] <xlefay> Ehm... yeah, then I got nothing
[21:54:22] <xlefay> Really? Wow.
[21:54:15] <paulej72> ssh: connect to host to port 22: Connection refused
[21:53:49] <xlefay> paulej72: ssh to 2001:41d0:1:dfa9::1
[21:53:12] <xlefay> Anyway, ehm, oxygen that's a tough one
[21:53:01] <paulej72> yea
[21:52:49] <xlefay> paulej72: I just want to note, this is a fine example why containers are better ;)
[21:52:47] <paulej72> OK we still need to figure out oxygen
[21:52:30] <xlefay> TheMightyBuzzard: it was probably just a package conflict that forced the removed all of some packages
[21:52:25] <TheMightyBuzzard> yays
[21:52:11] <xlefay> https://sentinel.soylentnews.org and it's back to life
[21:52:08] <paulej72> xlefay: i did some package updates, it might have been me.
[21:52:07] <TheMightyBuzzard> has me wondering wtf happened because if it was anyone getting their malice on, they did a pretty poor and very specific job of it
[21:50:36] <xlefay> TheMightyBuzzard: ok, seems like only one package or so icinga requires was removed.
[21:49:48] <xlefay> not sure if that still exists in the dns though..
[21:49:37] <xlefay> paulej72: I think you can only directly access it if you've got it's IP. Try nslookup ipv4.oxygen.li694-22
[21:49:02] <TheMightyBuzzard> remove doesn't kill configs like purge does
[21:48:59] <paulej72> xlefay: i can’t get to oxygen. do you have direct access to that server?
[21:48:37] <xlefay> I need to do a better dpkg query to figure this one out
[21:48:28] <xlefay> I might be wrong though, it's confusing since icinga's daemon + init script still exists as well as /etc/icinga
[21:47:20] <TheMightyBuzzard> sounds like it got apt-get remove'd
[21:46:55] <xlefay> ironically, icinga files still exists.. ugh oddness
[21:46:13] <xlefay> dpkg doesn't show it installed at all lol
[21:45:43] <juggs> maybe gluster dropped it when it had a brain fart :)
[21:45:42] <xlefay> apt
[21:45:35] <paulej72> xlefay: how was icigna installed
[21:45:16] <xlefay> don't think so, I think someone just accidently removed stuff. Let me check though.
[21:44:42] <paulej72> khyber?
[21:44:02] <xlefay> Someone broke our icinga lol
[21:42:52] <xlefay> Stuff's missing it seems
[21:42:46] <xlefay> docs images
[21:42:46] <xlefay> ls /usr/share/icinga/htdocs/
[21:42:37] <xlefay> ls: cannot access /usr/lib/cgi-bin/icinga: No such file or directory
[21:42:34] <xlefay> paulej72: I think I know why Icinga isn't working...
[21:41:31] <xlefay> hmm this is odd
[21:39:59] <xlefay> NC & I already discussed containers (a long time ago) - but those where just plain LXC containers for development environments.
[21:39:53] <paulej72> ok I have autossh started properly
[21:39:23] <xlefay> ^ paulej72 - I think it might be worth considering for our purposes. It would keep things nice and decoupled in general & a lot easier to get up, less dependency hell, etc.. Just plug 'n (mostly) play containers.
[21:34:27] <xlefay> Instead of the VM for instance, one could simply "pull" the docker image, and has a copy (well, minus the LDAP, and gluster) of the production environment (minus actual user data, of course) - wouldn't that be much easier than a VM?
[21:32:29] <xlefay> I can't help but think how much easier our infra would be if we were using containers like that. But putting that aside for now. Consider the development stuff.
[21:32:01] <xlefay> Yeah, they are. Docker's added some good stuff on top of that though. Like portfwarding built in, being able to deploy the same image on multiple places, etc..
[21:31:31] <paulej72> they are lik chroot jails
[21:30:42] <xlefay> paulej72: take a look at: https://www.docker.com - I think you'll see what I'm aiming at
[21:30:10] <xlefay> Do you know the concept of containers?
[21:30:01] <xlefay> I think, docker would be ideal for us. Even more so, we could even put up a few docker images of our production environment minus actual user data.
[21:29:52] <paulej72> no niethe
[21:29:35] <xlefay> paulej72: also, I was thinking the other day... have you ever worked with containers? Perhaps even docker?
[21:29:21] <xlefay> Too much stuff left unfixed.
[21:29:11] <xlefay> Yeah, I know there's something and I was going to make a startup script for it.. but.. ugh
[21:28:18] <TheMightyBuzzard> i remember SOMETHING had to be manually started every time but damned if i remember what
[21:27:39] <xlefay> I think Icinga might just not be started
[21:27:27] <xlefay> Oh.. then autossh might have been broken too, I'm a bit out of touch with how NC & I did all that. I've scribbbled it onto my notepad
[21:26:41] <paulej72> xlefay: i rebooted boron
[21:15:18] <xlefay> Wow, that's odd. I get the same error. I'll have to look into it. (re: icinga)
[19:19:40] -!- Bytram|away has quit [Quit: Leaving]
[19:19:03] <Bytram|away> okay, I gtg; Good Luck!!!!!
[19:17:48] <Bytram|away> and I'm not able to get onto icinga ( https://sentinel.soylentnews.org ) to see anything; tried twice to login; not allowed. Now I get a 403 when I try to bring up the page.
[19:16:11] <Bytram|away> looks like gluster is losing its luster. :
[19:15:58] <paulej72> NCommander: ping?
[19:15:48] <Bytram|away> ugh. blegh!
[19:15:35] <paulej72> gluster is not loading the slash file system
[19:10:22] <Bytram|away> what problem is that?
[19:10:10] <paulej72> fuck same problem as before
[19:04:25] <Bytram|away> :(
[19:04:16] <Bytram|away> main page "Australia Bans Reporting of Multi-Nation Corruption Case" story says 1 comment, story page ( http://soylentnews.org ) says there are 10 comments.
[19:03:26] <Bytram|away> paulej72: looks like the main page comments are not getting updated again... main page, driverless cars says 11 comments, story page ( http://soylentnews.org ) says 18;
[17:19:05] -!- Tachyon [Tachyon!Tachyon@hollhb.kolej.mff.cuni.cz] has joined #dev
[17:15:27] <mrcoolbp> paulej72: we're basically ready for a soft launch on the soylent store
[17:04:33] <TheMightyBuzzard> ya, i'll prolly kick out a few more lines after some tv and lunch.
[17:04:05] <TheMightyBuzzard> last time i did billing work there was no such thing as PCI
[17:03:55] <paulej72> np about slacking off for the rest of the day. I am mostly doing real work and will not code for another 4 hours.
[17:03:04] <TheMightyBuzzard> oh, handy
[17:02:57] <paulej72> TheMightyBuzzard: pp will do our credit cards. We do not want to do that ourselves PCI compilance is a real bitch.
[17:02:55] <TheMightyBuzzard> i'm dicking off for a while now. ping if you need anything.
[17:01:41] <TheMightyBuzzard> truth. but it's good practice for when we start accepting credit cards. should be easy with as much as we've already done.
[17:00:56] <paulej72> TheMightyBuzzard: i never would have expected that there was this much stuff missing from the subscription code, but this code was rally old.
[17:00:55] <TheMightyBuzzard> nod nod
[16:59:22] <paulej72> thinking about puttin it into a new text field called raw_transaction just like your new table.
[16:58:20] <paulej72> not for long :)
[16:58:03] <TheMightyBuzzard> you could put it in data i guess but it's blob rather than text type
[16:56:44] <TheMightyBuzzard> paulej72, nod nod
[16:56:30] <TheMightyBuzzard> that's a typo
[16:56:28] <paulej72> I am plannng on duming the raw data into the payments table as well, jsut so we see it.
[16:56:25] <TheMightyBuzzard> oh, remove the % in front of $logthis
[16:55:53] <TheMightyBuzzard> go for it
[16:55:42] <paulej72> return $dumped;}
[16:55:42] <paulej72> $dumped =~ s/^\s+//mg; $dumped =~ s/^.VAR1 = {\n//g; $dumped =~ s/};\n//g;
[16:55:41] <paulej72> raw_transaction => sub { my $dumped = Dumper(%$logthis);
[16:55:39] <paulej72> TheMightyBuzzard: I want to turn this into a real sub as I will use it as well
[16:55:00] <TheMightyBuzzard> see above for implementing logging
[16:54:35] <TheMightyBuzzard> excellent
[16:54:21] <paulej72> TheMightyBuzzard: yes transaction is a hash split on =
[16:53:11] -!- Bytram|afk has quit [Ping timeout: 244 seconds]
[16:50:02] -!- mode/#dev [+v Bytram|away] by SkyNet
[16:50:02] -!- Bytram|away [Bytram|away!~pc@Soylent/Staff/Developer/martyb] has joined #dev
[16:41:49] -!- Tachyon has quit [Ping timeout: 244 seconds]
[16:27:48] -!- Tachyon [Tachyon!~Tachyon@dgc-obm-01-40.cust.vodafone.cz] has joined #dev
[16:27:28] Bytram is now known as Bytram|afk
[16:23:26] <TheMightyBuzzard> barring the inevitible typo
[16:23:12] <TheMightyBuzzard> should work fine for pdt or ipn
[16:22:52] <TheMightyBuzzard> paulej72, for you (schema for paypal database logging and subs necessary to log with): http://pastebin.com and http://pastebin.com
[16:18:34] -!- Tachyon has quit [Ping timeout: 244 seconds]
[15:46:28] <TheMightyBuzzard> i ask because i'm writing the logging sub to accept a hash of a raw transaction split on equal signs
[15:43:40] <TheMightyBuzzard> paulej72, you had some bit of script that split the raw paypal transaction info into a hash by using split, yes?
[15:28:12] Bytram|away is now known as Bytram
[15:27:30] -!- mode/#dev [+v Bytram|away] by SkyNet
[15:27:30] -!- Bytram|away [Bytram|away!~pc@Soylent/Staff/Developer/martyb] has joined #dev
[13:58:23] -!- Tachyon [Tachyon!Tachyon@acznspx081.ms.mff.cuni.cz] has joined #dev
[13:01:43] -!- Tachyon has quit [Ping timeout: 244 seconds]
[13:00:10] <TheMightyBuzzard> done
[12:59:33] <paulej72> use this url to trigger a pdt dump: http://dev.soylentnews.org
[12:57:55] <TheMightyBuzzard> paulej72, still doing data dumper of the txns? i could use one over here to get an idea of field sizes.
[11:06:16] juggs|afk is now known as juggs
[01:26:58] juggs is now known as juggs|afk