mozilla :: #moc

8 Sep 2017
01:00nagios-scl3New Sysadmin OnDuty is ashlee
01:00fauweheyyy
01:01ashleeheyy
01:26ryancHey hey
06:04ewongit's fatt Albert!
07:27justdaveashlee: any issues reported with Zayo in SJC2?
07:28justdaveI think that's the trunk line between SJC2 and MTV2 actually
07:28justdavegot a notification from Zayo about an hour ago that the link was down
07:29ashleejustdave: I haven't had any alerts
07:30ashleeIs the link still down?
07:41justdavenope, looks up as far as I can tell
07:41justdavemaybe it wasn't down long enough for nagios to notice
07:44ashleejustdave: rgr, good news
09:00nagios-scl3New Sysadmin OnDuty is Usul
13:55mhoyehey, Usul
13:55mhoyeblog.mozilla.org is redirecting to a placeholder page.
13:56mhoyeWe should probably wake somebody up for that.
13:58mhoyeAny idea who?
14:00Usulyep
14:00Usullet me look at the maintainance schedule
14:01Usulmhoye: did you file a bug already ? if not I'll file
14:02yalam96i have no idea why thats pointing to our site but if you need me to look at settings i can
14:02mhoyenot yet, coming up
14:02jlinhi moc blog.mozilla.org is redirecting
14:02jlinah I see mhoye already alerted
14:02Usuljlin on it https://bugzilla.mozilla.org/show_bug.cgi?id=1398159
14:02havijlin - ok, its all of blog.moz?
14:02firebotBug 1398159 is not accessible
14:02jlinthanks!
14:03jlinhavi yes, you can check that bug usul just linked
14:04* mhoye dupes his own bug
14:04haviok, yeah, i have blog post scheduled to go in 1. does anyone know how long weve been down?
14:05mhoyehavi: one what
14:05Usulhavi: nope
14:05havisorry mhoye - 1 hr
14:05Usulworking on people to look at it
14:05haviof course, thanks all.
14:11haviUsul: are you trying to identify owners?
14:12Usulnope webops is looking at it I'm doing coms
14:12haviok, cool. thx
14:12jlinhmm as an additional data point, I am now able to see hacks in a private browser window
14:13unixfairyhavi: we have he right teams involved to look at the issue and are drafting comms
14:13haviunixfairy: Thanks! gotcha
14:16havijlin: yeah, me too, thats very interesting
14:16unixfairyhavi and mhoye works for you?
14:17haviwell, a non-logged in view of existing hacks.moz and blog.moz
14:17jcristauthe 302 was cacheable as far as i could tell. i didn't check for how long.
14:19havihey, i was just able to access the wp-admin console in hacks
14:20mhoyehavi: All the fireworks are happening over in #webops, but it looks like we've resolved immediate problem and ericz and tad are on the case with the followup.
14:21ericzYep, thanks mhoye
14:21havithx mhoye
14:21tadthanks mhoye
14:22mhoyeMuch gratitude to ericz and tad for turning this stuff around quickly.
14:23haviYes, yes
14:27jlinthanks very much!!
14:34ericzYeah thanks tad and yalam96 for the quick fix
16:20dylanUsul: looks like we might have someone hammering BMO
16:22dylaneither you did something or it went away
16:22dylan:-)
16:24fauwehdylan: I still see a pretty aggressive IP hitting access log on web1
16:27Usuldylan: yes saw the alerts
17:00nagios-scl3New Sysadmin OnDuty is sal
17:18hamisal: hi are you free?
17:19salhami: sup
17:19hamiIn YVR office, we are getting temperature alerts for the IDF. I don't see any nagios alerts here, but am wondering if you guys are aware?
17:20salhami: how are you getting the alerts?
17:20salfrom where?
17:25salhami: nvm I see it, nothing alrting so we're good for now
17:25salhappen to know the temp?
17:28hamiYeah, sorry to clarify it was vendors that alerted us to the temperature.
17:29hamiIt was 79F
17:30fauwehhami: hardware there should be fine at those temps
17:31hamiOk. What's the min/max for this room for future reference?
17:35fauwehmost hardware is perfectly good til it reaches 40C/104F, but that's internal temps so hard to say what room temp would cause it to rise to those values
17:35fauwehbut I wouldn&#39;t worry if room is <90F
17:36fauweh(netops has indicated much of their HW alerts at >60C/140F)
17:38hamifauweh / sal: thanks guys for the info
17:39fauwehhami: for sure, always here to help :)
17:42johnbJust clarification. 140F was Cisco. I think we determined the most sensitive of the network equipment has an intake temp of >110F alarm.
17:43fauwehjohnb: thanks for popping in, I wouldn&#39;t want to see 140F either, 110F is way more sane
17:46johnbIf we exceed 35C/95F intake air temp we are probably working on a limited time window before it hits critical for anything else in the room too.
17:47johnbThe smaller the room the more rapid the change will be.
17:47fauwehyeah, that&#39;s generally been my experience with smaller server rooms too
17:48johnbDo we think YVR1 is going to get warmer or is it handled but just warmer than we like?
17:48fauwehlast email update is that HVAC was restored
17:48fauwehso we should be good johnb
17:49johnbHandled. Thanks. :)
17:50johnbGoing back to my evening pastime in Paris, France. :)
17:51fauwehsounds terrible johnb lol
17:51fauwehenjoy your night :)
17:52johnbIt is horrible... like all they do here is drink wine, eat cheese, laugh... it is horrible... so horrible. But someone has to take it for the team. ;)
17:54fauwehI was in a lot of pain from eating too much in Lyon one time, so I understand haha
18:05jcristauhey we also drink beer
19:36mpoessy#moc any ircadmins online?
19:36salahh I cant help with that mpoessy :(
19:37mpoessysal: anyone one else or does ithave to be the owner of the channel?
19:37salashish: justdave any chance yall could op someone in #foxfooding
19:37mpoessysal++
19:37ashishsure
19:38mpoessyashish++
19:38salahh not registered :/
19:38ashishyep
19:38fauwehtisk tisk
19:39ashishso, who should be the op?
19:39salashish: Asa
19:39ashishokay
19:39salill let him know to register the chan
19:40ashishdone
19:41salashish++
19:47mpoessysal++ ashish++ ty gentlemen
22:43jhelmersAVops has begun the SIP testing to gateway1
22:44jhelmersWe will conclude testing in 16 minutes at 4pm PT
22:45saljhelmers: rgr!
23:44mpressmansal: ping
23:45salmpressman: sup
23:45mpressmanhey hey -so can you tell me how to get a user added to pagerduty?
23:45salmpressman: just a bug in moc:service request
23:45salwe&#39;ll handle it
23:46salwell someone with admin rights :P
23:46mpressmanawesome! thanks!
23:47salno!
23:47salnp!
23:47sallol
23:47mpressman:)
23:47sarnoldlol
23:47* sal needs coffeeeeee
23:59jhelmersAll SIP settings and Vidyo settings have been reverted back to the original settings and the window is closed. happy Friday and have a great weekend everyone.
9 Sep 2017
No messages
   
Last message: 13 days and 48 minutes ago