mozilla :: #moc

14 Jul 2017
01:00nagios-scl3New Sysadmin OnDuty is ashlee
01:27ryancHi
01:59nliashlee: hey, question - will the ongoing maintainence eventboard updates affect lobbyconnect?
02:01nliTPE lobbyconnect search box not showing contacts
02:04ashleeHim nli :) I'm not sure let me check
02:04ashleeHi*
02:05nliashlee: thanks!
02:05* nli tried troubleshoting tip on mana, but failed to bring it back. Thought it could be the server side issue. :)
02:10ashleeryanc: are you familiar with lobbyconnect?
02:10ryancHi
02:10ryancWhat is that?
02:11ryancSounds like an EUS issue
02:11ashleeyep
02:16ashleenli: I've reached out to pdang for some help.
02:17pdangnli: hi, i can look into this tomorrow if there's no urgency
02:19nlipdang: Sure thing. we always have a workaround. :)
02:20pdangnli: great!
02:20pdangnli: talk to you soon
02:24pdangashlee: thanks for the heads up
02:26ashleeNp pdang thanks for the quick response!
03:19Tony_6 nli: EventBoard refresh should not affect Lobby Connect
03:19Tony_6Are you still seeing the issue?
03:20nliTony_6: Yes, I did.
03:21nlichecked it again after I saw khanhs email
03:29khanhnli: Lobbyconnect isn't tied to calendar, so a calendar sync will not affect it.
03:30nlikhanh: roger Sir. Thanks. :)
09:00nagios-scl3New Sysadmin OnDuty is Usul
10:21jxiaall devices in Beijing office (PEK2 ) will be power off in 10 minutes due to power maintenance , see Bug 1380582, thanks.
10:21firebotBug https://bugzil.la/1380582 is not accessible
10:24Usuljxia: ta
12:18jhfordUsul: hey, is there something wrong with the networking in the Berlin office? (ber1)
12:19jhfordi'm having *TONS* of problems today
12:19jhfordi have interent, but a lot of services are hard to connect to
12:19jhfordapi.heroku.com earlier, *.docker.com and now debian mirrors
12:19jhfordnot everything is busted, but it's definitely not working for me as it should. I'm wired into the network, so it's not a wifi issue
12:22Usuljhford: roger
12:22Usulcan you file a bug ?
12:22Usulin infra:netops:office
12:23jhfordUsul: i really don't have time... i'm fighting to get critical taskcluster infrastructure back in service
12:23Usulroger
12:23jhfordwhich I cannot do because I am having tons of trouble talking to heroku and docker
12:23Usulwill do will hunt netops for you
12:23jhfordthanks :)
12:23jhfordthis is all intermittent, sometimes it's working and others not
12:24UsulBER1 , BER2 ?
12:24Usuljhford: ^^^
12:24jhford1
12:25jhfordinet 10.249.24.180
12:25jhford ether f4:f2:6d:05:9e:3a
12:31jhfordnow from yarn: warning There appears to be trouble with your network connection. Retrying...
12:31Usulbug filed / net.ops alerted
12:31johnbUsul: can you ping Hofznet?
12:32Usuljhford: anybody else complaining about the network in BER1 ?
12:32johnblooks like they might have a network loop
12:32UsulI will
12:32jhfordi see a lot of package loss
12:32jhford90.9% in mtr
12:32* TheOne is having the same issues
12:32TheOnelots of packet loss
12:32johnbtraffic to *.docker.com (for example) gets TTL exceeded
12:33johnbI loop between 13. 178-19-223-37.static.hofnetz.it and 14. 178-19-223-36.static.hofnetz.it
12:35jhfordyep!
12:36jhfordi'm getting 63% package loss on 200 packages sent
12:36TheOneI even get: "traceroute: unknown host google.com"
12:36jhfordi am having trouble reaching anything
12:41martyna#moc
12:41martynaBerlin has trouble with internet connection
12:41jhfordyep, it's being looked inot
12:41Usulmartyna: coms have been sent out
12:44freddybpreviously, einar suggested disabling IPv6, but I think the folks who did that seem still affected?
12:45Usulso I just got the isp and they told me that they've just removed a faulty edge router
12:45Usulthing should get better
12:45freddybah, thanks!
12:45Usulmartyna: ^^ freddyb ^^ jhford ^^ TheOne ^^^
12:45einrownfreddyb: hi Freddy
12:46Usulcan someone confirm network is better in BER1 ?
12:47jhfordi can confirm it is not
12:47jhfordi kept seeing my packet loss to api.heroku.com increase
12:47jhfordand now i cannot even resolve that host
12:47Usuljohnb: ^^^^
12:48einrownThe issue with the network I spoke with John Bircher yesterday
12:48einrownHe is also aware of the issue
12:49jhfordok, now I'm abel to resolve api.heroku.com and MTR doesn't suggest there's a loop
12:49TheOnestill seeing issues as well
12:49johnbapi.heroku.com is not reachable from outside of the office
12:49* Usul checks status.heroku.com
12:49johnbthey may be having their own issue too
12:49Usulgreen
12:49jhfordperfect :)
12:49johnbnothing on status yet
12:49Usulhttps://status.heroku.com/
12:50jhforddocker.com was also a site that was broken for me
12:50Usuljhford: are these v6 enabled can you force v4 ?
12:50jhfordUsul: v6 to google seems to work, v4 seems to be busted
12:50einrownjhford: strange yesterday was opposite
12:50jhfordactually, i get some packet loss to google on 2a04:a40::3:21:1
12:51jhfordjohnb: is the new office using the same ISP -- just curious
12:51einrownIpv6 should be disabled on your machine
12:51jhfordipv6 works fine
12:53TheOneRequest timeout for icmp_seq 116: 36 bytes from 178-19-223-36.static.hofnetz.it (178.19.223.36): Time to live exceeded
12:55jhfordif IPv6 in the office/isp is broken, we shouldn't be giving out ipv6 addresses
12:57Usulissue in hozt they can't even answer the phone right now
12:58UsulHotznet is still working on the issue.
13:01jhfordthe loop is back
13:01johnbI will have to look at who we are using in the new office. Hofnetz seems to have a lock on Berlin and is at least one of the two circuits.
13:04jhfordnot a rush, but is it likely for this to be resolved in the next hour or so?
13:06Usuljhford: the guy one the phone said sorry he couldn't tell me anything as of yet
13:06jhfordalright
13:06Usulif things aren't better at 25, I'll ring them again to get an update does that work for you jhford ?
13:06johnbboth offices have the issue so running across the street isn't an option
13:07jhfordwell, I've managed to get what I was working on completed, given it's 3pm on a friday I might head home
13:07jhfordit took running commands 10 times, but it did get through eventualy
13:28Usulmartyna: is internet working better ? cc jhford TheOne freddyb
13:29johnblooks better from what I can tell
13:29johnbat least to docker.com
13:29Usulyeah you stuff is not recovering.
13:29Usulta
13:30johnb?
13:30martynamartyna: for me it is sufficient
13:30Usulyou canary stuff alerted a but later than you
13:30Usuland didn't recover yet so some issue still
13:30martynabut I know for some people it is still not good enough to do their work
13:30johnbwell my ssh access stopped for a bit too
13:31UsulI called and endedup on voicemail
13:31Usulupdate sent
13:31johnbok
13:35TheOneUsul: still getting timeouts for addons.mozilla.org
13:35TheOne36 bytes from 178-19-223-36.static.hofnetz.it (178.19.223.36): Time to live exceeded
13:37UsulYeah Hotzner or what evr the proper name is, is having unresolved issues
13:41einrownUsul: is this then a ISP issue, is there anything I can do from Berlin
13:41Usulask johnb but I don'yt think so we only have one ISP.
13:41Usulyou could call them to get status update, but I'm managing right now
13:41Usul:p
13:41johnbonly one for each of those two offices
13:41johnbsame one too
13:42einrownjohnb: is there anything I can do from Berlin?
13:42johnb3yr old bandaid
13:42johnbnot really
13:42einrownjohnb: or ISP issue?
13:42johnbISP issue
13:47johnbgrr... Hofnetz is just making a mess of alerting for Berlin
13:48Usulindeed :)
13:48UsulI'll call again soon for an update, hoping not to get the voicemail
13:48martynamoc: again got worse :/
13:48martynaI can not acces my gmail
13:49Usulmartyna: yes - I'm seeing more alerts
13:49UsulI'll call the ISP and send an update in 10 minutes
14:45einrownUsul: please CC in the isp update
14:47Usulsure
14:47Usulwhat's your bugmail ?
14:47Usuleinrown: ^^^
14:47Usulhttps://bugzilla.mozilla.org/show_bug.cgi?id=1380988
14:47firebotBug 1380988 is not accessible
14:47Usulyou should be able to access it einrown
14:47Usulare you subscribed to the status pages updates ?
14:51einrownUsul: I don't think so, do I set that in the bugzilla ticket directly
14:51einrownUsul: ^^^
14:51Usulyes you add yourself on cc
14:52einrownUsul: okay thanks for the info and staying on top of the situation
14:52Usulalso on sso.mozilla.com/statuspage I believe you can subscribe to stuff you want to be made aware of
14:59einrownUsul: I added myself to the CC list for 0988 tick
14:59Usulkk
14:59Usuldo you get the update emails ?
15:01einrownUsul: yes I got em :)
15:01Usuleinrown: cool
17:19wlachashish: yup, looks good! I'm guessing I could have done this myself, had I known more about chanserv incantation. thanks so much
17:19ashishyw :)
17:41fauwehthanks ashish!
17:58catleeI need help resetting my MFA for bugzilla
20:00gbrowncan anyone here help me with bug 1380666? I need an update to sudoers list on brasstacks
20:00firebothttps://bugzil.la/1380666 NEW, server-ops-webops@mozilla-org.bugs Update sudoers list for brasstacks1.dmz.scl3.mozilla.com
20:08gbrownfauweh: ^^ ?
20:08fauwehgbrown: lemme look
20:08gbrownthanks
20:18fauwehgbrown: change completed, should be updated in a few minutes
20:18gbrownfauweh: thanks much -- that's a great help!
20:18fauwehanytime, glad to help
15 Jul 2017
No messages
   
Last message: 68 days and 16 hours ago