mozilla :: #releng

15 Mar 2017
00:47ewongjlund ping
00:49ewongany RelEng's around? I'd like to push https://bugzilla.mozilla.org/attachment.cgi?id=8845775 to default of buildbot-config.. is it safe to do so now?
01:02ewongI'm guessing it shouldn't be a problem since it's only default. :P I don't foresee any bustages.
01:04travis-cibuild-buildbot-configs#2485 (master - 0063c34 : Edmund Wong): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/211185209)
04:09jlund|awayewong|away: yep. No problem
04:29ewong|awayjlund|away: thanks!
13:54spacurar|builddutykmoir: Hi, this would be an example of parameter.yml -> https://public-artifacts.taskcluster.net/KsXhsnIXQkKleP6gVCJRgA/0/public/parameters.yml taken from -> https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=gecko%20decision%20task&selectedJob=83990923
14:13Callekdustin: ping -- https://treeherder.mozilla.org/logviewer.html#?job_id=83974842&repo=mozilla-central&lineNumber=603 Tomcat raised it to me
14:14rbarnesrail: not seeing SHA256SUMMARY in https://ftp.mozilla.org/pub/firefox/releases/53.0b2/
14:14Callekdustin: a key_error relating to task morphing stuff
14:16Callekdustin: broke android nightly graph...
14:29dustinCallek: plz open a bug
14:31Callekdustin: Tomcat|sheriffduty ^
14:31Callek:-)
14:32Callekdustin: current concern is that it seems to be blocking android nightlies, I'm hesitant to have us retrigger until we know why that broke
14:32dustinyeah, I'll look
14:32dustinI suspect it's simple (added a taskId to one thing and not to the other)
14:33Tomcat|sheriffdutydustin: filed bug 1347569
14:33dustinthanks!
14:34Tomcat|sheriffdutyCallek: it hit desktop nightlys too
14:34Tomcat|sheriffdutynot just the task for android nightlys
14:34CallekTomcat|sheriffduty: ok, thanks
14:34Tomcat|sheriffdutymentioned in the bug that we need to retrigger them when this is fixed :)
14:34CallekTomcat|sheriffduty: tobe clear, did any of the jobs relating to nightlies end up starting (I don't think so, but you'd see them for sure)
14:35Tomcat|sheriffdutyCallek: only the build bot ones
14:35Tomcat|sheriffdutyso no linux nightly so far
14:35Tomcat|sheriffdutybut windows osx
14:40railrbarnes: hmm, let me check it
14:42railrbarnes: I see it generated in http://mozilla-release-logs.s3.amazonaws.com/mozilla-beta/firefox-53.0b2/build1/firefox_mozilla-beta_checksums_builder-all-HeMuWnaCQga_-tAtM-zwHg-0, maybe it's not uploaded...
14:43railyea
14:44railrbarnes: doh, we need to add a line here: https://hg.mozilla.org/releases/mozilla-beta/file/f4c59968a6ea/testing/mozharness/scripts/release/generate-checksums.py#l294 and call _get_summary_filename()
14:45rbarnesrail: you want to write the patch or shall i?
14:45railup to you
14:46railI'll take it
14:47railrbarnes: thanks for verifying it
14:48rbarnesnp
14:48rbarnesthanks rail
14:49railno no, thank you :)
14:58ddurstI don't know who the right people are to ask about this, but I want to de-stupidify myself, so I'll start with rail
14:58railhahaha
14:58railyou can also ask to the void :D
14:58ddurstrail: when we're talking about deprecating sha1, wrt updates, this would seem to be only related to the move to sha384 and aus5
14:58* ddurst the void has been unforgiving in the past
14:59railhehe
14:59ddurstis that correct?
14:59railddurst: I think so
15:00railand aus5 is our current update server
15:00ddurstBecause aus3 only addresses to 35.0, aus4 36-41, and we have watersheds via aus5 -- anyone who needs to get to sha384 build has to get to aus5 first
15:00ddurstright
15:00railyeah
15:00ddurstSo when we're talking about any cert switching, that would only apply to aus5.
15:00ddurstNever to aus4/3
15:00railso the only remaining thing is the MAR signatures
15:01* ddurst go on...
15:02ddurstmeaning the cert that signs the MAR files?
15:02railyeah
15:02* rail finds the bug
15:02ddurstand that that would need to happen for the move to sha2 to really take effect.
15:02ddurstyeah
15:02railhttps://bugzilla.mozilla.org/show_bug.cgi?id=1324498
15:03ddurstright, we want to make that shift at the same time as the lzma change
15:03railah
15:04railyeah, so we have less watersheds
15:04ddurstbut all of that stuff happens via aus5 anyway, so there's no external cert switching involved.
15:04ddurstis that right?
15:06bhearsumddurst: that should be the case - the aus5 ssl cert doesn't impact watersheds in any way
15:06railyup
15:06ddurstok.
15:06railaus5 is ready
15:06ddurstThat's what I thought, I just didn't know if there was anything I was overlooking.
15:06catleewe don't do cert pinning for aus*, right?
15:07bhearsumcatlee: we do!
15:07rstrongddurst: if you would like me to I can try to write it up for you. Basically, anyone that is pointed at aus3 or aus4 is using a client that won't have sha1 deprecated. So, the only factor it when the certs expire for aus3 and aus4.
15:07bhearsumhttps://wiki.mozilla.org/Balrog/Client_Domains
15:07ddurstrstrong: I think I've got it.
15:07rstrongwe don't do cert pinning for app update with aus5
15:08rstrongiirc we do cert pinning with aus3 and aus4 with old Firefox version
15:09ddurstI heard mention of cert switching and just wanted to be certain that it wasn't applicable in this scenario. So I'm all good.
15:09* ddurst early for you, rstrong
15:09bhearsumrstrong: that's right, - we only pin for GMP and Thunderbird on aus5
15:09rstrongddurst: wanted to make sure dthayer was good to go for the demo but it was cancelled
15:10ddurstd'oh
15:31catleeTomcat|sheriffduty: approximately how many merges to m-c do we do a day?
15:33Sylvestreis it possible to remove files from tooltool ?
15:33SylvestreCallek, ^^
15:33Callekcatlee: average 2-3
15:33Tomcat|sheriffdutycatlee: 2 or 3
15:33Tomcat|sheriffdutybut we aim for 2
15:33CallekSylvestre: not ultra easy, but possible why?
15:33CallekSylvestre: garbas would be who I point you at (since I'm not 100% sure the way today, but I suspect its not hard)
15:34SylvestreI made a mistake, I don't see the point of keeping useless files
15:34Sylvestrebut it is just to keep a clean db
15:34Sylvestrenot like I uploaded my ssh private key :)
15:34CallekSylvestre: ahhh in the case of 'not really needed' might as well keep it given the pain associated with dropping it (imo)
15:35CallekSylvestre: the use case from dropping it would be a secret of some sort, or a "non-redist" thing being uploaded to public
15:35Sylvestreok, so, let's keep my mistake forever then :)
15:35Callek(yes, it does have s3 storage costs, but those are small compared to human costs)
15:36Callektheres been desires to eventually track what gets accessed there, and mark things as potentially expiring after X days/months/whatever of use.... (or no-use)
15:36Callek(as in, if something is only accessed 5 times, and then never, it is a great candidate, if something is accessed 1000 times a month until last week, for over a year, we probably don't want to kill it that fast)
15:36rstrongbhearsum: is there a list of watersheds available in any form?
15:38bhearsumrstrong: not really...but we do nightly dumps of the production balrog rules - you should fire up a local instance if you want to see them, or i could put together a one-off list
15:38bhearsumi think catlee has been thinking about how and where we could publish a maintained list of watersheds
15:38rstrongcool and thanks
15:39bhearsumnp, cloning https://github.com/mozilla/balrog and running "docker-compose up" should be all you need to do to see them locally
16:00rbarnesrail: so is it possible to download that SHA256SUMMRY file anywhere?
16:05catleerstrong: what kind of data would be helpful for you to have in that list?
16:06catleerbarnes: it's in the log
16:07catleehttps://irccloud.mozilla.com/pastebin/RHl7Po8s/
16:07catlee...
16:07rstrongcatlee: ddurst was asking where there was such a list. I would think the version, the reason for the watershed would be enough, and a bug #.
16:08catleerstrong: I started some ideas here: https://gist.github.com/catlee/f8363d6fc64e13b92bdefd1b6f612ebd as a schema, and https://gist.github.com/catlee/d93e314b045947aa1d1719a9e1e402e7 as an example
16:08catleeI think we need buildid to support nightly well
16:09rstronglooks good
17:35rbarnesrail: could you put your r+ in mozreview? won't let me autoland as it is :/
17:36railI though I did...
17:36railmaybe it's blocked by the "issues"
17:36railI marked them as "fixed"
17:36railtry now?
17:54rbarnesrail: thanks, should be autolanding
17:54railnp
18:01wsmwkrail: question ... if we build 53.0b1 and not want linux users to get it I know we can set barlog to offer updates. Would it be too much to ask to also have the linux builds/directories (and linux packaged sources) deleted? if too much, what are alternatives?
18:02bhearsumwsmwk: why build it at all if you're going to delete them?
18:02wsmwkbhearsum: true. I shouldn't have made it an XY problem. Can we avoid building linux altogether?
18:02railI wouldn't delete them, this may be problematic for the future builds
18:03wsmwkrail: problematic how?
18:03wsmwkor, deleting which parts causes problem?
18:04railI don't think we support partials-per-platform, so partials in the future release will fail trying to get linux complete MARs
18:05bhearsummight have issues disabling then re-enabling a platform, i'm not sure we ever do that...
18:05wsmwkOK, other recommendations?
18:06bhearsumbuild linux as usual, don't ship it through balrog, and don't delete the binaries :). that's your safest bet
18:07rail++
18:07bhearsumsorry, i don't mean to be snarky
18:07railor fix linux :)
18:08wsmwkbhearsum: no worries
18:08* wsmwk we sometimes have problems :)
18:09wsmwkrail: but bz2 sources don't have MARs right, so we could delete those? :)
18:10railyou mean xz! :)
18:11* wsmwk hm, i forgot about all the l10n
18:12wsmwksorry if i'm speaking confusion. i'm talking about the direct download stuff
18:13* wsmwk but can see things might get complex
18:49rstrongbhearsum: were the rules always set up to update systems without the modified url? I ask because it would explain why we saw the results we have been seeing.
18:50bhearsumrstrong: yeah - i'm just writing a comment about that in https://bugzilla.mozilla.org/show_bug.cgi?id=1339973 now
18:51rstrongwow... we did a ton of investigation as to what were seeing and this would explain it.
18:52bhearsumi'm sorry about that :(
18:52rstrongat least we have a viable explanation now
19:06ddurstbhearsum: I think you miswrote in your comment? Basically that if there is no indication of websense (neither "(websense" nor "(nowebsense)") then there is no block.
19:06lizzardoh, huh
19:07ddurstso it's not that we can't do #3, it's that we *do* do #3
19:07ddurstright?
19:07* ddurst not trying to nitpick, just don't want to be confused anymore
19:08rstronglizzard: this could explain to a large extent the original low uptake as well as when it was included in 47.0.2, etc.
19:08lizzardwe were originally blocking websense status unknown users from updating
19:08lizzardnow, we shouldnt be
19:09lizzardim missing what would explain the original low uptake. we figured there are many users who dont open firefox for very long, or very often (so they never get offered the add-on)
19:09lizzardbut now they will get offered updates , if they didnt get the addon, i think
19:09rstrongbhearsum: I took your "yeah" above to my question as confirmation that this has always been the case
19:10bhearsumlizzard: no, we were never blocking websense status unknown - we can't do that
19:10lizzard!
19:10bhearsumother than by blocking updates for every single firefox user
19:11lizzardi think we were blocking if they didnt have the addon? we did have a lot of users stuck there for a while
19:11rstrongbhearsum: you could offer updates only to Windows users that had (nowebsense)?
19:11bhearsumrstrong: if we're willing to accept the fact that non-websense users without the system addon get left behind
19:11lizzardI feel sure we were not letting the unknown status users update, for some time. we kept discussing when to let them .
19:12lizzardI dont want them to be left behind (anymore)
19:12lizzardIts just kind of gone on too long
19:12rstrongthat is what I thought we were doing. In time they would get the system add-on, etc.
19:13rstrongas is, even if they got the system add-on a decent amount of those clients wouldn't have restarted before the update check which is required for system add-ons
19:13rstrongoi
19:14bhearsumddurst: didn't mean to ignore you - i'm not 100% sure what you mean though
19:14rstrongthis explains so much going all the way back to the very first push of the websense add-on
19:14bhearsumwell, i'm glad we have that explained at least...
19:16rstrongseriously and for the fture, there is little point in deploying an add-on like this if we aren't going to hold clients back if they don't have the modified url.
19:18bhearsumagreed, unless we've deployed it so far in advance that we can be assured of extremely high uptake
19:19lizzardi am still somewhat lost
19:19lizzardwhat is it that we know thats new, that explains things?
19:19bhearsumshall we hop on vidyo? this is messy
19:20lizzardsure
19:20rstrongbhearsum: not crashing is as I see it more important. In this case we also deployed a new add-on that can tell us if the system's websense has been upgraded to a version that won't crash so we can update them. That is also what a bunch of people have been working on and this negates that.
19:21lizzardbhearsum: what room?
19:21bhearsummine, i guess
19:24lizzardsure, im there
19:24bhearsumrstrong: do you want to join?
19:24bhearsumno worries if not
19:24rstrongsure
19:31ddurstbhearsum: I understand your comment now. Nevermind. You're saying that we can't have a rule that does that. (which is why those users update)
19:31ddurstgot it
19:45bhearsumddurst: okay, wheh!
19:58ddurstI assume that we would have to have a rule to blocks 47.0.2 users, and a rule that overrides that if the update URL includes (nowbesense). But I will leave all this to the experts.
19:58* ddurst shuts up
20:51bhearsumrstrong, lizzard: so, i think we need to whitelist all known good websense versions - do we have a list of those?
20:52rstrongI saw a spreadsheet I think... maybe lizzard knows where that is
20:53bhearsumwe have https://docs.google.com/spreadsheets/d/18n8NEimajNpnGRHdQsrrnyJ-Ufgvmpz-AXCUpVaFRMk/edit#gid=0
20:54bhearsumbut that's only a few versions
20:54bhearsummaybe this is a question for jimm
20:59bhearsumi've gotta run in a minute, but i think i have enough to get initial rules on cdntest tomorrow morning - we can start with (nowebsense) passthroughs, and whitelist the specific versions later
21:00lizzardquestion for jimm or marco. i think they are shipping a new version now for the 52 release
21:00bhearsumlizzard: ok, i care about ALL websense versions though
21:01bhearsumnot just the latest round
21:01bhearsumotherwise people on older, known-good websense versions will get stuck on their current release
21:02bhearsumi'll poke about it tomorrow - we don't need to block on that
21:03bhearsumheading out now - i'll send mail tomorrow
22:20whimbooCallek: hi
22:20whimbooregarding the beta canddiate builds via taskcluster
22:21whimboois it this build we are using for linux 32? https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&count=50&tochange=a5ccc32310c9a424ccb4f5106f5155ca20dd360a&filter-tier=1&selectedJob=83504373
22:26* aki looks
22:27akiwhimboo: are you asking about the type of build or the actual 53.0b2 build?
22:27whimboothe actual build as built in CI
22:28whimbooi think it should be that one
22:28whimboothe build id at least matches
22:28* aki looks for requested changeset
22:29whimbooaki: looks like on mozilla-beta there is no tc(B) job but always tc(N)
22:29akiyeah, looks right
22:29akiyup
22:29akiin taskcluster we're using the nightly graph for beta promotable builds
22:29whimbook, the same would then also apply for release
22:29akiyes
22:30akionce 53 goes to release
22:30whimbook, thanks tht helps
22:30akinp
22:30whimbooso i have to look for either this build or if not present a buildbot one
22:30akiaiui yes
16 Mar 2017
No messages
   
Last message: 159 days and 11 hours ago