mozilla :: #releng

14 Mar 2017
01:24travis-cibuild-tools#1686 (master - 26ed8ff : ffxbld): The build passed. (
10:03jyawhat are win8 treeherder machine running ?
10:08aselagea|builddutyjya: win 8 machines are testers, so they run tests for win 8
10:09jyaI mean, what version of win 8 are they running? any particular edition, what service pack and so on?
10:09* aselagea|buildduty checks
10:11aselagea|builddutyjya: Windows 8 Enterprise
10:11jyai guess, I'll have to ask for a loaner and find out what's going on with those
10:12aselagea|builddutyjya: any particular issue you're concerned about?
10:13jya is a permafail on win8 debug
10:13jyalogs show that the built-in h264 decoder never returns a decoded frame
10:13jyai can't reproduce that locally, and it works on all other platforms (including other windows version)
10:13jyaso I have to see what's missing
10:14aselagea|builddutyjya: I see
10:14aselagea|builddutywell, feel free to file a bug for a loaner in this case...we can help with that
10:14jya shows that there were 20 frames being asked to be decoded, but 0 were returned.
10:15jyaso I just log a bug requesting for a loaner?
10:16aselagea|builddutyjya: basically, yes
10:16aselagea|builddutyjya: might help :-)
12:22CallekFlorinMezei: ping -- re: Bug 1347107 -- how bad is the issue for you guys (1-to-10 pain scale)
12:22CallekFlorinMezei: also, where is the relevant code about buildid's for you guys, is it difficult to eliminate the assumption that they are all === across platforms?
12:23CallekI admit its sort of expected at present, and trying to unify them is going to be pretty hard until its all scheduled via Taskcluster, but I don't want to leave you guys blocked if we can help it
12:23FlorinMezeiCallek: it's not blocking us, but it will always be very likely that our update tests will fail
12:24FlorinMezeiI'd say maybe a 3-4 on the pain scale
12:24FlorinMezeiso not really a blocker
12:24Callekyea, I'm less happy about update tests failing for known-things, since they *can* catch one of the worst possible user facing issues... (inability to update properly)
12:24FlorinMezeithe issue with the tests themselves seems to be
12:25CallekFlorinMezei: ahhh ok, let me peek
12:36CallekFlorinMezei: any clue where in code that regex is specified, or gets taken from (that regex that whimboo mentioned)?
12:37whimbooCallek: its in mozdownload
12:40whimbooCallek: the file naming convention for release builds is just different for windows and osx
12:41Callekwhimboo: offhand I don't see anything '.json' in mozdownload and couldn't find any relevant code in the mozmill-ci stuff
12:41Callekwhimboo: the &quot;firefox-<version>.json&quot; exists in both linux and windows/osx for betas right now
12:42whimboozeah. and mozdownload currently expects a Firefox Setup <version>.json
12:43Callekwhimboo: well afaict mozdownload doesn&#39;t expect anything *.json its for the actual &#39;binary&#39;
12:46whimbooits for all files
12:46Callekwhimboo: sure, but what is trying to get the *.json, specifically
12:51Callekcatlee: so far, aiui, if we named it (.json) same as the actual platform binary it&#39;d work, e.g. &quot;Firefox Setup.json&quot; since thats how mozdownload parses stuff
12:51CallekI&#39;m trying to figure out where/how we&#39;re requesting the .json here
12:54jlorenzojcristau: hey! re bug 1347107: you said crashstats uses the buildID to report crashes and it&#39;s preferable to have the same buildID across platforms. Did I get it correctly? cc catlee
12:55jcristaujlorenzo: i wouldn&#39;t say that
12:56jcristaucrash reports include the build id, and sometimes it&#39;s useful to aggregate by build id e.g. to figure out which builds are affected by a particular issue. i don&#39;t know if them being the same across platform makes a difference, though.
12:57jcristaumarco or philipp might have more of an opinion on that
12:58jlorenzookay! if a crash is cross-platform, would it be a pain to manually aggregate the numbers? (I&#39;m just trying to weight up the pros and the cons)
13:00WG9sjlorenzo: yes I said this also. It was prviously a requirement that the buildid match at least accross all desktop platroms. Now it seems, not only do they not match accross timezones but the ones in the non mozilla standard pacific time, don;t even match if timezone corrected. I am not at all sure this was adequately discoussed anywhere before th cahnge was made.
13:00catleeWG9s: sorry, it was never a requirement
13:00WG9sjlorenzo: that said, however, I think it makes more sense to abandon the buildid and just go by the changeset everywhere.
13:00catleeit was just generally the case that buildids matched
13:00catleebut not always
13:01catleeespecially in the case of respun releases or nightlies
13:01WG9scatlee: I see to remeber at one time great pains being taken to ensure they would be identical, but like I said it makes more sense to abndon buildid and use the changeset instead.
13:01WG9ssorry it was in to message
13:01WG9ssin too messages
13:02WG9ssorry i can;t type today, my cat is trying to sit ont he keyboard.
13:02catleeyeah, we did make changes to the scheduling many years ago to make them line up
13:02whimbooCallek: i may have a solution for mozdownload
13:05WG9scatlee: kind of my questions if such effort was made in the past to make them linup why make them not line up at all now with no real input from others about consequences.
13:05travis-cibuild-puppet#1064 (master - ad1eced : Alin Selagea): The build passed. (
13:05WG9si think the real issue is for carsh-reporter wheich needs to be using changeset rather than buildid.
13:06whimbooCallek: how can I best say in a regex &quot; Setup &quot; or &quot;-&quot;
13:06WG9scatlee: but this should have been recognized before the change was mad.
13:07catleeWG9s: we discussed it and decided not to force teh same buildids between TC and BB
13:07whimbooCallek: got it working
13:08WG9scatlee: but we need to get crash-reporter up to date so that we don;t think a windows crash and a liux crash are unrelated becuase they began with a different buildid, which is currently what crash-reporter uses to correlate
13:09Callekwhimboo: what was the fix, may I ask?
13:09jlorenzoI think input from release management will help in knowing whether that&#39;s a common case for crash-reporter
13:11WG9sjlorenzo: but getting the automated correleation to use changeset rather than buildid (shich is now meaninless) will help not having anyuone in releng to have to manually correlate
13:11Callekwhimboo: oooo I get it (?: Setup|-)
13:11Callek(I presume you used the non-matching notation)
13:12whimbooCallek: not really, is that necessary here?
13:12jlorenzoWG9s: I agree on this long term solution :)
13:12whimbooCallek: &quot;( Setup |-)&quot; is not enough?
13:12Callekwhimboo: probably not necessary, but it does make it *slightly* faster and *slightly* less memory
13:13* Callek just has that habit with matching/non-matching notation
13:13* Callek closes mozdownload/mozmill tabs
13:14whimboofor hanldig the fix
13:17travis-cibuild-puppet#1065 (production - f45a739 : Alin Selagea): The build passed. (
13:17WG9sjlorenzo, catlee i will file a bug on doing this.
13:21jlorenzoWG9s: thanks!
13:22CallekWG9s: fwiw, I know I had previously discussed with socorro team and was under the impression that buildid wasn&#39;t needed to match anymore (it used to be, for ingestion purposes aiui) -- I wasn&#39;t aware it was utilized in correlations/analysis that way
13:23CallekI don&#39;t think I came out and told them it was going away, because I was under the impression they didn&#39;t care anymore.
13:25WG9sCallek: I could be wrong but it seems to be you can search by buildid but not by changeset for example.
13:26Callekyea, I knew buildid was a searchable thing, but it still does have meaning, it just doesn&#39;t have meaning across platforms
13:30WG9sCallek: but you see buildid adn changeset used to be the same so if the buildids vary by timezone and you canot search by changeset we have lost something.
13:30CallekWG9s: I don&#39;t disagee fwiw on the &quot;we have lost something&quot;
13:31WG9sBut this is a crash reoporting issue i alsway thoug buildids were kind of lame.
13:32WG9sCallek: that said it still might be nice if all of the buildids were in UTC. so the changeset identeifies waht changes were useed and the buildid identifed the build tools but having them in multiple timezones is just confusing.
13:33CallekWG9s: once all stuff is in TC buildid&#39;s should end up matching again (aiui), and all be in UTC
13:34Callek(but we&#39;re still not guaranteeing matching buildid&#39;s, even then)
13:34catleeyeah, it&#39;ll just be more uncommon for them to differ
13:39WG9scatlee, Callek: used to be the triggering system for the build passed a build this changeset and here is the buildid to use.
13:40CallekWG9s: right now, we&#39;re triggering via two distinct ways, and the triggering system is indeed what generates the buildid (at present) in both cases
13:41Callekit is a problem we could have tacked brains and time on to solve to keep them aligned, but after discussions we decided that the added effort wasn&#39;t fruitful
13:42WG9sCallek: ok now that i know eferyting is going to taskcluster so the divergence whoudl be short lived. I get it .
13:42CallekWG9s: and again, we&#39;re still not guaranteeing no divergence
13:42Callekwe&#39;re just aware that with the way buildid is generated now, its less likely to diverge (once we converge everything on taskcluster) again
13:43WG9sCallek: adn nver was because of retriggers etc.
13:46WG9sCallek: I actually wonder if a buildid should be just a unique identifier and not a timestamp at all.
13:47CallekWG9s: iirc thats the technical intent, its considered an &quot;always increasing unique number&quot; timestamp is just an implementation detail (but i can see why some people think of it as part of the design)
13:47catleeright now the updater needs to see an update with a higher buildid to download and apply the update
13:47jcristaujlorenzo: also, the buildid is probably most relevant on nightly and aurora, where the version number is ~useless. on beta i doubt it matters.
13:47Callekor at least that was how it was explained to me something like 8 years ago
13:47catleejcristau: do you know much about nightly/aurora crash triage?
13:49WG9sCallek: well I am not sure i UUID is always increasing, but then if we are now generating buildids timezone specific and they are supposed to be monotomically increasing we have a problem there.
13:50jcristaucatlee: if you need people who do know about it, #uptime might be a good place
13:51catleeWG9s: that only applies within a single platform
13:52WG9scstlee: but if something goes very wrong with taskcluster and we do a buildbot build there is the possibility it will ook older.
13:53catleeWG9s: we wouldn&#39;t do that; we would fix up TC
13:54WG9scatlee: but just mr obvious pointing out the possible issue.
15:09jyaso I have my win8 loaner
15:09jyawhere are the source files installed ?
15:09jyaor do I have a blank machine and need to load them manually?
15:10jyaoh.. it&#39;s win8 with no SP and the original version
15:52tedcatlee: so did you find anything interesting about OSX signatures?
15:53catleeted: not yet. I managed to extract it properly. I started looking at that rust library
15:53catleeits macho support is pretty basic still
15:56catleebut yeah, a rust based utility with support for pkcs11 would be awesome
15:57tedfitzgen has done some mach-o stuff in rust as well, but i think his stuff assumes you&#39;re running on OSX
15:57catleethen you&#39;d get support for most HSMs for free
15:58tedcatlee: i&#39;m pretty ignorant on the crypto side of things, you&#39;d have to ask around
15:59catleethat&#39;s just a C level API
15:59catleeall the crypto would happen elsewhere
16:00tedi know someone was working on rust bindings for NSS, dunno if that got anywhere
16:00catleethat could work
16:01catleeI think the hard part is the macho munging
16:01catleeit uses pretty standard crypto iirc
16:13travis-cibuild-puppet#1067 (production - 28be7ad : Rob Thijssen): The build passed. (
16:13travis-cibuild-puppet#1066 (master - c2d5173 : Rob Thijssen): The build passed. (
16:14tedcatlee: the mach-o munging seemed pretty straightforward
16:15tedbut it was a while since i looked at it
16:15tedIIRC it just hashed some bits of the binary or something, then generated a signature of that, then stuck that into the binary
16:15catleeyeah, you&#39;d need to add a new section
16:15catleeand possilby adjust some offsets
16:16tedIIRC the load commands are just an array after the header, so you&#39;d have to append a new load command at the end, shift the rest of the file contents forward by that much, then adjust any offsets in other load commands
16:16tedunless the linker is forward-thinking and leaves you empty space at the end of the LCs
16:17catleesigning is last in all the examples I&#39;ve seen
16:17tedif they left empty space after the load commands that would simplify things, certainly!
16:18catleeso MacOS/firefox-bin is in CodeResources, but MacOS/firefox isn&#39;t
16:32tedcatlee: weird
16:32tedcatlee: so just from some random poking, it does look like the linker might leave some extra space after the existing load commands
16:32tedwould have to actually write some code to verify that, but that would make sense
16:32catleeyeah, in my initial attempts I saw lots of null bytes
16:38jyacurrently re-downloading everything on the win8 loaner I got.. but surely, there&#39;s an easier way to simply simulate running that job, one that doesn&#39;t require to install visual studio, mozilla-build, recompile everything and run mochitest manually
16:38jyawant to re-run that run:
17:05sfraserrail: whimboo: just acknowledging the n-i on the partials bug, am looking into it but won&#39;t cancel the n-i until I have an answer
17:05whimboosfraser: thanks!
17:06whimboosfraser: i hope it has enough info that you can act on it
17:07railsfraser: thanks!
17:42RyanVMaki: reading the discussion in bug 1344321 RE: SM builds on your Try pushes - is it possible your results are just depending on the parent rev you&#39;re pushing?
17:42RyanVMi.e. they only get scheduled when there&#39;s changes to js/src or a few other dirs
17:43akibut the top level push is just taskcluster/ changes in some cases
17:43akii think the latest 2 runs were like that
17:44akispidermonkey with just taskcluster/ changes:
17:44akihm, the beta version doesn&#39;t though
17:45RyanVMok, was just throwing it out there as a possibility in case it fit what you were seeing
17:47akiyeah. i don&#39;t know why anymore.
18:11bhearsumrstrong: are we talking about the same thing in i&#39;m just wondering about removing the &quot;(websense)&quot; blocking rule (only matches users with the original system addon)
18:18rstrongbhearsum: from your comment you specifically state 47.0.2. Clients that are on 47.0.2 will get an updated system add-on that has (websense-X.X.X) when they have websense. There is no mention of there being a rule for checking that. Just &#39;does anyone have a reason I can&#39;t remove the Rule that blocks queries with &quot;(websense)&quot;&#39;.
18:20rstrongI suspect that if you want to remove the (websense) rule that the (websense) rule would be remove on all versions and not just 47.0.2 and there would be a rule for (websense-X.X.X) rule .
18:25rstrongbhearsum: so, is there more to it than just removing the (websense) rule and if so, what other rules would come into affect? If there are no others and you just want to update everyone then that is a release drivers call.
18:25rstrongand why just 47.0.2?
18:30bhearsumrstrong: yeah, the &quot;(websense)&quot; blocking rule has been holding users on <=47.0.2 at 47.0.2 since we created it - that&#39;s how that plays in
18:31rstrongbhearsum: are you proposing just removing all websense rules on 47.0.2?
18:31rstrongand just letting everyone update?
18:32bhearsumno, just the one for &quot;(websense&quot;). we have additional rules that hold known bad websense versions to 47.0.2
18:32bhearsumeg: &quot;(websense-8.2.23&quot; is held to 47.0.2
18:32bhearsumi just want to get rid of the rule for the older system addon if it&#39;s not useful anymore
18:37jyathe mozconfig used for debug build, does that also disable optimisation?
18:39rstrongbhearsum: and clients that don&#39;t have websense added (e.g. either (nowebsense) or (websense-X.X.X)) aren&#39;t updated as well?
18:44RyanVMjya: i don&#39;t believe so - my recollection is that they&#39;re unusably slow
18:45RyanVMjya: backs that up since I don&#39;t see an explicit --disable-optimize there
18:45jyaRyanVM: thanks will try again without it then
18:46RyanVMjya: yeah, from - &quot;dnl = Enable code optimization. ON by default.&quot;
18:46jyaRyanVM: what would be the best way to create a mozconfig or passing an option to mach that uses the exact same build options as on the build bot?
18:46RyanVM^ is a good start :)
18:47jyajust include browser/config/mozconfigs/win32/debug ?
18:47RyanVMthere&#39;s a bit of nesting there to be aware of (and some automation-specific items), but that&#39;s probably the most-relevant stuff
18:47Callekjya: noteworthy, &quot;the in tree mozconfigs are expected to be only applicable to our automation, they may invoke/do things that are not expected in your local environment
18:47* Callek has no other context
18:47jyaCallek: i&#39;m building on a loaner
18:48bhearsumrstrong: we&#39;re blocking everyone sending &quot;(websense&quot; (new system addon) at 51.0.1 still
18:51rstrongbhearsum: I was asking about clients that don&#39;t have either either (nowebsense) or (websense-X.X.X) on 47.0.2. Will they be blocked?
18:52bhearsumthey will not be blocked
18:52bhearsumand they aren&#39;t blocked right now
18:59rstrongbhearsum: I thought they were and this can obviously lead to a crash if the client doesn&#39;t run for awhile and launches 47.0.2. Would you like me to comment in the bug to that affect or do you want to? Was this ok&#39;d by release drivers?
19:07bhearsumlizzard was OK with not blocking updates for users who don&#39;t have either of the System Addons IIRC
19:08bhearsumwe still don&#39;t have the ability to do multiple matches in OS_VERSION
19:08lizzardOh, i thought we were still holding nowebsense people back.
19:10bhearsumwe are holding back anyone who we know has websense installed
19:10bhearsumwe are not holding back people who definitely don&#39;t, or don&#39;t have either of the system addons installed
19:10rstronglizzard: this was in relation to clients on 47.0.2 that don&#39;t have the system add-on. For example, if Firefox doesn&#39;t run for awhile, launches, and there is an update check chances are they won&#39;t have the system add-on especially since system add-ons aren&#39;t restartless in 47.0.2.
19:12lizzardHow did I not realize this. OK. Thats a fairly significant amount of people last I checked, that are now at least up to 51.x
19:12lizzardrather than being held back on 47
19:12lizzardI emailed the forcepoint list to ask if they did their release in response to our 52 release (so that we can let through m ore users as planned)
19:16bhearsumcall me cold, but i&#39;d rather have the general population get up to date than stay on a super insecure 47.0.2 for the sake of a handful of websense users
19:19lizzardme too
19:19lizzardIts time
19:19lizzardso im glad it happened already (right?)
19:20lizzard(i have said what bhearsum said in every meeting about this ..)
20:05rstrongI&#39;m fine with taking that approach as long it is clear that is the approach we are going to take.
20:07rstrongFor example, if I had known we probably wouldn&#39;t have created and pushed the system add-on for 47 through 49.
23:37Aryxaki: hi, flake complains:
23:38akiAryx: ty, i&#39;ll push a fix
15 Mar 2017
No messages
Last message: 9 days and 7 hours ago