mozilla :: #build

13 Sep 2017
00:07glandiumdmajor: https://stackoverflow.com/a/28627790
00:09dmajorglandium: ok, thanks. bottom line is I don't think it's msvcrt
00:09glandiumdmajor: isn't int 3 a MOZ_ASSERT?
00:09glandiumor something like that
01:12dmajorglandium: MOZ_ASSERT is int 3, but not all int 3 is MOZ_ASSERT. It was doing something weird like "test rsp, 3" which looks like maybe some compiler or library generated goop
01:16dmajor_chkstk or something
01:18dmajorand it didn't have the telltale static string of gMozCrashReason
08:47bobowenglandium: ping
08:48glandiumbobowen: pong
08:48bobowenglandium: hi, it was just over bug 1380609
08:48firebothttps://bugzil.la/1380609 ASSIGNED, bobowencode@gmail.com Make Win10 SDK (minimum v10.0.10586.0) required for building Firefox
08:49bobowenglandium: I'd like to get it in for 57
08:58bobowenglandium: thanks, I put the version in a variable because I was using it in the error message as well, should I just make the variable a string?
08:58glandiumbobowen: ah, meh, either way is fine
08:59bobowenOK thanks
12:18rillianjkratzer: I think the crashreporter is enabled, but you have to set MOZ_CRASHREPORTER=1 in the environment for it to do anything with unofficial builds.
12:19rillianjkratzer: your pkg-config error just looks like the correct -dev packages aren't installed.
12:19rillian`apt-get builddep firefox` might help
12:21rillianor you can try going through the package list at https://github.com/mozilla/gecko-dev/blob/master/python/mozboot/mozboot/debian.py#L49
12:25rilliangandalf: https://groups.google.com/d/msg/mozilla.dev.platform/5srcnP-Wj4E/KvoaSUlRAgAJ is the best setup at the moment, I think.
12:29rillianyou can also just --enable-ccache=sccache but that (a) runs the preprocessor for C/C++ cache hits, which can be slower than traditional ccache, and (b) some people have reported conflicts between icecc and sccache. I haven't investigated so YMMV.
12:29rillianas for setting -j programmatically, mystor wrote https://gist.github.com/mystor/27beaf86a4868a357dcd58f585b42c6a
12:30rilliannote you need to update the scheduler to match your icecream config
12:37atorillian: That script cant be right. It reports 140 in London, but icecc-monitor is showing me a total of 60 at moment.
12:37atoI thought icecc-monitor jobs were total number of cores for the given arhcitecture.
12:38atoOh the IP of the scheduler is hardcoded (-:
12:39rillianaha :)
12:39atoUsing icecc-scheduler.corp.lon2.mozilla.com reports 80, which is still off though.
12:40rilliansurely they're both talking to the same scheduler interface?
12:41atoSomeone just switched on a new machine, so were up to 76 now in icecc-monitor. So I think the script includes the machines that are rejecting jobs. (Has job limit set to 0 in /etc/icecc.conf.)
12:41atoMy laptop is on the icecc network, but refuses icecc jobs, and this machines cores are included.
12:43rillianato: is your laptop mac? not the script adds the local cpu count unless uname is darwin
12:43atox86_64 Linux. The Darwin jobs appear to be excluded.
12:43atoTheres an 8 core Darwin machine, but 60 + 4 (number of cores on my laptop) makes sense.
12:44atoICECC_ALLOW_REMOTE="no" or ICECC_MAX_JOBS="0".
12:44rillianhuh, vlan won't let me query lon2
12:45rillianbut at least we've finally used the number in the office domains!
12:45atoI guess an extra 4 jobs isnt a big issue (-:
12:45rilliannot really. Or you can delete that part of the script
12:45atoI always use a few more jobs than I have because of poor saturation.
12:47rillian*nod*
12:57mystorato: in the newest version of the script I actually use 1.4x the number of jobs I theoretically have access to for saturation
12:58mystorAnd yes, it only searches for x86_64 jobs because I only built it for myself and I'm using Linux :-)
14:14jkratzerrillian: I believe I've installed all necessary dependencies - https://pastebin.mozilla.org/9032300
15:47rillianjkratzer: if you're not packaging firefox, I don't recommend using any of the in-tree mozconfig files. They assume a particular build environment which most developer systems won't reflect.
15:47jkratzerrillian: do you happen to have an example of a debug mozconfig?
15:47nalexandercoop: gps: hey, do we have any data on what editors Mozilla engineers are using? Any chance we have (editor, contribution area)?
15:48RyanVM(configure will actually error out if you try to use an in-tree mozconfig unless you set an env var)
15:48nalexandercoop: gps: I'm interested in what folks who write JS are using.
15:48rillianjkratzer: I have something like:
15:48rillianac_add_options --enable-debug
15:49rillianac_add_options --enable-optimize=-Og
15:49jkratzerRyanVM: I patched out the check in build/moz.configure/init.configure
15:49jkratzerrillian: that's it? Do you need to specify clang paths, etc?
15:49jkratzeror will it just use gcc if clang isn't specified
15:49jkratzerclang/llvm
15:49rillianjkratzer: it will use the default compiler on your system
15:50jkratzernalexander: I use WebStorm - we have a license for it
15:50rillianso you only need to set CC/CXX if you want something different
15:50jkratzerrillian: thanks - I'll give that a shot
15:50nalexanderjkratzer: ta. I'm looking for aggregate usage, though, to understand some editor integrations and where the highest value might be.
15:51froydnjnalexander: send a google survey to mmayo-all? :)
15:51jkratzernalexander: ah right, my apologies
15:51nalexanderfroydnj: thanks for your novel suggestion ;)
15:52nalexanderfroydnj: BTW, do you have time to do the next pass on the Debian build images patch?
15:53rillianjkratzer: also, try pkg-config --modversion freetype2
15:54jkratzerpkg-config --modversion freetype2 -- 11.0.5
15:54jkratzerrillian: ^
15:55froydnjnalexander: I think I mostly understood the feedback and was going to try to apply some of it today or later this week
15:56nalexanderfroydnj: ta. I'll try to get to that Pre: patch, but I need to wait for some management alignment before I commit to Android build work.
15:59froydnjwhy does the number of docker image layers matter (e.g. combining RUN commands)?
16:05nalexanderfroydnj: I think it's an efficiency thing; fewer layers to pull individually, faster resulting filesystems.
16:05nalexanderfroydnj: but gps would know more.
16:22gandalfrillian: thank you!
16:45rillianjkratzer: maybe the error message is bad, and it's actually the pango bits it's missing?
17:02gpsnalexander: no clue what editors people are using
17:02nalexandergps: it wasn't in the developer productivity surveys of yesteryear?
17:02gpsi don't think so. but i'll double check
17:03gpsnalexander: i don't see a question on editors
17:03gpsat least not one that answers what you want
17:04coopgps: could that be added to our telemetry data?
17:04gpscoop: that's kinda hard to get. but we could try e.g. scanning process lists for known editors
17:05gpsbut we'll only be able to detect what we teach it to
17:05gpsunless we send data on all running processes, which i'm not comfortable even asking for
17:10gpsfroydnj: regarding docker image layer count, bug 1329282 changes everything. so i wouldn't worry about it
17:10firebothttps://bugzil.la/1329282 NEW, jopsen@gmail.com Deploy QEMU engine to build docker images
17:10coopgps: could re read editor=... out of .hgrc if it exists?
17:10cooper, we read
17:10gpsthe vcs editor has a good chance of not being the primary editor
17:10coopfair enough
17:10gpse.g. i don't use visual studio as my vcs editor
17:11gpsthere are only a finite number of editors. if we can't identify an editor for a user making commits, we could ping them
17:11froydnjgps: just the title of that bug terrifies me
17:12gpsessentially, docker isn't very secure. building docker images in a proper vm is more secure
17:13froydnjImma gonna build a vm to build your vm
17:14froydnj(I know docker's not exactly a vm)
17:14gpsvm-ception
17:14froydnjgps: already consolidated RUN and ENV commands where I could, sooo...
17:15* froydnj unsure of how to sort out this python packaging stuff
17:16froydnjgps: my understanding of https://bugzilla.mozilla.org/show_bug.cgi?id=1396098#c6 is that you're saying to put all that in a script that would pull from tooltool and run those commands, and then that script can be run during docker image creation?
17:16firebotBug 1396098 NEW, nobody@mozilla.org move android builds to debian-based docker image
17:16gpshaving looked at debian a bit more since i did the review, are debian's built-in python packages not good enough?
17:17froydnjI don't know; I was cargo-culting from what nalexander already had
17:17froydnjso I assume they are not, but I don't really know
17:17gpsif you're using stretch, i think they should be good enough
17:18nalexanderfroydnj: I ended up using Debian's Python packages. In your changes, you found that Debian's Python didn't support sparsecheckout.
17:18nalexanderfroydnj: but for my usage, Debian's Python was good enough.
17:18gpshuh
17:18gpsstretch install pip 9.0.1, which is definitely new enough
17:19froydnjnalexander: you mean Debian's mercurial didn't support sparsecheckout?
17:19gpswe can't use the distro's mercurial. full stop.
17:19nalexanderfroydnj: huh, I meant Mercurial, not Python.
17:20nalexandergps: oh, okay then.
17:20gpsmodify install-mercurial.sh. i think i saw this in the patches.
17:20gpsactually...
17:20nalexandergps: yeah, I think froydnj did the right thing there. I was using Debian's Mercurial.
17:21gpsi wrote what is essentially a toolchain task for building a custom mercurial .deb: https://hg.mozilla.org/users/gszorc_mozilla.com/firefox/file/240b4a9e42ca/taskcluster/system-packages/mercurial-deb.sh
17:22gpswe could upload the .deb files and install in install-mercurial.sh
17:22gpsafter slightly more thought, the current patch is better
17:23gpswe'll do proper .deb packages in a follow-up
17:24froydnjI guess we're installing non-Debian pip because python-pip pulls in a bunch of things
17:25froydnjso sayeth the comments
17:25froydnjpython-pip on my ubuntu system has...ca-certificates, python-pip-whl, and python
17:25gpsoh right
17:26gpsyeah, it pulls in gcc and a bunch of random stuff for compiling python c extensions
17:26froydnjthat does not seem onerous
17:27froydnjwell, build-essential and whatnot (which are recommends, not requires/depends) would be a lot
17:28froydnjbut we have to install all that anyway
17:28gpsapt-get install --no-install-recommends
17:28gpsthat should probably be the default for *any* apt install invocation tbh
17:28froydnjat least, we currently are, because we're using system compilers for HOST_CC and friends
17:28gpsif we need something that is pruned out, it can be installed explicitly
17:28froydnjwe do in fact use this flag where appropriate
17:29froydnjI guess I will try just installing the system pip and seeing where that gets us
17:44gpstjr: i suspect you'll be interested in bug 1329282 (iirc tor uses qemu to build)
17:44firebothttps://bugzil.la/1329282 NEW, jopsen@gmail.com Deploy QEMU engine to build docker images
17:44gpstl;dr taskcluster is gaining support to run QEMU
19:11rilliangps: toolchain tasks seem to run greedily on try, regardless of whether there's another job that needs the artifact. Is that intentional?
19:12froydnjrillian: are you modifying files that the toolchain tasks depend on?
19:12gpsrillian: they shouldn't run unless files are modified
19:12gpsthe yml file defining toolchains counts :)
19:13* froydnj watches builds take five minutes or more just to hit configure
19:13gpsin automation?
19:13froydnjyes
19:14gpswhere?
19:14froydnjhttps://tools.taskcluster.net/groups/aAs9xLZPSuSmz2O7UCBdqg/tasks/BRSNEX8QTy6wnppgG5zmqg/runs/0/logs/public%2Flogs%2Flive.log for instance
19:15gpssingle threaded tooltool download and application FTL
19:15rillianfroydnj: yes.
19:15gpsthe docker and vcs bits are pretty long poles though
19:16rillianI'm still surprised that we do this complicated overlay thing instead of just pickling everything into docker containers.
19:16gpsbecause bug 1291940
19:16firebothttps://bugzil.la/1291940 NEW, garndt@mozilla.com TaskCluster slows down considerably when using AUFS
19:16gpsplus we're using xz for the archives
19:17gpsxz is slooooow
19:17gpsfaster to use lz4 and transfer more data because the network is fast in automation
19:18rillianso...our tasks aren't actually running in docker containers?
19:18froydnj...do we have lz4 installed, or does tar (python or shell) understand lz4?
19:18froydnjmaybe we should just use gzip -3 or so
19:20froydnjeven the android ndk (~300MB?) took us almost a minute to download
19:20rillianand do we make separate .xz packages for things like the clang build we download to dev machines?
19:20froydnj5MB/s, 40Mb/s
19:21gpswe probably want xz for developers and gzip -3 or so for automation
19:21gpstooltool does support some other archive formats
19:21rillianfroydnj: I was also sad to see the toolchain builds spending a minute or two syncing m-c just for a couple of scripts. Not that they run often.
19:21gpsbut it can be touch and go depending on what tools are installed on system
19:21gpsbest to stick to what python can handle natively
19:21froydnjrewrite tooltool in rust
19:22rillianare we not replacing tooltool with taskcluster artifacts?
19:22gpshttps://hg.mozilla.org/users/gszorc_mozilla.com/firefox/file/3603aa6733b9/taskcluster/docker/bootstrap/build-image-main#l84
19:22rillian"Oh, you want *that* index? Hang on while I rebuild the world..."
19:22gpshttps://hg.mozilla.org/users/gszorc_mozilla.com/firefox/rev/2c0d6e can speed up toolchain tasks
19:23tedhah
19:23froydnjrillian: it looks like even taskcluster artifacts (e.g. sccache) get downloaded via tooltool
19:24gpsIMO use of tooltool in taskcluster represents a failure for us to build something as part of the taskgraph
19:24gpswhere "tooltool" = "tooltool server"
19:24gpsdo we use tooltool.py for random HTTP fetches?
19:25gpsjonas wrote some custom "download URL robustly" in python this past week too
19:25rilliangps: I wondered about that. The logs aren't clear if it's constructing a temporary manifest, or doing something unrelated and it's just next to the tooltool downloads.
19:26gpslet's extract this to a standalone python module and do away with all this wheel reinvention
19:26* gps lunches
19:26rilliangps: requests? :-)
19:27tedhah
19:38gpsrequests doesn't do hash verification
19:38gpsbut, yes, we basically need a simple wrapper around requests
19:41rillianrequest_but_verify
19:44rillianhash verification, tar/zip unpacking, output renaming, gpg signature checking.
19:44rilliananything else on the wishlist?
19:48nalexanderrillian: command line uploading that doesn't include manifests. The tooltool upload process is counterintuitive to me.
19:48nalexanderAbility to upload from an existing (public) URL. I need to shunt the ANdroid SDK or NDK into tooltool a lot.
19:49gpswe're not writing a new tooltool client
19:49gpsjust a Python API to securely download a URL
19:50gpswe could potentially also make it intelligently unpack archives too
19:50gpsso it doesn't perform decompression that isn't needed
19:50gpsalthough, there aren't hashes in tar files are there
19:50gpsboo
19:50rilliannalexander: what would you like the interface of this tool we're writing to look like?
19:51rillianthe taskcluster thing isn't bad; you just write a script to leave a certain file in a certain directory.
19:51rilliandon't know if that works for private things though.
19:51rilliansorry, this tool we're _not_ writing.
19:51* rillian messed up the joke
19:51nalexanderrillian: oh, sounds like you're producing something different. Basically, I want tooltool upload URL to "just work".
19:52nalexanderhttps://github.com/mozilla/build-tooltool/issues/39 was part of that. But let's leave it for now, tooltool sounds like it's not long for this world.
19:52gpsnalexander: write a simple shell script that does:
19:52nalexandergps: I know, I know.
19:52gpstooltool.py add --visibility public $1
19:52nalexandergps: but I need to download it locally, and then upload again.
19:52gpstooltool.py upload --authentication-file /path --message $2
19:53nalexanderAnd my upload bandwidth is very limited.
19:53rilliannalexander: I usually ssh into the office and run uploads from there.
19:53nalexanderrillian: I used to ssh into people, but it's gone. What host do you have access to?
19:53rillianstill limited, but much better than at home.
19:54rilliannalexander: I have a desktop machine in the office.
19:54rillianI can make you an account if you'd like.
19:54rillianor you could have my old one..
19:54nalexanderrillian: oh, right. Would you? Username nalexander
19:54nalexanderrillian: I'll send you my public key.
19:55* froydnj realizes his uploads could have gone so much faster with people.m.o
19:55froydnjmissed opportunity, there
19:55gpsssh.mozilla.com still exists IIRC
19:55nalexanderLike the old days, frantically searchign for leech shells :)
19:55nalexanderon T1 lines, when it mattered :)
19:55nalexandergps: I thought that didn't let you run a shell?
19:56nalexanderOr is that hg.m?
19:56rilliangps: I can't log into it
19:56gpsi have no clue. i've used it as a jumphost in the past
19:56rillianthere's jump1.community.scl3.mozilla.com or something like that
19:56gpsi also have gregoryszorc.com on some ec2 micro instance or something
19:56gpsif i need fast bandwidth, i ssh into that :)
19:56gpssuper cheap
19:57rillianyes. since they took away people I've just expensed vps rentals
19:57gpsi am using gregoryszorc.com as my IRC client. maybe i could get that expensed :)
19:57gpsi'm sure someone will tell me i should use IRC Cloud
19:58gps... which i don't have permission to use for some weird reason
19:58gpsi don't care enough to address that :)
19:58rilliangps: it costs money either way. go for whichever works.
20:01froydnj"remote: waiting for lock on working directory of /repo/hg/mozilla/try held by process '29707' on host 'hgssh4.dmz.scl3.mozilla.com/effffffc'" :(
20:01froydnjbeen a while since seeing a message like that
20:10* RyanVM wonders who's holding up the line
20:10RyanVMdon't they know we're wealthy businessmen who need our Try runs now?!
20:22froydnjmoar docker changes \o/
20:31froydnjargh, the clang build has suddenly decided it has link errors on x86
20:34RyanVMyou trying to make hazard builds work with 4.0 or something?
20:34RyanVMno, that'd be sfink
20:34RyanVMnvm me
20:35froydnjRyanVM: android builds on clang two weeks ago: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5ec6dfc1924d89732fcdd058ceb5f0a5b4c62973&selectedJob=127870760
20:35RyanVMoooooo
20:35froydnjRyanVM: android builds on clang today: https://treeherder.mozilla.org/#/jobs?repo=try&revision=24950a04ad153bc540fe34473e55e331cd6d25c6&selectedJob=130768584
20:35froydnjwhyyyy
20:35RyanVMignorant me wants to know what that buys us
20:35RyanVMbesides getting us closer to one less compiler to have to support
20:35froydnjwell, the ability to keep compiling firefox for android when gcc is really truly dropped from the android ndk
20:35gpsandroid upstream moved to clang
20:35froydnjslightly smaller binaries
20:35RyanVMaha!
20:36froydnjpossibly faster code (haven't tested yet)
20:36RyanVMwell, that's a pretty good motivation :D
20:36froydnjoh, and the ability to move to C++14
20:37froydnjstatic analysis on android is also a possibility
20:37froydnjwhich opens up other interesting things
20:37froydnjlike faster debug tests
20:37RyanVMneato
20:41froydnjaha, lth is my nemesis
20:46gpshttps://docs.python.org/dev/library/pathlib.html is so nice
20:48gpsit's little things like symlink_to() being named that so you don't have to remember what direction the symlink is in
20:53ted 0:39.11 error: environment variable `MOZ_TOPOBJDIR` not defined
20:53ted 0:39.12 --> C:\build\mozilla-central\xpcom\rust\nserror\src\lib.rs:59:22
20:53teduggh
20:55tedactually shit, building rust code is going to be a PITA in WSL ;-(
20:58froydnjargh, our configury doesn't detect that x86 android with clang actually needs -latomic
20:58froydnjfantastic
20:59* froydnj decides to put that off until tomorrow
20:59tedfroydnj: what would you think about wrapping cargo invocation in a python script?
21:00tedwe have a lot of makefile goop in rules.mk around it right now
21:00tedwe could do that in python, or make a bunch of it configure-substed or whatever
21:00froydnjted: ooo, that's not a bad idea
21:00froydnjI'd rather write python conditionals for all our cargo nonsense than makefile conditionals
21:00tedyeah
21:01tedi'm probably going to need to do the same thing for cargo i did for midl.exe: shell out to cmd.exe to run a batch script so i can set windows env vars
21:01froydnjprobably makes passing around environment variables and quoting and whatnot more straightforward too
21:01tedyeah
21:01tedhttps://irccloud.mozilla.com/pastebin/soHNzNGY/
21:01tedi don't love that, but there's currently no way (AFAIK) to pass environment variables to a windows process when running it from inside WSL
21:02tedso my workaround is to put the env vars into a batch file, run cmd.exe on it, and then run the command from in there
21:02tedfroydnj: OK, i'll work up a patch
21:02* ted needs to untangle the 50 other changes in his working copy
21:02tedalso i need to go start making dinner
21:02froydnjso, uh, this whole process-launching overhead win that we thought we might see isn't happening, eh?
22:15tedfroydnj: it's not *that* bad
22:15tednot everything needs this
22:15tedwe can run cl.exe just fine, and other stuff like that
22:15tedit's just things where we need to communicate info via env vars, and there's no other way
22:16tedmidl.exe needs to find cl.exe to run the preprocessor, but we can't set PATH for it
22:16tedit has a /cpp_cmd argument but if you pass a path with spaces something breaks
22:19tedmost of the build is just running linux binaries, so there's no fiddling at all
22:19tedthere's only a handful of midl.exe invocations
22:19tedand cargo only gets invoked a few times
22:19tedonce i get the whole build working we can see what the timing looks like
22:19tedconfigure is ~10s faster, i think
22:43nemodonHow do I build a code coverage build without crashing on asserts? I tried xpcom_debug_break env variable as warn, but that doesn't seem to work.
14 Sep 2017
No messages
   
Last message: 6 days and 16 hours ago