mozilla :: #rust-infra

18 May 2017
01:11simulacrumI'm seeing (error reading rustc version) on stable, beta, and nightly with rustup after running rustup update.. is it just me?
01:30simulacrumhttps://github.com/rust-lang-nursery/rust-forge/pull/62 posting for aidanhs :)
01:55simulacrumWhat do the checkmarks on the forge platform support mean?
01:56simulacrumAh, whether it works on the platform
02:31simulacrumLess than 2800 issues! :)
02:37jonhooHey all \o/
02:38jonhooI noticed there hasn't been a Nightly release two nights in a row now. Is this a known issue?
02:39simulacrumhm, that's odd
02:39* simulacrum runs rustup update
02:40jonhoo$ curl -s https://static.rust-lang.org/dist/channel-rust-nightly.toml | grep -A1 '\[pkg\.rust\]'
02:40jonhoo[pkg.rust]
02:40jonhooversion = "1.19.0-nightly (75b056812 2017-05-15)"
02:41simulacrumYeah, I don't know. It might be because we've recently oxidized the rust-installer, but not sure what that would've affected
02:41jonhoohttps://rusty-dash.com/nightlies reports the same thing
02:41simulacrumI'll bring it up on Friday's infra meeting if it's not fixed by then
02:42jonhooThanks. I might end up joining in as well actually. Here at 2:30 PST, right? I have a thing at 3:00 PST, but might poke in for a short while.
02:43* simulacrum goes off to convert time zones
02:43simulacrumerr... PST is pacific time?
02:43jonhoothat's what I was told. I'm on Eastern myself, so also had to convert :p
02:43simulacrumin that case, yeah
02:44* simulacrum is in Colorado :)
04:38acrichtosimulacrum: FWIW I haven't looked at the logs
04:38acrichtowill do so now
04:38acrichtobut I'm betting it's the submodule issue
04:40acrichtobingo
04:40acrichtoerror: failed to read `/tmp/nightly/rust/src/tools/rust-installer/Cargo.toml`
04:40acrichtothe fix for that is now on master
04:40acrichtoso hopefully we'll get a nightly tmw
04:46nagisated, acrichto: x86 SHA-2 instruction can hash 2GB of data per second
04:46nagisajust sayin
04:47nagisayes, I did benchmark it for ZFS purposes.
07:09HavvyWhat's the easiest way to check the current version/date of Rust nightly?
08:08onurHavvy: you can check pkg.rust.version in https://static.rust-lang.org/dist/channel-rust-nightly.toml
08:09onurI think somebody made a twitter bot for this
08:09Havvyonur: The twitter bot punts to rustup.
08:09HavvyWell, rustup and rustc --version
08:10HavvyIs there a way that doesn't involve downloading such a large file?
12:27aidanhsHavvy: that file is ~250K, which doesn't seem that large! I suppose you could always partially download until you get the pkg.rust line, which seems to be about 20K in
12:27aidanhsalternatively, it doesn't seem unreasonable to get cloudfront to perform gzip compression, which doesn't seem to be enabled afaics
12:28aidanhsacrichto: something you might know about?
12:30est31https://rusty-dash.com/
12:30est31err
12:30est31https://rusty-dash.com/nightlies
12:31est31i dont understand though why there are no nightlies
12:31aidanhshmm, maybe it's not compressing because of `Content-Type: binary/octet-stream`, which I suppose needs to be fixed on the upstream server
12:32aidanhs(at least, that's what http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html is making me suspect)
12:34est31nagisa: 2 GB per second, really?
12:34est31care to make a comment to this thread: https://github.com/rust-lang/rust/issues/41215
12:35est31if its really 2 gb, then sha-2 is really fast: https://blake2.net/skylake.png
12:35est31(of course one should check how well those hash algos perform on ryzen)
12:35est31maybe we can determine the hash algo depending on the cpu we run on
12:36est31idk, if we find intel sha extensions, we use sha, otherwise some super fast non crypto hash
12:37simulacrumest31: Nightlies are broken right now but should hopefully be fixed 'soon'
12:37simulacrumi.e. today
12:38simulacrumsubmodule trouble
12:38est31simulacrum: what is the reason?
12:38est31I've thought the new mechanism makes this impossible
12:38simulacrumrust-installer moved locations, IIRC
12:38simulacrumI don't really know the specifics
12:38est31ok
12:38* est31 shrugs
12:38est31good luck!
12:38simulacrumThanks :)
12:41nagisaest31: I did benchmark it specifically for ZFS purposes
12:41nagisaagainst the ZFSs own very fast hasher which is pretty trivial, and also used AVX2
12:42nagisaest31: and that one does like 4GB/s
12:42nagisaest31: and yes I did benchmark on ryzen, because I havent any other machine that supports AES-NI
12:43est31what does AES-NI have to do with this?
12:43est31ryzen is the only desktop cpu being sold that has intel sha extensions
12:43est31intel originally made them for mobile
12:43nagisabecause those instructions are part of AES-NI
12:43nagisaIIRC anyway
12:44est31https://en.wikipedia.org/wiki/Intel_SHA_extensions
12:45est31AES-NI is only about ... AES?
12:46est31but anyways, this is off topic to this channel
12:46* est31 reads a book
13:30tednagisa: we wound up using ring's SHA-512, which is pretty fast: https://github.com/mozilla/sccache/issues/108#issuecomment-296006697
13:31tedbriansmith indicates there that he has a WIP impl of it that's even faster (1000MB/s)
13:33nagisated: still half as slow as the dedicated instructions for SHA256 :P
13:33tedsure
13:52acrichtoaidanhs: yeah I've often thought we should gzip the manifest
13:52acrichtoaidanhs: should be a relatively easy addition to the build-manifest builder!
13:52acrichtounsure if cloudfront can do it automatically
13:54IcefozI wonder if it's worth making cargo's package manifest be something more compact than a billion little json files in subdirectories...
13:56aidanhsacrichto: yeah was thinking it'd be easier to punt to cloudfront, just needs a setting toggling and the upstream server to serve toml as text
13:57aidanhsare the upstream webserver configs on github anywhere? oh wait...it's just s3 isn't it? dunno about configuring content type there :/
14:02acrichtoaidanhs: this may be a mime type thing
14:02acrichtowehere we just need to tell it the mime type of *.toml files
14:02acrichtohttps://github.com/aws/aws-cli/issues/1249#issuecomment-109391564
14:02acrichtomaybe that'd help?
14:03acrichtoif you're curious should be a relatively easy tweak to rust-central-station
14:33aidanhsoho neat, I'll do that later today
14:38steveklabnikalmost serious bors feature request: @bors:
14:39steveklabnikheh
14:39steveklabnikand @bors: for r-
14:49frewsxcvoooooo i want that
14:58larsbergshouldn't be too rough to add some config options to the main file and then look for more than `r+` or `r=` (which IIRC is all it does today)
14:58larsbergGood E-Easy :-)
15:35acrichtoonur: I said I'd get back to you yesterday about easydns things
15:35acrichtoI did not
15:35acrichtobut we're ok with it
15:35acrichtoonur: lemme know when you're around and I'll start the process
15:35acrichtonot that I really know what the process is
15:35acrichtobut I'll go poke at the easydns console
15:44jdmaturon: there are three parts to servo's intermittent failure tracking setup
15:44jdmthe first is that all intermittent failures are tracked as issues in servo/servo
15:44jdm(with a particular label)
15:44jdmthe second is https://github.com/servo/intermittent-tracker/ which duplicates the information from the issue tracker to avoid having to use github's API
15:44jdmthis means that we don't have to deal with rate limiting
15:45jdmit's a webhook, so anytime an intermittent issue is opened/closed/edited, the information gets replicated to a local database
15:45jdmthe third part was recently created by students from NCSU and has not been put into production yet
15:46jdmthat work exists in https://github.com/adamw17/csc517ossproject
15:46jdmthe role there is to have a database of the actual failures that occur and details of the builder, OS, time, etc.
15:47aturonok cool
15:47aturonso let me tell you where we are on Rust, and what we'd like
15:47jdmwe already have a infrastructure that classifies failures that occur as known intermittent or not based on that database from the second part
15:47aturonwe also use issues with a particular label for tracking
15:47aturon"already have" == used in prod?
15:47jdmyes
15:47aturonor part of the NCSU project?
15:47aturonok
15:48aturonso we are currently good about making sure every @bors: retry is labeled with a tracking issue
15:48aturonbut we don't have a process to track information about individual failures
15:48aturonwe've been trying to log manually in a spreadsheet, but that's gone about as well as you'd expect :)
15:48aturonjust knowing the count of a given failure over a span of time would be great
15:49jdmyeah, that's the purpose of this third part
15:49aturongotcha
15:49jdmbecause we made it possible for homu to merge things automatically if the only failures are classified as intermittent
15:49jdmbut we lost any sense of what intermittent failures were actually common
15:49aturonmm interesting
15:50aturon(separate note, we really need rust to get on the same homu as servo)
15:50aturon(and highfive, etc)
15:50aturonok so what's the status? are the students actively working on this project?
15:50jdmno, the students are finished their part
15:51jdmwe need to integrate the reporting into https://dxr.mozilla.org/servo/source/python/servo/testing_commands.py#509
15:51jdmwhich is the mach command that runs as part of our CI to classify failures
15:51aturonjdm: oh so to clarify, is known intermittent *detection* now automatic for you?
15:52jdmaturon: yes
15:52* aturon is jealous
15:52jdmaturon: note, only for automated test failures at the moment
15:52jdmsince it was a matter of parsing structured log files and querying a database
15:52aturonyeah
15:52steveklabnikaturon: speaking of, i've always wanted to re-write highfive's message
15:53steveklabnikit gives imho questionable advice we don't always follow
15:53steveklabnikaround not rebasing, etc
15:53steveklabnikand could do a better job of providing context around the whole r+ system
15:53steveklabniktoo many things to do...................
15:53aturonsteveklabnik: that much shoudl be easy enough -- we use the one under nrc
15:53steveklabnikyeah
15:53aturonjdm: ok, well thanks so much for filling me in!
15:53aturonjdm: i'll bring all this back to the full infra team and may ping you with some more detailed questions later on
15:54jdmcool cool
15:54noxjdm: Are there docs about how it works in TreeHerder?
15:54jdmnox: "it"?
15:54noxIntermittent tracking.
15:54jdmnox: bugs in bugzilla and database queries, and http://brasstacks.mozilla.com/orangefactor/
15:55* jdm wants to build an orangefactor equivalent for servo
15:55noxjdm: So it's just smarter than ours?
15:55noxjdm: I was mostly curious about whether it's able to detect that such stacktrace corresponds to this or that bug,
15:55jdmnox: no, it just looks at the full test output and does substring matching.
15:55noxas opposed to ours which only knows which tests are intermittent or not.
15:57noxjdm: So it's not like your tool is that far away from Gecko's, right?
15:58jdmnox: not really
15:58noxjdm: I mean on the recognition front.
15:58jdmright, I was agreeing
15:59noxjdm: Lol, I had forgotten I had written that question in a negative way.
16:21acrichtosteveklabnik: ping
16:21steveklabnikpong
16:21acrichtosteveklabnik: hey docs.rs things
16:22acrichtoI've configured easydns to serve up the docs.rs domain
16:22acrichtowanna switch over the nameservers?
16:22steveklabnikyes
16:22acrichtoeasydns informs me to look at this page https://fusion.easydns.com/Knowledgebase/Article/View/41/7/settingchanging-nameservers
16:24steveklabnikso wait
16:24steveklabnikwhat do i set them to?
16:25acrichtodns1.easydns.com
16:25acrichtodns2.easydns.net
16:25acrichtodns3.easydns.ca
16:25acrichtosteveklabnik: ^
16:25steveklabnikah
16:25steveklabnikthis page was from the perspective of setting it *away* from those
16:26acrichtooh lol
16:26acrichtothanks easydns
16:28steveklabnikokay
16:28steveklabniki clicked some things
16:28steveklabniklet's see if everything breaks
16:28onurthanks again, I thought you could add it into a master account or something, but looks like it's costing money per domain :/
16:29acrichtoonur: on easydns is definitely penny-and-diming us
16:29acrichtobut no worries!
16:29acrichtoit's just pennies and dimes
16:52acrichtosimulacrum: r? https://github.com/rust-lang/rust/pull/42086
17:02tedoh, is the rust project taking ownership of docs.rs?
17:04acrichtoted: nah just hosting the domain
17:04acrichtoonur's still running all the infra
17:04acrichtoand steve still owns thedomain name itself :P
17:07tedah
17:07tedhah
17:07tedi'd love to have the option to upload docs from builds on master to docs.rs
18:32Havvyaidanhs: It's more than 250kb when you want 20 bytes or so is overkill.
18:42aidanhspresumably you're checking the last nightly to potentially download it? if we assume 1 rust nightly every 5 days (i.e. our reliability at producing nightlies has dropped to 'pretty bad'), downloading 250kb every day is a under 1% of your transfer
18:42aidanhsmaybe I don't understand your use-case
18:43aidanhsyou could also try querying the url you'd expect a nightly to end up at directly with a HEAD request and check for 404 :)
18:43aidanhsjust throwing some suggestions out there
20:31simulacrumacrichto: r+-ed
20:31acrichtook this graph now has a scray trend --
20:31acrichtohttps://docs.google.com/spreadsheets/d/1aBfKT9j4lwpDQePRggRCy7zqhv46hCtRvTGGC9bPSn4/edit#gid=1400027294
20:31acrichtosimulacrum: thanks!
20:32simulacrumI think it's just more PRs, the percentage wise is about the same?
20:33simulacrumThough I agree the trend is overall somewhat frightening
20:33acrichtooh it was the uptick in "needs review" to me
20:33acrichtoeverything else seemed to stay constant
20:33acrichtobut lots of prs seem to not be getting reviewed?
20:34simulacrumI think part of it might be that we're labeling differently now
20:35simulacrumThat is, presumably, before needs-review was in the spreadsheet only when PRs went without review for more than a day
20:35simulacrumNow, every PR / most PRs pretty much ends up in the spreadsheet as needs-review at some point
20:36simulacrumLooking at the ages of the PRs there are a few that are really old (28 days) but most are ~5 days
20:38simulacrum1/3 of the current Needs-review are opened within 48 hours
20:38acrichtosimulacrum: hm perhaps yeah
20:38simulacrumer, 24 hours
20:38acrichtoalthough it feels gut-wise like we've been reviewing less this week
20:39acrichtonot that that feel has ever gone away :P
20:39simulacrumheh
20:39simulacrumI agree with the feeling though
20:39simulacrumMight be just slowness for w/e reason
20:39simulacrumCould be people posting more PRs as the semester ends
20:40acrichtomaybe yeah
20:40acrichtolots of people have been gone this week as well
20:40acrichtobrson/niko/felix
20:40simulacrumYeah that would explain a lot :)
20:41PhrohdohTwo questions regarding https://github.com/rust-lang/crates.io/issues/717#issuecomment-302526700
20:41Phrohdoh1) Where does this `human` come from? (is it the `fn human() -> bool`? seems unlikely)
20:41Phrohdoh2) Why is that code calling human to begin with?
20:42simulacrumcarols10cents: ^, do you know?
20:46PhrohdohDoes the crates.io codebase support API/auth keys that aren't at all related to github?
20:48simulacrumI don't think so but not sure
20:49PhrohdohOkay thanks. I may do some research on that subject then.
20:51PhrohdohIt looks like https://github.com/rust-lang/crates.io/issues/717 may have a dependency on https://github.com/rust-lang/cargo/issues/3995 from a UX standpoint
21:45PhrohdohFrom the index: {"name":"byteorder","req":"^0.5","features":[],"optional":false,"default_features":true,"target":null,"kind":"normal"}
21:45PhrohdohWhat are "target" and "kind" here?
21:53Phrohdohping
21:54edunhamaturon: in the drive infra folder, I was trying to open the FAQ doc (clicked on it but my mouse may have registered a click and drag event?) and Drive gave me a "deleted the document but it's still available to collaborators" message, and i hit undo and it gave a server error, and now I cannot find the FAQ doc anywhere. It's also not showing up in my trash or the folder's activity screen. Do you have
21:54edunhamspecial owner perms to see or retrieve it?
22:22carols10centsPhrohdoh: the `human` fn makes a human-focused error message
22:22PhrohdohWhere is it defined?
22:23carols10centsPhrohdoh: https://github.com/rust-lang/crates.io/blob/master/src/util/errors.rs#L226
22:23PhrohdohSo how does this break reporting 404s?
22:25carols10centslooking
22:30carols10centsPhrohdoh: ok so the way a 404 gets returned is the `response` implementation on NotFound https://github.com/rust-lang/crates.io/blob/master/src/util/errors.rs#L175
22:31carols10centsPhrohdoh: Diesel::NotFound errors get turned into NotFound errors here: https://github.com/rust-lang/crates.io/blob/master/src/util/errors.rs#L115
22:31carols10centsPhrohdoh: but calling `human` gets you a ConcreteCargoError, not a NotFound https://github.com/rust-lang/crates.io/blob/master/src/util/errors.rs#L227
22:32Phrohdohah ok yep that's an issue
22:34carols10centsok cool :)
22:35PhrohdohI'm not sure how the client should handle just 404s though
22:36PhrohdohBecause that could mean many things
22:36Phrohdohhost not found, package not found, etc
22:36carols10centshmm yeah. what about customizing the message that goes with the 404?
22:36carols10centsi have no idea if that'll work well or not, it's an idea
22:37PhrohdohIf you can send a string back with a 404, then sure that should be fine I think
22:38PhrohdohHow much of a pain is it to setup a crates.io instance locally?
22:40carols10centsPhrohdoh: yeah the current 404 response sends back json in the body with a string https://github.com/rust-lang/crates.io/blob/master/src/util/errors.rs#L172-L174
22:41carols10centsand i've done some work recently to make setting up crates.io locally less of a pain
22:41Phrohdohawesome
22:41carols10centshttps://github.com/rust-lang/crates.io/blob/master/docs/CONTRIBUTING.md#setting-up-a-development-environment
22:41carols10centslet me know if anything in there is wrong or is a pain :)
22:41PhrohdohI have a feeling that what your company is working on and something I started brainstorming overlap and you have way more time and knowledge than me :#
22:41PhrohdohSure thing!
22:41carols10centsi still don't have an easy way to get some data in the database
22:42carols10centsbut if you're testing 404s that won't be much of an issue ;)
22:43Phrohdohugh I hate using psql
22:43Phrohdohalways a pain just to get it running
22:43Phrohdoh...and it killed my shell
22:43carols10centso.O
22:43carols10centswhat OS are you on?
22:43PhrohdohmacOS
22:43carols10centsand how did you install postgres?
22:44Phrohdohvia brew
22:44Phrohdohbrew install postgresql
22:44Phrohdohthen: pg_ctl -D /usr/local/var/postgres start
22:44carols10centshm, me too, but i've never had psql kill my shell o.O
22:45carols10centsoh i just leave the server running all the time
22:45PhrohdohDo you know how to blow away the data dir entirely?
22:45Phrohdohhmm
22:45carols10centsum not off the top of my head
22:46carols10centsyou could like... blow away all of postgres entirely and install it again
22:46PhrohdohThat's what I've done
22:46Phrohdohit still says there is 9.4 data lying around
22:46carols10centso.O
22:46carols10centsidk, i google for the error messages at that point
22:47Phrohdohheh ok
22:49aidanhsPhrohdoh: looking at the path you mention, maybe /usr/loca/var/lib/postgres? on linux data directories tend to live in /var/lib
22:49Phrohdohunfortunately that path doesn't exist
22:51aidanhsoh wait, -D gives the data directory (wasn't thinking straight). so...have you nuked /usr/local/var/postgres?
22:52aidanhs(dunno if that's what you meant by "That's what I've done")
22:52Phrohdohpsql: FATAL: database "thill" does not exist
22:52Phrohdoh:-(
22:53Phrohdohpsql is one of the most un-userfriendly tools people apparently love
22:54carols10centsPhrohdoh: your username is "thill"?
22:54Phrohdohyes
22:54carols10centsI KNOW THIS ONE
22:54Phrohdohwoot
22:54carols10centshttps://github.com/rust-lang/crates.io/blob/master/docs/CONTRIBUTING.md#postgres
22:54carols10centsThen creating a template database for yourself:
22:54carols10centssudo -u postgres createdb [yourusername]
22:55PhrohdohDoes installing psql via brew automatically add that user?
22:56carols10centsmmmm i think so?
22:56carols10centsbut you deleted your data?
22:56Phrohdohhm my machine doesn't have that
22:56PhrohdohI deleted then installed again
22:56carols10centsdoesnt have what
22:56Phrohdohthe postgres user
22:57carols10centsok try just `createdb` by itself?
22:57Phrohdohsudo: unknown user: postgres
22:57Phrohdohok
22:57Phrohdohcreatedb: could not connect to database template1: FATAL: role "root" does not exist
22:57Phrohdoh:|
22:57carols10centsok now i think you might be root
22:57carols10centshttp://stackoverflow.com/questions/17633422/psql-fatal-database-user-does-not-exist
22:59PhrohdohI think I got it
22:59carols10centshttp://stackoverflow.com/questions/15301826/psql-fatal-role-postgres-does-not-exist
22:59carols10centsthose two things are in my SO favorites :-X
22:59* Phrohdoh scribbles down "host a meetup about how to use `psql` 101"
23:00carols10centsthis isn't even using psql, this is just setting up databases and templates the way the server wants
23:01carols10centslike you should'nt need to do this stuff every time you use psql
23:01carols10centsunless you blow away your data dir every time you use psql :-X
23:01carols10centsthere's a reason i just have SO questions saved, i only need to do this every once in a whlie
23:04Phrohdoher when I ran `createdb` what did it name that db?
23:04carols10centsi think your username
23:04Phrohdohok
23:04carols10centsunless you were root
23:04Phrohdohdo I _have_ to clone the entire index?
23:04carols10centsi mean... that's the easiest way to set up crates io
23:05carols10centsit's actually not very big
23:05PhrohdohBut I won't be poking at the frontend
23:05Phrohdohbut ok
23:05carols10centsthe backend needs it too...
23:06PhrohdohNOTICE: table "schema_migrations" does not exist, skipping
23:06Phrohdohexpected?
23:06carols10centsyep
23:06Phrohdohok
23:08Phrohdohyikes `cargo run --bin server` failed hard
23:09Phrohdoherror: failed to run custom build command for `libgit2-sys v0.6.9`
23:09Phrohdohhttps://gist.github.com/Phrohdoh/e25121a902e30bd19c31159424b689e8
23:10carols10centsuhhhhhhh
23:10carols10centsi haven't seen that before
23:10Phrohdoh:o
23:10Phrohdohfwiw: Warning: openssl is a keg-only and another version is linked to opt.
23:11carols10centser idk if that's relevant or not
23:11Phrohdohok
23:12frewsxcvtomprince: I'll try to look at your cargobomb PRs tonight
23:12carols10centshttps://github.com/alexcrichton/git2-rs/issues/147 maybe?
23:13Phrohdohoh right
23:13frewsxcvKeg only is a Homebrew term
23:13Phrohdohfrewsxcv: indeed
23:13Phrohdohthat message came from brew
23:14Phrohdohno dice
23:14PhrohdohShould I open a ticket in git2-rs?
23:15carols10centsprobably
23:16frewsxcvDo you have libssh installed?
23:16Phrohdohhuh strange that brew didn't show that when I did `brew search ssh`
23:16PhrohdohI'll try again
23:17tomprincefrewsxcv: Thanks.
23:18Phrohdohstill no dice, opening a ticket
23:22PhrohdohAlright well I suppose that is on-hold
19 May 2017
No messages
   
Last message: 91 days and 6 hours ago