mozilla :: #datapipeline

11 Sep 2017
11:31gfritzschejoy: that scalar is on 56, you can see it when switching the channel: https://georgf.github.io/fx-data-explorer/?search=active&searchtype=in_any&optout=true&channel=nightly&constraint=is_in&version=any
14:10Dextertrink, nice PR -> https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/82 (I'm acting on the behalf of Github's notification system)
14:16trinkDexter: it is not doc'ed as optional https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/environment.html
14:17Dextertrink, documentation update will follow up after the PR is merged
14:17Dexteror I can make the doc change first
14:17DexterDo you have a specific preference?
14:17Dextertrink, ^
14:17trinkjust wanted to confirm it will happen
14:18Dexterah, sure, it will ^^
14:18trinkmerged
14:18Dextersuper, thanks, I'll update the docs right now
14:20Dexterah hang on trink
14:20Dextergood catch :)
14:22joygfritzsche: but then how do i access active_ticks in main_summary?
14:22gfritzschesunahsuh can probably help with that? ^
14:23mreidjoy: you can continue to use the top-level "active_ticks" field
14:24Dextertrink, can I revert https://github.com/mozilla-services/mozilla-pipeline-schemas/pull/82 ? You got me thinking, that's actually not desired (the PR, not my thinking)
14:24joyoh but it isn;t a scalar which is why it is not in the doc(for release)?
14:24mreidjoy: https://github.com/mozilla/telemetry-batch-view/pull/282
14:25sunahsuhJoy: it hasn't ridden the train to release quite yet
14:25trinkDexter: what is the current plan?
14:25joysunahsuh: my question is a bit different
14:25Dextertrink, drop that changeset and ignore data with null creationDate
14:25joyactive_ticks has been in main_summary for a long tme
14:26Dextertrink, so restoring the status quo
14:26joybut it seems that variable is not a scalar. So, the q is, what sort of variable was it?
14:26sunahsuhIt was in the simple measurements section
14:27trinkDexter: ignore where?
14:27joyaah, okay. so is the plan to move all (or some) of simple measurements to scalars?
14:27sunahsuhThe eventual goal is to be able to remove that entire section
14:27Dextertrink, in the parquet. Leaving things as they are, so that null creationDates don't get into the parquet dataset
14:31trinkDexter: hmm so we will just ignore the error count in the output? (that may mask real/other errors). If we want to go that path the matcher should be changed so the parquet output doesn't even subscribe to them
14:35trinkDexter: the current PR is a better synchronization between the json and parquet schemas (as it is already optional there)
15:17Dextertrink, good points
15:18Dexterthen I think it makes more sense to keep the current PR, as that field is a "good to have" in parquet when it's there, but not something to discard update pings on if the field is not there
16:00FallenSo, I was just putting together some queries and made a mistake, redash gave me "Error running query: error totally whack". It looks like this is the case if the mysql driver doesn't understand the (new) error messages from mylsql 5.7. Is that something you can fix by upgrading things, or does it require an upstream change in redash?
16:01mreidRaFromBRC: ^
16:02RaFromBRCFallen, mreid: hrm... not sure... i'll check in w my team and will see what i can learn
16:02Fallenthanks!
16:07mreidchutten: I would like a link to said not-yet-existent blog post pls ;)
16:37Fallenrobotblake: I'm experiencing redash queries staying in queue and never executed (waiting a matter of minutes), can you check the worker queues?
16:44Fallen12 minutes and counting :)
16:45robotblakeThere's a pretty big queue backed up at the minute
16:45Fallenit could have been mine by accident. I terminated a pretty big queue, but it never went through
16:45robotblakeIt is slowly chugging through
16:45Fallen(the termination)
16:46FallenI'll wait then, thanks!
16:47robotblakeJust kicked the adhoc worker, looks like it also might've been hung
16:47Fallenthat worked!
16:48robotblakeCool, gotta take the dog out but I'll check the status again when I'm back
16:48Fallenthanks for taking care, happy walking!
18:53frankmreid, trink: do we have per-version validation for pings yet
18:53frankor is it just validated on the latest schema
18:55mreideach ping type has a version
18:55mreidyes
18:55mreidfrank: ^
18:56frankmreid: so to be clear: they are validated on the schema version that matches the ping's version
18:58mreidcorrect
18:58mreidI think "core" may be the only one that is actively using multiple versions
19:54ashortFallen: what db were you seeing this error on?
19:55Fallenashort: the AMO-DB
19:57Fallenashort: example query: SELECT JSON_CONTAINS(fv.validation, "EVAL", "$.messages[*].id[0]") FROM file_validation fv LIMIT 1
19:57ashortinteresting
22:21joywhat python library is used for google login in atmo?
22:21joy(i think when flask was used)
22:53RaFromBRCjoy: unfortunately the right people to answer your question are out this week... i can say, though, that going forward we want to switch to using the nginx auth0 module and having ATMO get the auth information directly from the http headers
22:57ashortinteresting
22:57ashortwhoops wrong window
22:59ashortjoy: django-allauth
22:59ashortjoy: atmo is a django app
23:00ashortsql.tmo is flask, and uses flask_oauthlib
12 Sep 2017
No messages
   
Last message: 9 days and 19 hours ago