mozilla :: #datapipeline

12 Jul 2017
00:21jgauntwhat the heck? &quot;TypeError: Can not merge type <class &#39;pyspark.sql.types.StringType&#39;> and <class &#39;pyspark.sql.types.LongType&#39;>&quot;
00:22jgauntgot this error when I tried .toDF() method on an RDD
00:25jgauntthis would suggest I need to specify the schema verbosely https://stackoverflow.com/questions/38794522/pandas-dataframe-to-spark-dataframe-can-not-merge-type-error
00:26jgauntI&#39;ve seen Frank do that in his SHIELD+Data+Join notebook
00:26jgaunta bit frustrating not to understand when that&#39;s a necessity and when it&#39;s not
14:13frankhttps://github.com/mozilla/telemetry-dashboard/issues/248
14:18chuttenhttps://github.com/mozilla-services/buildhub
15:34Dextermreid, FYI -> http://reports.telemetry.mozilla.org/post/projects/mainping_beta_latency.kp <- (plots at the end of the page)
15:36frankjgaunt: it&#39;s a bit of magic for spark to infer the schema, basically it&#39;s &quot;we won&#39;t know whether we can infer the schema until we try&quot;
15:51gfritzscheharter: were there any interesting new articles added that we should call out?
15:51gfritzschethats what i was trying to ask :)
16:02gfritzscheRaFromBRC: whats the vidyo room?
16:02gfritzscheor mreid: ^
16:03gfritzsche... ok, rmillers apparently
16:43Fallenbmiroglio: heads up for your queries: https://github.com/mozilla/addons-server/issues/5896
18:50bmiroglioFallen: thanks for the heads up
20:24hartergfritzsche: There are a few coming, but they&#39;re still under review
20:31kitcambridge|sfmreid: hi! i&#39;m trying to run the telemetry-batch-view tests in a docker container, and i see they&#39;re aborted early. any tips? (or if i can just run the sync view tests, i don&#39;t really need to run the entire test suite)
20:31kitcambridge|sfhttps://irccloud.mozilla.com/pastebin/MFxUDYOI/
13 Jul 2017
No messages
   
Last message: 73 days and 21 hours ago