Now we wait - while BigQuery shows us the progress of our training:Īnd when it’s done, we even get an evaluation of our model:ĭo we really need 4,000 one-hot encoded dimensions to obtain better clusters? Turns out that 500 are enough - and I like the results better. With this line, I’m creating a one-hot encoding string that I can use later to define the 4,000+ columns I’ll use for k-means:įORMAT("IFNULL(ANY_VALUE(IF(tag2='%s',1,null)),0)X%s", tag2, REPLACE(REPLACE(REPLACE(REPLACE(tag2,'-','_'),'.','D'),'#','H'),'+','P'))Īnd training a k-means model in BigQuery is really easy:ĬREATE MODEL `deleting.kmeans_tagsubtag_50_big_a_01` Now - instead of using this small table, let’s use the whole table to compute k-means with BigQuery. ,IFNULL(ANY_VALUE(IF(tag2='jquery',1,null)),0) XjqueryįROM `deleting.stack_overflow_tag_co_ocurrence` ,IFNULL(ANY_VALUE(IF(tag2='android',1,null)),0) Xandroid ,IFNULL(ANY_VALUE(IF(tag2='python',1,null)),0 ) Xpython
,IFNULL(ANY_VALUE(IF(tag2='javascript',1,null)),0) Xjavascript You can reduce or augment the sensibility of these relations with the percent threshold:
#WIN32COM FREEZE MULTIPLE PANES IN EXCEL 2016 PLUS#
So I’ll take these relationships and I’ll save them on an auxiliary table - plus a percentage of how frequently a relationship happens for each tag.ĬREATE OR REPLACE TABLE `deleting.stack_overflow_tag_co_ocurrence`įROM `fh-bigquery.stackoverflow_archive.201906_posts_questions` Let’s find tags that usually go together:Ĭo-occurring tags on Stack Overflow questions ORDER BY 2 DESC Top Stack Overflow tags by number of questions. In this picture I only have 240 tags - how would you group and categorize 4,000+ of them?įROM `fh-bigquery.stackoverflow_archive.201906_posts_questions`,
These are the most active Stack Overflow tags since 2018 - they’re a lot. You can check out more about working with Stack Overflow data and BigQuery here and here. In this post he works with BigQuery – Google’s serverless data warehouse – to run k-means clustering over Stack Overflow’s published dataset, which is refreshed and uploaded to Google’s Cloud once a quarter. Felipe Hoffa is a Developer Advocate for Google Cloud. Visualizing a universe of clustered tags. How would you group more than 4,000 active Stack Overflow tags into meaningful groups? This is a perfect task for unsupervised learning and k-means clustering - and now you can do all this inside BigQuery.