SQL 使用 github_repos 数据集在 GCP BigQuery 上查询验证失败

Question

I would like to get a list all unique repositories on GutHub by using the following command:

SELECT DISTINCT repo_name FROM `bigquery-public-data.github_repos.commits`

However I get the following error:

Column repo_name of type ARRAY cannot be used in SELECT DISTINCT at [1:17]

In the schema it says repo_name is of type STRING, what am I doing wrong?

Answer 1

repo_name is defined as a "string" with mode "repeated" in the table schema which roughly means an ARRAY of STRING in BigQuery.

https://cloud.google.com/bigquery/docs/nested-repeated

What does REPEATED field in Google Bigquery mean?

Answer 2

You can use the below query

SELECT 
    commit
   , repo_name 
FROM 
   `bigquery-public-data.github_repos.commits`, 
    UNNEST(repo_name) as repo_name 
WHERE 
    commit = 'c87298e36356ac19519a93dee3dfac8ebffe45e8' 

Which will give a result like below

Row |  commit                                  | repo_name
===================================================================
1   | c87298e36356ac19519a93dee3dfac8ebffe45e8 | noondaysun/sakai
2   | c87298e36356ac19519a93dee3dfac8ebffe45e8 | OpenCollabZA/sakai

Answer 3

As another user posted, in the schema of the bigquery-public-data.github_repos.commits table you can see that the repo_name field is defined as a STRING REPEATED which means that each entry of repo_name is an array constituted by string-type elements. You can see this with the following query:

#standardSQL
SELECT repo_name 
FROM `bigquery-public-data.github_repos.commits` 
LIMIT 100;

In order to find the distinct repo names you can employ the UNNEST operator to expand each one of the repo_name elements. The following query performs a CROSS JOIN that adds a new field repo_name_single to the table constituted by the individual repository names. This way, the DISTINCT function can be employed.

#standardSQL
SELECT DISTINCT(repo_name_unnest) 
FROM `bigquery-public-data.github_repos.commits` 
CROSS JOIN UNNEST(repo_name) AS repo_name_unnest;