Bigquery multiple join using 子句

Question

I need to get the BGP AS details for the IP addresses in a table, the table contains SrcAddr and DstAddr as mentioned in table1

table1

SrcAddr	DstAddr	Bytes
1.1.1.1	8.8.8.8	1005

Table2 contains the BGP as number details.

Table2

IPaddr	Organization	network_bin	mask
1.1.1.0/24	Cloudflare	asdjqowiq	24
8.8.8.0/24	Google	asdqwrqsd	24

I want to build a final table like below

Table3

SrcAddr	SrcAS	DstAddr	Dst AS	Bytes
1.1.1.1	Cloudflare	8.8.8.8	Google	1005

I used the below query by referring to the doc https://cloudplatform.googleblog.com/2014/03/geoip-geolocation-with-google-bigquery.html and was able to get the src_as field but was not able to resolve the dst_as. can someone help me with this?

WITH source_of_ip_addresses AS (
  SELECT  SamplerAddress, REGEXP_REPLACE(SrcAddr, 'xxx', '0')  srcip, REGEXP_REPLACE(DstAddr, 'xxx', '0')  dstip
  FROM `fluentd.netflow_message`
  WHERE SrcAddr IS NOT null 
  GROUP BY 1,2,3
)

SELECT *, srcip, src_as,
FROM (
  SELECT srcip, network_bin, mask, autonomous_system_organization as src_as
  FROM (
    SELECT *, NET.SAFE_IP_FROM_STRING(source_of_ip_addresses.srcip) & NET.IP_NET_MASK(4, mask) network_bin ,
    FROM source_of_ip_addresses, UNNEST(GENERATE_ARRAY(9,32)) mask
    WHERE BYTE_LENGTH(NET.SAFE_IP_FROM_STRING(srcip)) = 4
  )
  JOIN `fluentd.asn_block_processed`  USING (network_bin, mask) 

Answer 1

just repeat the same process. Also it is more convenient to use WITH clause instead of nested queries to make it simpler to repeat this code. Something like below. I obviously don't have access to your tables, so cannot check syntax, there will likely be duplicate columns you'll need to remove by using explicit column names rather than * .

WITH source_of_ip_addresses AS (
  SELECT  SamplerAddress, REGEXP_REPLACE(SrcAddr, 'xxx', '0')  srcip, REGEXP_REPLACE(DstAddr, 'xxx', '0')  dstip
  FROM `fluentd.netflow_message`
  WHERE SrcAddr IS NOT null 
  GROUP BY 1,2,3
), source_with_masks AS (
  SELECT *, NET.SAFE_IP_FROM_STRING(source_of_ip_addresses.srcip) & NET.IP_NET_MASK(4, mask) network_bin ,
  FROM source_of_ip_addresses, UNNEST(GENERATE_ARRAY(9,32)) mask
  WHERE BYTE_LENGTH(NET.SAFE_IP_FROM_STRING(srcip)) = 4
), source_processed AS (
  SELECT *
  FROM source_with_masks
  JOIN `fluentd.asn_block_processed`  USING (network_bin, mask)
), dest_with_masks AS (
  -- same as above, with dstip instead of srcip
  SELECT *, NET.SAFE_IP_FROM_STRING(source_of_ip_addresses.dstip) & NET.IP_NET_MASK(4, mask) network_bin ,
  FROM source_processed, UNNEST(GENERATE_ARRAY(9,32)) mask
  WHERE BYTE_LENGTH(NET.SAFE_IP_FROM_STRING(srcip)) = 4
), dest_processed AS (
  SELECT *
  FROM dest_with_masks
  JOIN `fluentd.asn_block_processed`  USING (network_bin, mask)
)
SELECT * from dest_processed