Use SELECT AS alias in following CASE WHEN statement - Spark SQL

Question

It seems I am not able to do the following, as it will not recognize the StageCodeTableB valuable in the CASE logic. But it's weird, because I was able to do it once in a different exercise. Is there any way to access the StageCodeTableB variable in the CASE logic?

You could just refer to tableB.stageCode in the CASE logic instead but then if you did some transformations in the SELECT stage, you would need rewrite those again and again in each WHEN statement ...

%sql
DROP TABLE IF EXISTS database.tableA;

CREATE TABLE database.tableA PARQUET AS
  SELECT 
    CAST(tableB.stageCode AS STRING) AS StageCodeTableB,
    CASE
      WHEN StageCodeTableB = '33' THEN 1
      ELSE NULL
      END AS StageCodeTableBSCORE
  FROM database.tableB AS tableB

Answer 1

SparkSQL supports Common Table Expressions (CTEs) even with CTAS (CREATE TABLE AS) so you can use them together. A simple example;

%sql
DROP TABLE IF EXISTS sparkDb.tableA; 

CREATE TABLE IF NOT EXISTS sparkDb.tableA USING PARQUET 
AS
WITH cte AS  (
SELECT
  stageCode,
  CAST(tableB.stageCode AS STRING) AS StageCodeTableB
FROM sparkDb.tableB AS tableB
)
SELECT *,
  CASE
    WHEN StageCodeTableB = '33' THEN 1
    ELSE NULL
  END AS StageCodeTableBSCORE
FROM cte;

SELECT * FROM tableA;