Unable to read terraform variables.tf files into may go program

Question

I am attempting to write a go program that reads in a terraform variables.tf and populates a struct for later manipulation. However, I am getting errors when attempting to "parse" the file. I Am hoping someone can tell me what I am doing wrong:

Code:

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "os"

    "github.com/hashicorp/hcl/v2"
    "github.com/hashicorp/hcl/v2/gohcl"
    "github.com/hashicorp/hcl/v2/hclsyntax"
)

type Config struct {
    Upstreams []*TfVariable `hcl:"variable,block"`
}

type TfVariable struct {
    Name string `hcl:",label"`
    // Default     string `hcl:"default,optional"`
    Type        string `hcl:"type"`
    Description string `hcl:"description,attr"`
    // validation block
    Sensitive bool `hcl:"sensitive,optional"`
}

func main() {
    readHCLFile("examples/string.tf")
}

// Exits program by sending error message to standard error and specified error code.
func abort(errorMessage string, exitcode int) {
    fmt.Fprintln(os.Stderr, errorMessage)
    os.Exit(exitcode)
}

func readHCLFile(filePath string) {
    content, err := ioutil.ReadFile(filePath)
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("File contents: %s", content) // TODO: Remove me

    file, diags := hclsyntax.ParseConfig(content, filePath, hcl.Pos{Line: 1, Column: 1})
    if diags.HasErrors() {
        log.Fatal(fmt.Errorf("ParseConfig: %w", diags))
    }

    c := &Config{}
    diags = gohcl.DecodeBody(file.Body, nil, c)
    if diags.HasErrors() {
        log.Fatal(fmt.Errorf("DecodeBody: %w", diags))
    }

    fmt.Println(c) // TODO: Remove me
}

ERROR

File contents: variable "image_id" {
  type        = string
  description = "The id of the machine image (AMI) to use for the server."
  sensitive   = false
}

variable "other_id" {
  type        = string
  description = "The id of the machine image (AMI) to use for the server."
  sensitive   = true
}
2021/03/13 19:55:49 DecodeBody: examples/string.tf:2,17-23: Variables not allowed; Variables may not be used here., and 3 other diagnostic(s)
exit status 1

Stack driver question is sadly for hcl1

Blog post I am referencing.

Answer 1

It looks like it's a bug/feature of the library, since as soon as you change string to "string" , eg,

variable "image_id" {
  type        = string
  ...

to

variable "image_id" {
  type        = "string"
  ...

gohcl.DecodeBody succeeds.

--- UPDATE ---

So, they do use this package in Terraform, BUT they custom-parse configs , ie, they don't use gohcl.DecodeBody . They also custom-treat type attributes by using hcl.ExprAsKeyword ( compare with description ). As you assumed, they do use a custom type for type , but with custom parsing you don't have to.

Below is a working example:

package main

import (
    "fmt"
    "log"
    "os"

    "github.com/hashicorp/hcl/v2"
    "github.com/hashicorp/hcl/v2/gohcl"
    "github.com/hashicorp/hcl/v2/hclsyntax"
)

var (
    configFileSchema = &hcl.BodySchema{
        Blocks: []hcl.BlockHeaderSchema{
            {
                Type:       "variable",
                LabelNames: []string{"name"},
            },
        },
    }

    variableBlockSchema = &hcl.BodySchema{
        Attributes: []hcl.AttributeSchema{
            {
                Name: "description",
            },
            {
                Name: "type",
            },
            {
                Name: "sensitive",
            },
        },
    }
)

type Config struct {
    Variables []*Variable
}

type Variable struct {
    Name        string
    Description string
    Type        string
    Sensitive   bool
}

func main() {
    config := configFromFile("examples/string.tf")
    for _, v := range config.Variables {
        fmt.Printf("%+v\n", v)
    }
}

func configFromFile(filePath string) *Config {
    content, err := os.ReadFile(filePath) // go 1.16
    if err != nil {
        log.Fatal(err)
    }

    file, diags := hclsyntax.ParseConfig(content, filePath, hcl.Pos{Line: 1, Column: 1})
    if diags.HasErrors() {
        log.Fatal("ParseConfig", diags)
    }

    bodyCont, diags := file.Body.Content(configFileSchema)
    if diags.HasErrors() {
        log.Fatal("file content", diags)
    }

    res := &Config{}

    for _, block := range bodyCont.Blocks {
        v := &Variable{
            Name: block.Labels[0],
        }

        blockCont, diags := block.Body.Content(variableBlockSchema)
        if diags.HasErrors() {
            log.Fatal("block content", diags)
        }

        if attr, exists := blockCont.Attributes["description"]; exists {
            diags := gohcl.DecodeExpression(attr.Expr, nil, &v.Description)
            if diags.HasErrors() {
                log.Fatal("description attr", diags)
            }
        }

        if attr, exists := blockCont.Attributes["sensitive"]; exists {
            diags := gohcl.DecodeExpression(attr.Expr, nil, &v.Sensitive)
            if diags.HasErrors() {
                log.Fatal("sensitive attr", diags)
            }
        }

        if attr, exists := blockCont.Attributes["type"]; exists {
            v.Type = hcl.ExprAsKeyword(attr.Expr)
            if v.Type == "" {
                log.Fatal("type attr", "invalid value")
            }
        }

        res.Variables = append(res.Variables, v)
    }
    return res
}

Add for completeness, example/string.tf :

variable "image_id" {
  type        = string
  description = "The id of the machine image (AMI) to use for the server."
  sensitive   = false
}

variable "other_id" {
  type        = string
  description = "The id of the machine image (AMI) to use for the server."
  sensitive   = true
}

Answer 2

Since the Terraform language makes extensive use of various HCL features that require custom programming with the low-level HCL API, the Terraform team maintains a Go library terraform-config-inspect which understands the Terraform language enough to extract static metadata about top-level objects, including variables. It also deals with the fact that Terraform allows variable definitions in any .tf or .tf.json file interleaved with other declarations; putting them in variables.tf is only a convention.

For example:

mod, diags := tfconfig.LoadModule("examples")
if diags.HasErrors() {
    log.Fatalf(diags.Error())
}
for _, variable := range mod.Variables {
    fmt.Printf("%#v\n", variable)
}

This library is the same code used by Terraform Registry to produce the documentation about module input variables , so it supports all Terraform language versions that the Terraform Registry does (at the time of writing, going back to the Terraform v0.10 language, since that's the first version that can install modules from a registry) and supports both the HCL native syntax and JSON representations of the Terraform language.