Script works from shell but not Ruby

Question

I have a script that fills in parameters for multiple spark-submit jobs. The jobs work when I copy and paste them into shell, but they fail when I put the string in backticks and execute it from within Ruby.

spark-submit --master yarn-cluster \
  --num-executors 2 \
  --files ... \
  --class ... \
  -otherflags ...

Ruby returns

sh: 1: spark-submit: not found

even though spark-submit is in the system path. Even more confusing is that:

`spark-submit` 

within Ruby seems to call the shell command correctly.

Answer 1

Consider what's different when you are at the command-line in the shell, versus when code is running in Ruby.

The shell has PATH defined, which it walks looking for the command, hopefully eventually finding the desired one.

Ruby doesn't do that, and instead, you need to define the absolute path to the executable. There are various ways of doing it but the simplest is to use

where spark-submit

at the command-line, then copy/paste that into your Ruby script. Most likely you'll end up with something like:

/usr/local/bin/spark-submit

or

/path/to/rubys/gems/.../bin/spark-submit

---

spark-submit

within Ruby seems to call the shell command correctly.

Ruby is trying to find the command in the current directory.