How to download a file with curl -O excluding url parameters?

Question

I am stucked for 2 hours on a simple process because I am not confortable with bash script & regex :(

I would like to download a file using a bash script. The target file is accessible at a url with the following name pattern :

https://domain-name.com/JAWSDB_xxxxxxxxxxxxxxxxx.sql.gz?AWSAccessKeyId=XXXXXXXXXXXXXXXX&Expires=xxxxxx&Signature=%2XXXXXXXXXXXXXXXXXXXX

My first idea was to use curl -O but it creates really dirty filenames including url parameters, so I would like to only keep the following part JAWSDB_xxxxxxxxxxxxxxxxx.sql.gz (and remove trailing url parameters).

I found a regex .+?(?=\\?) that seems to remove the url prams but I don't know how to make it work with the curl -O command.

Thanks a lot.

(Ps: I am developping on osx and plan to run the script on linux)

Answer 1

If you wish to have curl do the heavy lifting, give --remote-header-name a shot which will accept the name of the object from the server, and not extract the file name from the curl dereferenced url.

curl --remote-header-name -O 
 https://codeload.github.com/curl/curl/zip/master?name=value
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4708k    0 4708k    0     0  1677k      0 --:--:--  0:00:02 --:--:-- 1676k
curl: Saved to filename 'curl-master.zip'
ls -lt
total 21304
-rw-r--r--     1 randrews  wheel  4821944 Mar  6 13:56 curl-master.zip

It appears JAWSDB_xxxxxxxxxxxxxxxxx.sql.gz may be parameterized in your script, which if included here will help provide guidance.

If this is the case, then you may redirect the output of curl via -O to a file of your choosing.

curl -O https://domain-name.com/JAWSDB_xxxxxxxxxxxxxxxxx.sql.gz?AWSAccessKeyId=XXXXXXXXXXXXXXXX&Expires=xxxxxx&Signature=%2XXXXXXXXXXXXXXXXXXXX > JAWSDB_xxxxxxxxxxxxxxxxx.sql.gz