psql loop in shell script

Question

I need to read from a file.csv 2 values and make more then 13.000 queries in a PostgreSQL.

It should be very simple task as you may see in the code above, but there are some issues.

#!/bin/bash


MSISDN=($(head file.csv | awk -F ";" '{print $1}' | sed -e "s/^/55/"))
APPID=($(head file.csv | awk -F ";" '{print $3}'))

NUMBER_OF_LINES=$(wc -l file.csv| grep -o "[0-9]*")

for i in $(seq 0 "$NUMBER_OF_LINES")
do

    export PGPASSWORD='MY_PASSWORD'
    psql -q -A -h VERY-LONG-HOST -U MYUSER -d DATABASE -p 1111 -t -c "select 'http://API-HOST/subscription/cancel?subscriptionId=' + s.subscription_id + '&phone=' + s.phone + '&enabled=0&statusId=7&notifyActionListeners=false&extraInfo=TICKET_NUMBER' from sbs.subscription s (nolock) join sbs.configuration c on s.configuration_id = c.configuration_id where c.application_id = ${APPID[$i]} and c.carrier_id = 2 and s.phone = ${MSISDN[$i]};"

done

when the code executes there is a error:

 ERROR: syntax error at or near "112940676229" LINE 2: and c.carrier_id = 2 and s.phone = 55112940676229; 

How can I work with multiples queries and close the connection in each of them before make another query, thus far how to solve the error showed before?

Example of the content in file.csv:

112940676229;Sevice;333                                               
113429402012;Sevice;929  
111429402013;Sevice;888
11240672940;Sevice;445
11320294034;Sevice;333
11429294056;Sevice;22
11942940281;Sevice;122
11962940895;Sevice;233

Answer 1

I would recommend you trying such approach:

create table file_csv (tel bigint, dk text, apid int);
copy file_csv from file.csv delimiter ';';

and then just join against it in your query.

update Thanks to @Charles Duffy

Might make it a temporary table ( create temporary table file_csv (tel bigint, dk text, apid int) ), if the intent is to be transient.

Answer 2

For both security and performance, I strongly advise using a different language for this. That said, to answer the narrow question:

while IFS=';' read -r msidn _ appid _ <&3; do
  appid=${appid%$'\r'} # fix DOS newlines, should they be present in your input
  psql -q -A \
    -h VERY-LONG-HOST -U MYUSER -d DATABASE -p 1111 -t \
    -c "select 'http://API-HOST/subscription/cancel?subscriptionId=' + s.subscription_id + '&phone=' + s.phone + '&enabled=0&statusId=7&notifyActionListeners=false&extraInfo=TICKET_NUMBER' from sbs.subscription s (nolock) join sbs.configuration c on s.configuration_id = c.configuration_id where c.application_id = $appid and c.carrier_id = 2 and s.phone = $msidn;"
done 3< file.csv

---

That said, a less-awful Python implementation might look like:

#!/usr/bin/env python

import csv
import sys
import psycopg2

query='''
select
  s.subscription_id,
  s.phone
from
  sbs.subscription s (nolock)
  join sbs.configuration c on s.configuration_id = c.configuration_id
where
  c.application_id = %(appid)s
  and c.carrier_id = 2
  and s.phone = %(msidn)s
'''

# user is expected to pass CSV file name as our first argument
filename = sys.argv[1]

# user is expected to pass a PostgreSQL connect string as our second argument
conn = psycopg2.connect(sys.argv[2])

curs = conn.cursor()

with open(filename, 'rb') as csv_file:
  csv_reader = csv.reader(csv_file, delimiter=';')
  for csv_row in csv_reader:
    msidn = csv_row[0]
    appid = csv_row[2]
    curs.execute(query, {"msidn": msidn, "appid": appid})
    for result_row in curs.fetchall():
      sid = result_row[0]
      phone = result_row[1]
      print 'http://API-HOST/subscription/cancel?subscriptionId=' + sid + '&phone=' + phone + '&enabled=0&statusId=7&notifyActionListeners=false&extraInfo=TICKET_NUMBER'