Google App Engine and Upload script

Question

I'm currently using Eclipse that came with Android Development bundle and following the GAE tutorial:

https://cloud.google.com/developers/articles/how-to-build-mobile-app-with-app-engine-backend-tutorial

I never had any server or backend experiences and I am hoping to make a Android application using features of the App Engine.

Everything seems to be going smoothly following the tutorial until I got to the part where you need to "Create an upload script". I have no idea how I should proceed the more I read into it the more confused I am.

The tutorial said to "Create a directory and make it your current directory" then create a script file but am I suppose to make the new directory and put it in Eclipse's AppEngine project in the package explorer or somewhere else? I don't really know how does all of this work together.

I just got python 2.7 installed for Eclipse and made sure that the PyDev interpreter is pointing to python.exe but besides that I don't really know where to go from that point on.

EDIT1: I am using Cygwin and run "sh upload_data.shPlace". And when I run it I get the following error:

C:\Python27\python.exe: can't open file '/cygdrive/c/Program Files (x86)/Google/google_appengine/appcfg.py': [Errno 2] No such file or directory

I tried adding "C:\\Program Files (x86)\\Google\\google_appengine\\appcfg.py" to the upload.sh file. It looked like the following:

C:\Program Files (x86)\Google\google_appengine\appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e myemail@mail.com

The following error occurs:

upload_data.sh: line 2: syntax error near unexpected token `('
upload_data.sh: line 2: `/cygdrive/c/Program Files (x86)/Google/google_appengine\appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e mymail@mail.com'

EDIT2: Ok I added '\\' to before the '(' and used '\\' for spaces as well. The following is code in upload.sh right now.

C:\Program\Files\(x86\)\Google\google_appengine\appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e mymail@mail.com

Then the following error it came up with the following error:

upload_data.sh: line 2: C:ProgramFiles(x86)Googlegoogle_appengineappcfg.py: command not found

I went to the google app engine directory and the appcfg.py is there. Additionally I am not too sure if I am actually using the right command. I used:

sh upload.data.sh (arrow bracket)places.csv(arrow bracket)Place

Does the scripts or anything needs to be placed in certain directory? I've tried putting the appcfg.py file in my scripts directory where I keep the upload_data.sh file just said [Errno 2] no such file or directory. Then I tried put the files, upload.data, places.csv and bulkloader.yaml into Google/google_appengine directory where appcfg.py is stored then it will return a "Permission Denied" error. I also re-installed python27 into "google_appengine" file. I tried just running "appcfg.py" on Cygwin and just return [Err2] No such file or directory.

Then I tried running running on Windows command prompt writing in:

appcfg.py upload_data --config_file
C:\Users\User\Documents\AndroidApps\Scripts\bulkloader.yaml--url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e mymail@mail.com

The following error gets returned:

11:09 PM Uploading data records.
Traceback (most recent call last):
  File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 126, in
<module>
    run_file(__file__, globals())
  File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 122, in
run_file
    execfile(_PATHS.script_file(script_name), globals_)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 4934, in <module>
    main(sys.argv)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 4925, in main
    result = AppCfgApp(argv).Run()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 2648, in Run
    self.action(self)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 4605, in __call__
    return method()
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 4417, in PerformUpload
    run_fn(args)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\ap
pcfg.py", line 4298, in RunBulkloader
    sys.exit(bulkloader.Run(arg_dict))
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\bu
lkloader.py", line 4406, in Run
    SetupLogging(arg_dict)
  File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\bu
lkloader.py", line 4362, in SetupLogging
    file_handler = logging.FileHandler(log_file, 'w')
  File "C:\Program Files (x86)\Google\google_appengine\lib\logging\__init__.py",
 line 889, in __init__
    StreamHandler.__init__(self, self._open())
  File "C:\Program Files (x86)\Google\google_appengine\lib\logging\__init__.py",
 line 908, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: 'C:\\Program Files (x86)\\Google\\google_
appengine\\bulkloader-log-20140314.230944'

It seems like I can't use appcfg.py in Cygwin at all. I've been stuck for almost 2 days now.

Thanks.

Answer 1

The linked tutorial does not say so, but the "Create Upload Script" section seems to be limited to uploading data from a .csv file to the development server. My interpretation is that the directory can be anywhere and has not relation to other projects in the tutorial.

When the tutorial says "Create a new directory and make it your current directory" this implies opening a command shell in a text console. The tutorial assumes you are using a Unix type system, not Windows. Imagine the directory will be called mobile-assistant-upload and that you use vi as your text editor, then your shell commands would be:

mkdir mobile-assistant-data
cd mobile-assistant-data
vi upload_data.sh

Note that all the text from appcfg.py to the email address must all be in one long line, and that you should use your own email address.

EDIT: I think line 2 of upload_data.sh should look more like:

/cygdrive/c/Program\ Files\ \(x86\)/Google/google_appengine\appcfg.py --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e mymail@mail.com

and the command line to run it:

sh upload.data.sh places.csv Place

without arrow brackets, they were just documentation. Of course this assumes that places.csv is in the current directory.

Answer 2

I finally got this working by running windows command prompt in administrator mode. I then navigated to google_appengine directory. I then used the following command in Windows cmd:

appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename places.csv --kind=Place -e mymail@mail.com

I also placed the the bulkloader.yaml and places.csv file in google_appengine directory before running the command. It will ask for the password simply press enter then the file should upload to the localhost.

Answer 3

Ran into the same problem.
The tutorial is not very well written at this particular point. I did the following on an Ubuntu system

• Download google_appengine for Python https://developers.google.com/appengine/downloads
• Extract the folder somewhere
• Copy the places.csv and bulkloader.yaml from MobileAssistant-Data to the folder
• Create the file upload_data.sh file in that folder

• Write the following two lines and save the file

   #!/bin/sh ./appcfg.py upload_data --config_file bulkloader.yaml --url=http://localhost:8888/remote_api --filename $1 --kind=$2 -e nobody@nowhere.com 

• Make the file executable chmod a+x upload_data.sh

If all wen't well, running:

<google_appengine folder>./upload_data.sh places.csv Place

Should output something like:

12:55 PM Uploading data records.
[INFO    ] Logging to bulkloader-log-20140918.125544
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Password for nobody@nowhere.com: 
[INFO    ] Opening database: bulkloader-progress-20140918.125544.sql3
[INFO    ] Connecting to localhost:8888/remote_api
[INFO    ] Starting import; maximum 10 entities per post
.
[INFO    ] 2 entities total, 0 previously transferred
[INFO    ] 2 entities (1696 bytes) transferred in 58.8 seconds
[INFO    ] All entities successfully transferred

Hope it helps!