NLTK数据可以安装在AWS Redshift环境中吗？

Question

I am trying to create a Python user defined scalar (UDF) function in an AWS Redshift DB. The UDF wraps the following Python code:

CREATE or replace library nltk language plpythonu from 's3://xxx/dev/python-libraries/nltk-3.2.1.zip'
credentials 'aws_access_key_id=xxx;aws_secret_access_key=yyy' region as 'eu-west-1';

CREATE or replace library textblob language plpythonu from 's3://xxx/dev/python-libraries/textblob-0.15.1-py2.py3-none-any.zip'
credentials 'aws_access_key_id=xxx;aws_secret_access_key=yyy' region as 'eu-west-1';

CREATE or replace FUNCTION f_sentiment_polarity (comment varchar(1000)) RETURNS float IMMUTABLE as $$
from textblob import TextBlob
return TextBlob(comment).sentiment.polarity
$$ LANGUAGE plpythonu;

SELECT f_sentiment_polarity('this would be very useful if the corpora were loaded');

f_sentiment_polarity
--------------------
                   0

The result of the select statement gives me 0

When I run the same Python code in a local environment (Python 2.7 on Windows with NLTK v3.2.5, I get 0.39 :

Python 2.7.10 (default, May 23 2015, 09:44:00) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from textblob import TextBlob
>>> TextBlob('this would be very useful if the corpora were loaded').sentiment.polarity
0.39
>>>

I presume that this is because the various NLTK Corpora have not been loaded in the AWS Redshift Python environment. Creating another Redshift UDF as follows seems to bear this out:

CREATE or replace FUNCTION f_num_brown_words () RETURNS int IMMUTABLE as $$
from nltk.corpus import brown
return len(brown.words())
$$ LANGUAGE plpythonu;

select f_num_brown_words();

ERROR: XX000: LookupError: 
**********************************************************************
  Resource u'corpora/brown' not found.  Please use the NLTK
  Downloader to obtain the resource:  >>> nltk.download()
  Searched in:
    - "'/'/nltk_data"
    - '/usr/shar

Question: Is there a way of loading the NLTK Corpora in the AWS Redshift Python environment so that my UDF will function correctly?

Answer 1

You can load custom libraries onto your cluster, more info in the official docs .

I followed the instructions and it worked for me with another library.