AWS Sagemaker java.lang.IllegalArgumentException

Question

I have a SageMaker endpoint. The SageMaker endpoint is being invoked via a Lambda function. Part of that Lambda function code that invokes it is:

SM_ENDPOINT_NAME = "pytorch-inference-2021-xx-xx"
sm_runtime= boto3.client('runtime.sagemaker')
txt = "Canon Compact Photo Printer"
response = sm_runtime.invoke_endpoint(EndpointName=SM_ENDPOINT_NAME, ContentType='text/plain', Body=txt)
print(response)

The response does not get logged. It is supposed to return a vector

When the Lambda function calls the SageMaker endpoint it times out. I started seeing this error suddenly today in the SageMaker Endpoint logs:

java.lang.IllegalArgumentException: reasonPhrase contains one of the following prohibited characters: \r\n: tokenizers>=0.10.1,<0.11 is required for a normal functioning of this module, but found tokenizers==0.11.2.

Try: pip install transformers -U or pip install -e '.[dev]' if you're working with git master

For context, The ML Model is supposed to accept a string input and return a vector (list). All of my code base is in Python3. The Java exception seems to be from the SageMaker Backend.

I have not updated the SageMaker endpoint or the Lambda function in a few months. This seems like it has to be an issue with SageMaker.

Is this an issue I can fix or do I need to contact AWS? If so how can I do that?

Answer 1

Based on your error, it appears you are using v0.11.2 of the tokenizers module, and the message indicates you should use a version between 0.10.1 and 0.11.

Try to uninstall the tokenizers module, and then pull an older release via pip...

pip install 'tokenizers>=0.10.1,<0.11'

Here are all the release versions for reference: https://pypi.org/project/tokenizers/#history

After you update the module, deploy your lambda function again and see if that does the trick.