How to properly serialize and deserialize paging_size in Python?

Question

In my Python application, I make the query to the Cassandra database. I'm trying to implement pagination through the cassandra-driver package. As you can see from the code below, paging_state returns the bytes data type. I can convert this value to the string data type. Then I send the value of the str_paging_state variable to the client. If this client sends me str_paging_state again I want to use it in my query.

This part of code works:

query = "select * from users where user_type = 'clients';"
statement = SimpleStatement(query, fetch_size=10)
results = session.execute(statement)

paging_state = results.paging_state
print(type(paging_state)) # <class 'bytes'>

str_paging_state = str(paging_state)
print(str_paging_state) # "b'\\x00C\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x03_hk\\x00\\x00\\x00\\x11P]5C#\\x8bGD~\\x8b\\xc7g\\xda\\xe5rH\\xb0\\x00\\x00\\x00\\x03_rk\\x00\\x00\\x00\\x18\\xee\\x14\\xf7\\x83\\x84\\x00tTmw[\\x00\\xec\\xdb\\x9b\\xa9\\xfd\\x00\\xb9\\xff\\xff\\xff\\xff\\xfe\\x01\\x00'"

This part of code raise error:

results = session.execute(
    statement,
    paging_state=bytes(str_paging_state.encode())
)

Error :

[ERROR] NoHostAvailable: ('Unable to complete the operation against any hosts')
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 51, in lambda_handler
    results = cassandra_connection.execute(statement, paging_state=bytes(paging_state.encode()))
  File "/opt/python/lib/python3.8/site-packages/cassandra/cluster.py", line 2618, in execute
    return self.execute_async(query, parameters, trace, custom_payload, timeout, execution_profile, paging_state, host, execute_as).result()
  File "/opt/python/lib/python3.8/site-packages/cassandra/cluster.py", line 4877, in result
    raise self._final_exceptionEND RequestId: 4b7bf588-a2d2-45e5-ad7e-8611f1704313

In Java documentation I found the .fromString method which creates a PagingState object from a string previously generated with toString() . Unfortunately, I didn't find an equivalent for this method in Python.

I also tried to use codecs package to decode and encode the paging_state .

str_paging_state = codecs.decode(paging_state, encoding='utf-8', errors='ignore')
# "\u0000C\u0000\u0000\u0000\u0002\u0000\u0000\u0000\u0003_hk\u0000\u0000\u0000\u0011P]5C#GD~grH\u0000\u0000\u0000\u0003_rk\u0000\u0000\u0000\u0018\u0014\u0000tTmw[\u0000ۛ\u0000\u0001\u0000"

# Raise error
results = session.execute(statement, paging_state=codecs.encode(str_paging_state, encoding='utf-8', errors='ignore'))

In this case I see next error :

[ERROR] ProtocolException: <Error from server: code=000a [Protocol error] message="Invalid value for the paging state">
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 50, in lambda_handler
    results = cassandra_connection.execute(
  File "/opt/python/lib/python3.8/site-packages/cassandra/cluster.py", line 2618, in execute
    return self.execute_async(query, parameters, trace, custom_payload, timeout, execution_profile, paging_state, host, execute_as).result()
  File "/opt/python/lib/python3.8/site-packages/cassandra/cluster.py", line 4877, in result
    raise self._final_exceptionEND RequestId: 979f098a-a566-4904-821a-2ce06522d909

In my case, protocol version is 4.

cluster = Cluster(..., protocol_version=4)

I would appreciate any help!

Answer 1

In my Python application, I make the query to the Cassandra database. I'm trying to implement pagination through the cassandra-driver package. As you can see from the code below, paging_state returns the bytes data type. I can convert this value to the string data type. Then I send the value of the str_paging_state variable to the client. If this client sends me str_paging_state again I want to use it in my query.

This part of code works:

query = "select * from users where user_type = 'clients';"
statement = SimpleStatement(query, fetch_size=10)
results = session.execute(statement)

paging_state = results.paging_state
print(type(paging_state)) # <class 'bytes'>

str_paging_state = str(paging_state)
print(str_paging_state) # "b'\\x00C\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x03_hk\\x00\\x00\\x00\\x11P]5C#\\x8bGD~\\x8b\\xc7g\\xda\\xe5rH\\xb0\\x00\\x00\\x00\\x03_rk\\x00\\x00\\x00\\x18\\xee\\x14\\xf7\\x83\\x84\\x00tTmw[\\x00\\xec\\xdb\\x9b\\xa9\\xfd\\x00\\xb9\\xff\\xff\\xff\\xff\\xfe\\x01\\x00'"

This part of code raise error:

results = session.execute(
    statement,
    paging_state=bytes(str_paging_state.encode())
)

Error :

[ERROR] NoHostAvailable: ('Unable to complete the operation against any hosts')
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 51, in lambda_handler
    results = cassandra_connection.execute(statement, paging_state=bytes(paging_state.encode()))
  File "/opt/python/lib/python3.8/site-packages/cassandra/cluster.py", line 2618, in execute
    return self.execute_async(query, parameters, trace, custom_payload, timeout, execution_profile, paging_state, host, execute_as).result()
  File "/opt/python/lib/python3.8/site-packages/cassandra/cluster.py", line 4877, in result
    raise self._final_exceptionEND RequestId: 4b7bf588-a2d2-45e5-ad7e-8611f1704313

In Java documentation I found the .fromString method which creates a PagingState object from a string previously generated with toString() . Unfortunately, I didn't find an equivalent for this method in Python.

I also tried to use codecs package to decode and encode the paging_state .

str_paging_state = codecs.decode(paging_state, encoding='utf-8', errors='ignore')
# "\u0000C\u0000\u0000\u0000\u0002\u0000\u0000\u0000\u0003_hk\u0000\u0000\u0000\u0011P]5C#GD~grH\u0000\u0000\u0000\u0003_rk\u0000\u0000\u0000\u0018\u0014\u0000tTmw[\u0000ۛ\u0000\u0001\u0000"

# Raise error
results = session.execute(statement, paging_state=codecs.encode(str_paging_state, encoding='utf-8', errors='ignore'))

In this case I see next error :

[ERROR] ProtocolException: <Error from server: code=000a [Protocol error] message="Invalid value for the paging state">
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 50, in lambda_handler
    results = cassandra_connection.execute(
  File "/opt/python/lib/python3.8/site-packages/cassandra/cluster.py", line 2618, in execute
    return self.execute_async(query, parameters, trace, custom_payload, timeout, execution_profile, paging_state, host, execute_as).result()
  File "/opt/python/lib/python3.8/site-packages/cassandra/cluster.py", line 4877, in result
    raise self._final_exceptionEND RequestId: 979f098a-a566-4904-821a-2ce06522d909

In my case, protocol version is 4.

cluster = Cluster(..., protocol_version=4)

I would appreciate any help!