how to send encoded data from python to nodejs

Question

I want to send some string data from Python3 to nodeJs . The string is Korean characters and I am encoding it to utf8 .(Cause I don't know other ways sending data safely.) When I send(from python) it is ByteStream and in nodeJs I receive it as Array. I convert this Array to String. But now I cannot decode string back to original Korean characters. Here are some codes I am using.

python

input = sys.argv[1]
d = bot.get_response(input)
data = str(d).encode('utf8')

print(data)

nodeJs

var utf = require('utf8');
var python = require('python-shell');

var pyt = path.normalize('path/to/my/python.exe'),
        scrp = path.normalize('path/to/my/scriptsFolder/'),


    var options = {
        mode: 'text',
        pythonPath: pyt,
        pythonOptions: ['-u'],
        scriptPath: scrp,
        encoding: 'utf8',
        args: [message]
    };

    python.run('test.py', options, function (err, results) {

        //here I need to decode 'results'
        var originalString = utf.encode(results.toString());// that code is not working for me

    });

I have used several libs like utf8 to decode but didn't help. Can someone please give some idea how to make it work.

EDIT

I have to edit with some more info. I have tried @smarx approach but did not work.

I have two cases:

1. if I send data as string from python here is what I get in nodeJs b'\\xec\\x95\\x88\\xeb\\x85\\x95\\xed\\x95\\x98\\xec\\x8b\\xad\\xeb\\x8b\\x88\\xea\\xb9\\x8c? \\xec\\x9d\\xb4\\xed\\x9a\\xa8\\xec\\xa2\\x85 \\xea\\xb3\\xa0\\xea\\xb0\\x9d\\xeb\\x8b\\x98! \\xeb\\x8f\\x99\\xec\\x96\\x91\\xeb\\xa7\\xa4\\xec\\xa7\\x81\\xec\\x9e\\x85\\xeb\\x8b\\x88\\xeb\\x8b\\xa4 b'\\xec\\x95\\x88\\xeb\\x85\\x95\\xed\\x95\\x98\\xec\\x8b\\xad\\xeb\\x8b\\x88\\xea\\xb9\\x8c? \\xec\\x9d\\xb4\\xed\\x9a\\xa8\\xec\\xa2\\x85 \\xea\\xb3\\xa0\\xea\\xb0\\x9d\\xeb\\x8b\\x98! \\xeb\\x8f\\x99\\xec\\x96\\x91\\xeb\\xa7\\xa4\\xec\\xa7\\x81\\xec\\x9e\\x85\\xeb\\x8b\\x88\\xeb\\x8b\\xa4

2. if I encode data and send. I get ȳ Ͻʴϱ ? ȿ ! ȳ Ͻʴϱ ? ȿ !

Answer 1

I'm still not sure what python.run does, since you won't share that code, but here's my version of the code, which is working fine:

test.py

print("안녕 세상")

app.js

const { exec } = require('child_process');

exec('python3 test.py', function (err, stdout, stderr) {
    console.log(stdout);
});

// Output:
// 안녕 세상

Answer 2

I had the totally same issue on my project and now I finally found the answer.

I solved my problem by using these codes.

It works on windows (macOS and Linux, their default system encoding it 'utf8' so the issue doesn't happen).

I hope it might help you, too!

#in the python file that your javascript file will call by python-shell module put those code
import sys    
sys.stdout.reconfigure(encoding='utf-8')

I found the hints from python-shell description.

feature >Simple and efficient data transfers through stdin and stdout streams

Answer 3

I have the same issue when using python-shell .

Here is my solution:

The string after .encode('utf-8') is a binary string. So you need to print it on stdout directly.

in test.py , it print a utf-8 json which include some chinese char:

sys.stdout.buffer.write(json.dumps({"你好":"世界"}, ensure_ascii=False).encode('utf8'))
print() # print \n at ending to support python-shell in json mode

in main.js

let opt = {mode: 'json', pythonOptions: ['-u'], pythonPath: 'python', encoding: 'utf8'}

let pyshell = new PythonShell('lyric.py', opt);
pyshell.on('message', function (message) {

    console.log(message);  //*** The console msg may still wrong (still ���)
    let json = JSON.stringify(message);
    let fs = require('fs');
    fs.writeFile('myjsonfile.json', json, 'utf8', function () {
    }); //*** The output json file will be correct utf8 output
});

result: This shows the msg is correctly receive in utf-8, because the json output is correct. However console.log output apparently failed. I don't know is there any way to fix console.log output. (Windows 10)

Answer 4

I had same trouble in using data(string) from python in node js.

I solved this problem in this way:

Try change default code page of Windows Console to UTF-8 , if your code page of Windows Console is not UTF-8. (In my case default code page was CP949.)

In my case:

I got message like 2 .

I tried encoding on online ( http://code.cside.com/3rdpage/us/url/converter.html )

then I found my strings encoded cp949 -> decoded utf-8.