I have code like this:
import requests
import multiprocessing as mp
import json
import time
BASE_URL = 'http://127.0.0.1:3000/employees'
with open('data.json', 'r') as f:
array = json.load(f)
def internet_resource_getter(post_data):
stuff_got = []
response = requests.post(BASE_URL, json=post_data)
stuff_got.append(response.json())
print(stuff_got)
time.sleep(1)
return stuff_got
if __name__ == '__main__':
# freeze_support() here if program needs to be frozen
start=time.time()
with mp.Pool(mp.cpu_count()) as pool:
pool.map(internet_resource_getter, array)
elapsed = (time.time() - start)
print("\n","time elapsed is :", elapsed)
in file data.json contains 500 records, for example:
[{"first_name":"John","last_name":"Swen"},{"first_name":"Ricard","last_name":"Candra"}]
in BASE_URL there is data like this:
[
{
"id": 1,
"first_name": "Sebastian",
"last_name": "Eschweiler"
},
{
"id": 2,
"first_name": "Steve",
"last_name": "Palmer"
},
{
"id": 3,
"first_name": "Ann",
"last_name": "Smith"
}
]
expected output after POST API:
[
{
"id": 1,
"first_name": "Sebastian",
"last_name": "Eschweiler"
},
{
"id": 2,
"first_name": "Steve",
"last_name": "Palmer"
},
{
"id": 3,
"first_name": "Ann",
"last_name": "Smith"
},
{
"id": 4,
"first_name": "John",
"last_name": "Swen"
},
{
"id": 5,
"first_name": "Ricard",
"last_name": "Candra"
},
]
with my code above, the data that enters the url is only 420 records, even though my data.json is 500 records. how do I solve this so that I post 500 records to url. I don't know why only 400 data are processed. I have an error like this:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1347, in getresponse
response.begin()
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 307, in begin
version, status, reason = self._read_status()
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\util\retry.py", line 531, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\packages\six.py", line 734, in reraise
raise value.with_traceback(tb)
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 1347, in getresponse
response.begin()
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 307, in begin
version, status, reason = self._read_status()
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\http\client.py", line 276, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "paralel_pro.py", line 28, in <module>
pool.map(internet_resource_getter, array)
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\multiprocessing\pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "C:\Users\admin\AppData\Local\Programs\Python\Python38\lib\multiprocessing\pool.py", line 771, in get
raise self._value
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
To get your answer, check your server logs. I suspect bad data and/or server not handling unexpected data gracefully.
See if the 80 records are always the same 80 records.
Can you find the logs for your server that is running on http://127.0.0.1:3000/employees
? It looks like it is running on your same machine. The server's application or service logs should have the smoking gun.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.