How to read data from a website through its API?

Question

I'm very new to Spark. I need to read data from the website Opensky, using the api they have for it ( https://openskynetwork.github.io/opensky-api/python.html ). The bbox parameter is a tuple of exactly four values (min_latitude, max_latitude, min_longitude, max_latitude). The following code shows the flights registered on certain coordinates:

import json
from random import sample

from opensky_api import OpenSkyApi
api = OpenSkyApi()
states = api.get_states(bbox=(45.8389, 47.8229, 5.9962, 10.5226))

for s in sample(states.states,5):
    flight = {
            'callsign':s.callsign,
            'country': s.origin_country,
            'longitude': s.longitude,
            'latitude': s.latitude,
            'velocity': s.velocity,
            'vertical_rate': s.vertical_rate,
        }
flight_data= json.dumps(flight, indent=2).encode('utf-8')
print("(%r, %r,%r, %r, %r, %r)" % (s.callsign, s.origin_country, s.longitude, s.latitude,s.velocity,s.vertical_rate))

I need to create a python program to be able to send flight information every 10 seconds (through a port that I have assigned). First I have to run the python program with the socket server that reads from Opensky in a terminal, and then I have to run the Spark program with structured streaming in another terminal. I need to send the data and display it by the terminal in json format (using the json.dumps function).

I have the following templates to do it, but I don't know how I should modify them to be able to read the data. The templates are as follows:

Server Socket:

import socket
server = socket.socket()
host = ????
port = ????
server.bind((host, port))
server.listen(2)
client_socket, addr = server.accept()
print("connection established.")

# Sending data
client_socket.sendall("Text".encode())

Spark Structured Streaming:

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode
from pyspark.sql.functions import split

spark = SparkSession \
    .builder \
    .appName("FlightsInformation") \
    .getOrCreate()

    flights= spark \
    .readStream \
    .format("socket") \
    .option("host", "????") \
    .option("port", ????) \
    .load()

flights_information= ????

 query = flight_information\
 .writeStream \
 .outputMode("complete") \
 .format("console") \
 .start()

query.awaitTermination()

How can I do it?

Answer 1

This is how I create the socket to send the JSON data through the socket.

import socket
import sys
import json
from random import sample
from time import sleep

from opensky_api import OpenSkyApi
api = OpenSkyApi()
states = api.get_states(bbox=(45.8389, 47.8229, 5.9962, 10.5226))
# Create a socket (SOCK_STREAM means a TCP socket)
try:
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error as err:
    print('Socket error because of %s' %(err))

try:  
    # Connectar al server
    sock.bind(('127.0.0.1', PORT))
except socket.error as err:
    print('Error, could not bind to server because of %s' %(err))
    sys.exit

sock.listen(2)
client_socket, addr = sock.accept()
print("connection established.")

while True:
    
    for s in sample(states.states, 5):
        vuelo_dict = {
                    'callsign':s.callsign,
                    'country': s.origin_country,
                    'longitude': s.longitude,
                    'latitude': s.latitude,
                    'velocity': s.velocity,
                    'vertical_rate': s.vertical_rate,
                }
        flight_data = json.dumps(vuelo_dict, indent=2).encode('utf-8')
        print("(%r, %r,%r, %r, %r, %r)" % (s.callsign, s.origin_country, s.longitude, s.latitude,s.velocity,s.vertical_rate))

        try:
            client_socket.sendall(flight_data)
            sleep(10)
            #print('Sent: {}'.format(flight_data))

        except socket.gaierror:
            print ('There an error resolving the host')

sock.close()

Spark Structured Streaming:

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode
from pyspark.sql.functions import split

spark = SparkSession \
    .builder \
    .appName("FlightsInformation") \
    .getOrCreate()

flights_information= spark \
    .readStream \
    .format('socket')\
    .option('host', 'localhost')\
    .option('port', XXXXX)\
    .load()

query = flights_information\
    .writeStream \
    .outputMode("append") \
    .format("console") \
    .start()

query.awaitTermination()