Python3 threading, trying to ping multiple IPs/test port simultaineously

Question

Full (non-working) code below

Full (working, w/o threading) code here: http://pastebin.com/KUYzNtT2

I've written a small script that does the following:

1. Pull network information from a database
2. Ping each IP in a cidr (ie - 192.168.0.0/24); if it's up, test to see if a certain port is open
3. Display the results

This is working fine, but I'd like to implement threading to make the script run faster; as is I have thousands of IPs to scan and it takes forever.

I've played around with threading tutorials but can't seem to grasp how to implement it in my script.

Any thoughts or suggestions are appreciated.

EDIT: I went in a different direction based on this guide: http://chriskiehl.com/article/parallelism-in-one-line/

Now I run the program and get: File "port_test.py", line 39, in display_results for (client, location, cidr) in results: ValueError: too many values to unpack (expected 3) and I don't understand why. Thoughts?

**EDIT: I think I figured out why it failed, looks like pool.map expects only one data point. If I only query the DB for CIDRs instead of the other two columns, the program starts spitting out data (MUCH faster than before). So now I need to figure out how to add the other two columns to the results, then sort the results to they make sense (there's no order to the results, which I suppose makes sense)

#! /usr/bin/python
# Import modules
import socket
import subprocess
import ipaddress
import mysql.connector
import configparser
import logging
import coloredlogs
from multiprocessing.dummy import Pool as ThreadPool

#logging.basicConfig(format='%(levelname)s:%(message)s',level=logging.INFO)


coloredlogs.install(level='DEBUG')
coloredlogs.DEFAULT_LOG_FORMAT = '%(asctime)s -- %(message)s'
# read from the config file
config = configparser.ConfigParser()
config.read('config.ini')
db=config['mysql']
net=config['network']
port = int(net['port'])

# create the connection, connect, and setup the query
cnx = mysql.connector.connect(user=db['user'], database=db['database'], password=db['password'])
cursor = cnx.cursor()

query = ("select fw.net_cidr as cidr "
        "from firewalls fw "
            "left join clients c on c.id = fw.client_id "
            "left join locations l on l.id = fw.location_id "
                "where fw.net_cidr <> '' and c.active = '1' and fw.active = '1'")

cursor.execute(query)
results = cursor.fetchall()

def display_results(results):
# execute and display the results
    for (cidr) in results:
            logging.info("{} --> ".format(cidr))
            try:
                # Prompt the user to input a network address
                net_addr = str(cidr)

                # Create the network
                ip_net = ipaddress.ip_network(net_addr)

                 # Get all hosts on that network
                all_hosts = list(ip_net.hosts())
            except ValueError as e:
                logging.warning(e)
                continue

            # For each IP address in the subnet, test to see if port 3389 is open
            for i in range(len(all_hosts)):
                sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
                sock.settimeout(.25)
                result = sock.connect_ex((str(all_hosts[i]),port))
                if result == 0:
                        logging.info(str(all_hosts[i]) + ": " + net['port'] + " is open")
            else:
                    logging.debug(str(all_hosts[i]) + ": " + net['port'] + " is not open")

# make a pool of workers
pool = ThreadPool(4)

# ping the cidrs in their own thread
pool.map(display_results, results)
pool.close()
pool.join()

# close the database connection
cursor.close()
cnx.close()

Answer 1

Grab all the data initially and store it in a Queue .

Create a function that runs continuously until the Queue is empty (ie while my_queue.empty() is False .

Grab the first object in the Queue with Queue 's get() method.

Then process it.

Initialize as many threads as you want, they will execute until the Queue is empty.