Python Multiprocessing - Netmiko

Question

I really a newbie to python and network automation; I am trying out multiprocessing with Python and netmiko, but haven't successful; the code keeps being executed sequentially per device.

Below is my code and results:

========================

import datetime
from netmiko import ConnectHandler
import threading
from time import time
import multiprocessing
from multiprocessing import Process, Lock


starting_time = time()

def newthread():
    with open('routers.txt', 'r') as devices:
        for line in devices:
            deviceip = line.strip()
            host = {
                'device_type': 'cisco_ios',
                'ip': deviceip,
                'username': 'cisco',
                'password': 'cisco',
                'secret': 'cisco'
            }

            try:
                connection = ConnectHandler(**host)
                print('Trying router', deviceip)
                print('Connection Established to Host:', deviceip)
                connection.enable()
                sendcommand = connection.send_command('sh run | i hostname')
                print(sendcommand)
            except:
                print('Connection Failed to host', deviceip)


threadtask = Process(target=newthread)
threadtask.start()
threadtask.join()

print('Time Elaspsed:', time() - starting_time)


====Result===
Trying router 10.10.32.2
Connection Established to Host: 10.10.32.2
hostname R1
Trying router 10.10.32.3
Connection Established to Host: 10.10.32.3
hostname R2
Trying router 10.10.32.4
Connection Established to Host: 10.10.32.4
hostname R4
Trying router 10.10.32.5
Connection Established to Host: 10.10.32.5
hostname R3
Time Elaspsed: 26.788068771362305

Process finished with exit code 0

What could i be doing wrong? am kinda stuck. Thank You.

__ Regards Desmon K

Answer 1

Use concurrent.futures inbuilt module. It provides high level API's to execute tasks asynchronously.

https://docs.python.org/3/library/concurrent.futures.html

Below is modified code. Hope it helps.

import time
import concurrent.futures
from netmiko import ConnectHandler

hosts_info = []
with open('routers.txt', 'r') as devices:
    for line in devices:
        deviceip = line.strip()
        host = {
            'device_type': 'cisco_ios',
            'ip': deviceip,
            'username': 'cisco',
            'password': 'cisco',
            'secret': 'cisco'
        }
        hosts_info.append(host)

starting_time = time.perf_counter()

def open_connection(host):
    try:
        connection = ConnectHandler(**host)

        print('Trying router', host['ip'])
        print('Connection Established to Host:', host['ip'])
        connection.enable()
        sendcommand = connection.send_command('sh run | i hostname')
        return sendcommand
    except:
        print('Connection Failed to host', host['ip'])


with concurrent.futures.ProcessPoolExecutor() as executor:
    results = executor.map(open_connection, hosts_info)

    for result in results:
        print(result)

finish = time.perf_counter()
print('Time Elapsed:', finish - starting_time)

Answer 2

After following @Sandeep's code i edited a few things and finally achieved the primary goal;

The final working code is here

import time
import concurrent.futures
from netmiko import ConnectHandler

hosts_info = []

starting_time = time.perf_counter()

with open('routers.txt', 'r') as devices:
    for line in devices:
        deviceip = line.strip()
        host = {
            'device_type': 'cisco_ios',
            'ip': deviceip,
            'username': 'cisco',
            'password': 'cisco',
            'secret': 'cisco'
        }
        hosts_info.append(host)

def open_connection(host):
    try:
        connection = ConnectHandler(**host)
        print('Connection Established to Host:', host['ip'])
        connection.enable()
        sendcommand = connection.send_command('sh run | i hostname')
        return sendcommand + ' ' + 'For Device:' + host['ip']
    except:
        print('Connection Failed to host', host['ip'])


if __name__ == '__main__':
    with concurrent.futures.ProcessPoolExecutor() as executor:
        results = executor.map(open_connection, hosts_info)
        for result in results:
            print(result)

    finish = time.perf_counter()
    print('Time Elapsed:', finish - starting_time)

Output:

Connection Established to Host: 10.10.32.2
Connection Established to Host: 10.10.32.4
Connection Established to Host: 10.10.32.5
Connection Established to Host: 10.10.32.3
hostname R1 For Device:10.10.32.2
hostname R2 For Device:10.10.32.3
hostname R4 For Device:10.10.32.4
hostname R3 For Device:10.10.32.5
Time Elapsed: 7.035125793000589

Please note that ip is supposed to be in quotes as 'ip'. Output is not that Fancy, but this will do for now. Thank you @ https://stackoverflow.com/users/4087758/sandeep-nagendra

Answer 3

确保在 BIOS 中启用了超线程设置/多核支持。