Automating Tasks through a VNC Connection within a Browser

Question

So first off, overall what I'm trying to accomplish is for a base machine (as in a VPS) to run automated task through Firefox using Python.

Now the object or goal is to have Firefox run the given tasks in the browser itself, though then connect to a VPS (through the browser) using a VNC connection, and control or issue tasks as well to that VPS (this is the part I'm having trouble with); with also as little memory required as possible for maximum efficiency.

To give an example, if you've used Digital Ocean, you can view your VPS's specific screen or terminal within the current browser.

To be clear, the VPS OS I'm using to run the base process is Linux, though the VPS that the program is connecting to (through the browser) is using a Windows OS. Something such as this let's say (note I didn't screenshot this):

My problem lies with that after running through all of the scripted tasks using Selenium in Python (with Firefox), once I open up the VPS in the browser, I can't figure out how to access it properly or issue jobs to be completed.

I've thought about maybe using (x,y) coordinates for mouse clicks, though I can't say this would exactly work (I tested it with iMacros, though not yet Selenium).

So in a nutshell, I'm running base tasks in Firefox to start, and then connecting to a VPS, and finally issuing more tasks to be completed from Firefox to that VPS that's using a Windows OS environment.

Suggestions on how to make this process simpler, more efficient, or further its reliability?

Answer 1

There is a class in java called Robot class which can handle almost all keyboard operation

There is a similer thing present in python gtk.gdk.Display .

Refer below:-

Is there a Python equivalent to Java's AWT Robot class?

Take a screenshot via a python script. [Linux]

OR

Python ctypes keybd_event simulate ctrl+alt+delete

Demo java code:-

 try{   
    Robot robot = new Robot();
    robot.keyPress(KeyEvent.VK_CONTROL);
    robot.keyPress(KeyEvent.VK_ALT);
    robot.keyPress(KeyEvent.VK_DELETE);
    robot.keyRelease(KeyEvent.VK_CONTROL);
    robot.keyRelease(KeyEvent.VK_ALT);
    robot.keyRelease(KeyEvent.VK_DELETE);

 }
 catch(Exception ex)
 {
     System.out.println(ex.getMessage());
 }

Hope it will help you :)