• Introduction
• Format
  • Mouse Events
  • Keyboard Events
  • Special Keyboard Keys
  • LoadFile
  • WaitForIt
  • ClickIt
• Tools
  • Type a string
  • Type a file
  • Replacing mouse movements with merged waits in recordings
  • Printing coordinates as your mouse moves
  • Checking if a smaller picture exists on the screen.
• Example VNC Script

Introduction

VNC recordings are just plain text files with commands.

Format

Files can contain: mouse events, keyboard events, load events, and no actionable lines. Anything that does not have proper syntax or has invalid key definitions will be ignored.

Mouse Events

574598:PointerEvent,0,100,669

574598 – Is the time to wait in nanoseconds before doing an action.

Note: Adding 6 decimal places to microseconds will get you nanoseconds. A, 1 and 9 zeros is a second. In this case 574598 is ~0.575ms.

PointerEvent – Issues either a mouse click or a click release

0 – Is a non clicked state. 1 is a clicked state.

100,669 – Are the x and y mouse coordinates. Where 0,0 is the top left.

When writing mouse events by hand you should surround them with non-click presses as well; that way, the OS can move the mouse to where it needs to be and you can stop clicking.

50000000:PointerEvent,0,100,669 50000000:PointerEvent,1,100,669 50000000:PointerEvent,0,100,669

Keyboard Events

1000000:KeyEvent,true,a 1000000:KeyEvent,false,a 1000000 - Is the time to wait in nanoseconds before doing an action. (1ms)

KeyEvent – Issues a either a keydown or keyup event.

true – Is a key down event, false is a key up event.

Special Keyboard Keys

Every single key is defined in a map in the source code link

Here are most of the ones you will use. Note that these keys are case-sensitive.

space BackSpace Tab Return Super_L Shift_L Control_L Alt_L Escape Delete Left Up Right Down Home End Insert slash backslash underscore bracketleft bracketright braceleft braceright question exclam at numbersign dollar percent ampersand parenleft parenright asterisk apostrophe quotedbl colon semicolon comma period minus grave less equal greater bar 1 a F1

If a key requires a shift on your keyboard you will likely need to do the same regardless of the key’s name. For example, colon:

50000:KeyEvent,true,Shift_L 50000:KeyEvent,true,colon 50000:KeyEvent,false,colon 50000:KeyEvent,false,Shift_L

LoadFile

The playback file supports a LoadFile event. LoadFile will take an existing VNC keyboard/mouse recording and, if a playback is currently running, preempt the running playback and play the specified playback file to completion. After the LoadFile playback completes, the previously running playback will resume and continue playing. If LoadFile is injected with no playback currently running it will start a new VNC playback with the specified playback file.

10000000000:LoadFile,/home/john/recordings/reboot_windows.vnc

You can inject LoadFile events in the same way as other keyboard or mouse events:

# inject the LoadFile event to vm foo to play the playback bar.kbr vnc inject foo LoadFile,bar.kbr  

WaitForIt

Another special event is the WaitForIt event. This event causes the playback to search the VM’s screenshot for the template image and continue once the template image has been found. The event supports a timeout which will cause the playback to stop if exceeded.

1000:WaitForIt,10s,template.png

You may also base64-encode the image and include it in place of the filename. This allows your VNC scripts to be self-contained.

ClickIt

Similar to the WaitForIt event, the ClickIt event waits until a template image appears in the VM screenshot but has an additional action to click on the center of the template image in the screenshot.

1000:ClickIt,10s,template.png

As with the WaitForIt event, you may base64-encode the image to use in place of the filename.

Tools

Creating these files by hand may be exhausting. Here are some scripts to help out.

Type a string

# genKeys.py import sys def vncType(t):  retstr = ""  shift = 0  for c in t:   shift = 0   if c == ".":    c = "period"   if c == " ":    c = "space"   if c == "/":    c = "slash"   if c == "\\":    c = "backslash"   if c == ":":    c = "colon"    shift = 1   if c == "@":    c = "at"    shift = 1   if c == "!":    c = "exclam"    shift = 1   if c == "#":    c = "numbersign"    shift = 1   if c == "$":    c = "dollar"    shift = 1   if c == "%":    c = "percent"    shift = 1   if c == "&":    c = "ampersand"    shift = 1   if c == "\'":    c = "apostrophe"   if c == "(":    c = "parenleft"    shift = 1   if c == ")":    c = "parenright"    shift = 1   if c == "*":    c = "asterisk"    shift = 1   if c == "+":    c = "plus"    shift = 1   if c == ",":    c = "comma"   if c == "-":    c = "minus"   if c == ";":    c = "semicolon"   if c == "<":    c = "less"    shift = 1   if c == ">":    c = "greater"    shift = 1   if c == "?":    c = "question"    shift = 1   if c == "=":    c = "equal"   if c == "[":    c = "bracketleft"   if c == "]":    c = "bracketright"   if c == "{":    shift = 1    c = "braceleft"   if c == "}":    shift = 1    c = "braceright"   if c == "_":    c = "underscore"    shift = 1   if c == "\"":    c = "quotedbl"    shift = 1   if c == "\n":    c = "Return"   if c == "\t":    c = "Tab"   if shift == 1:    retstr = retstr +"50000000:KeyEvent,true,Shift_L\n"   retstr = retstr + "50000000:KeyEvent,true,"+c+"\n50000000:KeyEvent,false,"+c+"\n"   if shift == 1:    retstr = retstr +"50000000:KeyEvent,false,Shift_L\n"  return retstr.rstrip("\n") print vncType(sys.argv[1])

# python genKeys.py "P@ss w0rd" 50000000:KeyEvent,true,P 50000000:KeyEvent,false,P 50000000:KeyEvent,true,Shift_L 50000000:KeyEvent,true,at 50000000:KeyEvent,false,at 50000000:KeyEvent,false,Shift_L 50000000:KeyEvent,true,s 50000000:KeyEvent,false,s 50000000:KeyEvent,true,s 50000000:KeyEvent,false,s 50000000:KeyEvent,true,space 50000000:KeyEvent,false,space 50000000:KeyEvent,true,w 50000000:KeyEvent,false,w 50000000:KeyEvent,true,0 50000000:KeyEvent,false,0 50000000:KeyEvent,true,r 50000000:KeyEvent,false,r 50000000:KeyEvent,true,d 50000000:KeyEvent,false,d

Type a file

Add this to the end of genKeys.py to type a whole file.

with open ("file.txt", "r") as myfile:     data=myfile.readlines() for c in data:     vncType(c)

Replacing mouse movements with merged waits in recordings

#!/usr/bin/python  import sys  if len(sys.argv) != 2:  print "Syntax Error: Expecting\n./shrink.py input_file"  exit(1)  global timer timer = 0 global changetimer changetimer = 1 with open (sys.argv[1], "r") as myfile:  for line in myfile:   if 'PointerEvent,0' in line:    split = line.split(":")    split = split[0]    timer = timer + int(split)   else:    if 'PointerEvent,1' in line or 'PointerEvent,4' in line:     if timer == 0:      print line.rstrip('\n')      split = line.split(",")      print ("10000:PointerEvent,0,"+split[-2]+","+split[-1]),     else:      split = line.split(",")      print (str(timer)+":PointerEvent,0,"+split[-2]+","+split[-1]),      line = line.lstrip('\n')      print line.rstrip('\n')      print ("10001:PointerEvent,0,"+split[-2]+","+split[-1]),    elif 'KeyEvent' in line:     if timer != 0:      print str(timer)+":PointerEvent,0,0,0"      print line.rstrip('\n')     else:      print line.rstrip('\n')    else:     # looks like a comment     print line.rstrip('\n')     changetimer = 0    if changetimer ==1:     timer = 0    else:     changetimer = 1  print str(timer)+":PointerEvent,0,0,0"

Printing coordinates as your mouse moves

nano minimega/misc/web/novnc/include/input.js ctrl+w and search for onMouseMove(pos.x Modify the code to have the nested if/else section added.

var evt = (e ? e : window.event); var pos = Util.getEventPosition(e, this._target, this._scale);     if (this._onMouseMove) {         if (noVNC_status.innerHTML.indexOf('|')==-1){             noVNC_status.innerHTML += " | X:"+pos.x+" Y:"+pos.y;         }         else{             noVNC_status.innerHTML=noVNC_status.innerHTML.split("|")[0]+"| X:"+pos.x+" Y:"+pos.y;         }         this._onMouseMove(pos.x, pos.y);     }

The result should look like this.

Checking if a smaller picture exists on the screen.

Useful for waiting for something before clicking on it.

#apt-get install python-opencv import cv2 smallimgpath="/home/ubuntu/small.png" largeimgpath="/home/ubuntu/large.png" small = cv2.imread(smallimgpath) large = cv2.imread(largeimgpath) method = cv2.TM_CCOEFF_NORMED res = cv2.matchTemplate(large,small,method) min_val,max_val,min_loc,max_loc=cv2.minMaxLoc(res) threshold=0.8 if(max_val < threshold):  print "not found" else:  small_h,small_w,small_k=small.shape  x_center =max_loc[0] +small_w/2  y_center =max_loc[1] +small_h/2  print "Center found: "+str(x_center)+","+str(y_center)

You can take a screenshot of a vm using vm screenshot

vm screenshot myvm file /home/ubuntu/test.png

Example VNC Script

# Click on the screen to disable screensaver 50000000:PointerEvent,0,100,750 50000000:PointerEvent,1,100,750 50000000:PointerEvent,0,100,750 # Wait 10s 10000000000:PointerEvent,0,0,0 # Press Windows+r 50000000:KeyEvent,true,Super_L 50000000:KeyEvent,true,r 50000000:KeyEvent,true,r 50000000:KeyEvent,false,Super_L # Wait 10s 10000000000:PointerEvent,0,0,0 # Type cmd.exe enter 50000000:KeyEvent,true,c 50000000:KeyEvent,true,c 50000000:KeyEvent,true,m 50000000:KeyEvent,true,m 50000000:KeyEvent,true,d 50000000:KeyEvent,true,d 50000000:KeyEvent,true,period 50000000:KeyEvent,true,period 50000000:KeyEvent,true,e 50000000:KeyEvent,true,e 50000000:KeyEvent,true,x 50000000:KeyEvent,true,x 50000000:KeyEvent,true,e 50000000:KeyEvent,true,e 50000000:KeyEvent,true,Return 50000000:KeyEvent,true,Return # Wait 10s 10000000000:PointerEvent,0,0,0 # Alt+F4 50000000:KeyEvent,true,Alt_L 50000000:KeyEvent,true,F4 50000000:KeyEvent,true,F4 50000000:KeyEvent,false,Alt_L

Authors

The minimega authors

Created: 14 Jun 2017

Last updated: 3 June 2022