Have you ever encountered a scenario where the application you need to automate is only loaded through a remote desktop screen? or worse, Citrix?
I assume that is why you are here :)
So to automate these kind of applications we cannot use any ordinary automation tool like Selenium, WinAppDriver etc;
Since there is no DOM for the loaded application, the elements cannot be captured using these kinds of automation tools. To overcome this, you will need to use the GUI based image recognition automation solution.
Email Virtual Network Computing (VNC) technology enables sharing a copy of one computer's screen display with another computer over a network connection. Also known as remote desktop sharing, VNC is typically used by people wanting to monitor or control a computer from a remote location rather than just accessing shared files. There is a bug in Windows NT 4.0 prior to Service Pack 3 which will cause the operating system to fail with the familiar blue-screen of death if VNC is run in service mode. Running VNC in application mode appears to be safe. Windows Remote Desktop: Can I make it share the screen? Install VNC Server on it and VNC viewer on your PC (both same program and free) and then use it instead of RDP. However, that does. Download Screens - Remote Desktop, VNC, Screen Sharing for iOS to leave your computer behind and travel light. May 09, 2019 Zoho Assist Remote Desktop Sharing 1. TigerVNC is a free, open source, high-performance, platform-neutral VNC implementation. It is a client/server application that allows users to launch and interact with graphical applications on remote machines.
Lucky for us, we have a Sikuli which is built for the exact purpose. (praise the open source gods)
But starting with Sikuli and may be integrating the tool with your existing framework can be a challenging work since there are several aspects we need to consider.
- The integration should not affect the existing scripts
- The code should handle image based automation challenges
- The integration should provide a user friendly interface for the users
Structure of the integration
So for the integration of the Sikuli commands we will create a central component for managing the image based actions. So your test script bed will be separated out and provide you with the abstract layer for the image capabilities.
For this example we will try to automate a VNC connected screen.
Getting Started With Sikuli – Dependencies
Add necessary dependencies for the project.
Adding the Sikuli Abstract Commands
Let’s create an abstract command base for the image based command to keep the maintainability high and abstract out the complexity.
So now we have a abstract command class where all the complexities of the framework is hidden from the user.
Here I have added open command to open an application installed in the machine.
Note : So for Remote desktop the app name can be passed as “mstsc.exe“
the OSUtils.java class is used to determine the operating system used for the execution.
![Screens 4 vnc remote desktop screen sharing 4 0 4 152 Screens 4 vnc remote desktop screen sharing 4 0 4 152](https://edovia.com/en/screens-mac/images/features/mac/feature-mac-big-sur@2x.png)
Let’s move on with our quest to find the image elements in automation code.
We need to have a method which finds an image element in the screen before we perform any actions to it.
Code Desription
Here we have a findImageElement method which locates an image element in the given screen area and returns an actionable target screen region.
Thee above code line creates a desktop region where you are targeting for the capture. If you need you can define a custom screen region you need to search for elements as below.
desktopScreen.wait(…) command is the location where the actual image screen is located. We can pass two parameters for this.
- Target image element we are trying to locate.
- Time out for the element to be located. (In milliseconds)
The result of this is a ScreenRegion object which we can use to fire actions on the found element. If the given image could not be located with the given base image, the screenRegion will be null.
Now let’s add some standard actions based on the above for the command base.
Princess maker 5 psp english patch. Now what if we need to click a location based on a static image? A typical example of this would be clicking on a dynamic element which we do not know at the scripting time.
Sikuli Relative Image Capturing
Desktop Screen Parts
Here we can use the known static image a base and use that base image to relatively find the target image.
For the relative image you need to define the x, y, width, height relative to the base image.
Screens 4 Vnc Remote Desktop Screen Sharing 4 0 4
Sikuli Relative Action Firing
So what if we need to fire our action on the found screen region specific location?
In the above scenario if we need to click on the drop down of the button to access the list, we can do 2 things.
- Capture the dropdown element region only and fire action on that
- This can lead to script failures if there are multiple elements with the same UI.
- Recommended : Capture the element as a whole and fire an action relatively.
Let’s see how we can do it in the recommended safer way.
Now provide a point relatively to the base image for your action.
Yay. Now we have a complete working command base to get started with the image based automation.
The complete source code for this tutorial can be found at https://github.com/codelabslk/image-based-automation-example