Command with Target

We mentioned in the chapter "A Brief Introduction to RPA" that one of the main features of RPA is that it is non-invasive. In other words, although RPA works with other software programs, it does not require them to provide a special interface. Instead, it directly interacts with the user interface of other software programs to simulate human interactions. However, there are often many input boxes and buttons that we can interact with in a given software interface. How does Laiye RPA know where and how we want to operate? The "interface elements" and "Commands with target" introduced in this chapter will solve this problem.

Interface Element Overview

If you have some programming experiences and know what a "control" is, then please forget this concept for now. "Controls" and "interface elements" have something in common, but they are not exactly the same, so we must try to avoid conceptual confusion.

Other than programmers, most people interact with the graphical interface of the operating system when using computers.Whether it is the commonly used Windows and Mac OSXor Linux, which is not well-known amongst non-programmers, they all have their own graphical interfaces.As web browsers became widely adopted, more and more graphical interfaces are displayed in a browser.These graphical interfaces can take on many shapes and forms, but when we click on them with our mouse, there is a small graphical component to register that we have clicked on something. We will call these graphical components "interface elements".

For example, the figure below is a commonly seen window on the Windows operating system, as well as a typical graphical user interface. What are the interface elements in this window?

First of all, there are various options in the menu bar above, such as "File", "Home", "Share", and "View", and these are all interface elements.The icons and the text in the toolbar below, such as "copy" and "paste", are also interface elements. Likewise, the "Quick access", "Desktop", and "Downloads" in the navigation bar on the left are all interface elements. Each file displayed in the main area of the window (highlighted by the red box in Figure 31) is also an interface element.

Notice how interface elements can also contain one another. For example, the area of the window highlighted by the red box is a large interface element, and it contains the interface element representing each file inside it.

In Laiye RPA,the role of the interface element is to serve as the " target" in Commands " with targets" (i.e., Commands that require a target).

As mentioned earlier, Laiye RPA provides a large number of preset Commands, which you can find in the command area. Among them, the most basic ones are as shown in Figure 32.

Among them, all Commands under the "Interface Element" and "Text" categories are Commands with target. Likewise, Commands under the "Mouse" and "Keyboard" categories that contain the word "target" are also Commands with target. These Commands are highlighted by the red boxes in Figure 33.

Commands with target specify an interface element as a Property. When you run this Command, it will first check whether this interface element exists. If it exists, the Command will perform its specific operation on it.For example, if the specified interface element is a button, then the Command "Click Target" will click on this button.If the interface element does not exist, the Command will search for the interface element repeatedly until some time limit is reached (this is known as "timeout" which could be set in Property field). An error will be output, and in turn, the process will be terminated.

Conversely, Commands without target do not take an interface element as a Property. For example, the Command "Simulate Click" does not require a target. When running this Command, it will click where the mouse currently is. As another example, the command "Simulate Key" also requires no target. When running, it will simulate a key operation where the input focus is.

Whenever possible, you should always prefer using commands with target, because they are much more accurate. Only when you cannot specify the target as an interface element should you use commands without target as the second best choice.

Therefore, selecting a target accurately is critical when using Laiye RPA. As long as the target is selected correctly, simulating an operation is relatively simple. Here is a detailed description of how to select a target.

Select Target

Laiye RPA provides a fully automatic method to select targets. Let's consider the Command "Click Target" in the category "Mouse" as an example.

Suppose we want to execute a very simple process, which only has one step: click the start menu button of Windows (it locates at the lower left corner of your screen by default). First, create a new process and open the only Block in it. Then, find the Command "Click Target" in the "Visual View" and insert it into the assembly area by dragging and dropping or double-clicking the command. You should already be familiar with these steps. If you are not, please go back and read Chapter 2, "Basic Concepts."

Now, there should be a Command in the assembly area. Note that there is a button on this command with the text "Selector" next to a target icon, like the one in Figure 34.

Click this button. The RPA Creator interface will be temporarily hidden, and a blue translucent mask with red border appears. This is the "selector", which shows you the available interface elements that you may select as the target. It follows wherever the mouse moves. When we left-click on an interface element, the selector disappears and the Laiye RPA interface reappears. The interface element covered by the selector when we left-click is the target we have selected.

As mentioned earlier, interface elements may be nested, and the mouse may be hovering over multiple interface elements at the same time. In this case, the selector will automatically choose the interface element you most likely need and cover it.Therefore, before left-clicking to confirm your selection, please move your mouse patiently until the selector perfectly covers the interface elementyou need.

Let's have a try. Use the selector to cover the start menu button, and makes sure the selector covers it perfectly.When the mask changes to the state shown in Figure 35, left-click to confirm the selection. Note that the figure below is what it looks like in theWindows 10. If you're using a different version of Windows, what you see will be different, but the idea remains the same.

Once selected, RPA Creator reappears, and the button "Selector" you just pressed now looks like a thumbnail of the target interface element you selected. This thumbnail helps you remember which target you just selected and has no impact on how the command executes.Moreover, this thumbnail is actually a button. If you click on the thumbnail, it will take you to the exact same interface that you see when you click on the "Selector" button.If the previously selected target is not what you need, or if you accidentally selected the wrong target, just press this button and try again.

Notice that every Command with target has a Property named "Target". When we have not yet selected a target, the value of this property is a pair of curly brackets "{}", as shown in the left image in Figure 36. (Note that because no target is selected, something will definitely go wrong if we try to run the command.) After we have selected a target, the Property "Target" would be populated with a complicated value, but it is still surrounded by a pair of curly brackets, as shown in the right image of Figure 36.

The value should look something like this:

Once you have learned Laiye RPA's programming language, BotScript, you will know that the value shown here is actually a "dictionary" in BotScript. For now, don't worry about these details. Just know that this is a special piece of data that specifies the interface elements you have selected. Using this data, Laiye RPA can reselect the target consistently.

If you have developed Windows applications before (if you haven't, just skip to the next section), you know that there are actually many development frameworks for applications on Windows, including SDK, MFC, WTL, WinForm, WPF, QT, Java, etc. If you count web applications that run in IE and Chrome, there are even more types.These frameworks all provide the capability of searching interface elements within its applications. From a technical standpoint, Laiye RPA is simply referencing the interface element searching APIs provided by these frameworks. This might not seem complicated, but the methods to call all these APIs are different from each other, and even a skilled developer would struggle to learn all these interfaces in a short time, let alone users not familiar with software development.

However, if you use Laiye RPA, all of these different frameworks are all abstracted as the same interface elements, and you can search and operate on them in the same way. For example, a button in an MFC program and a button in Chrome may seem like they are all just buttons, but the technical difference in simulating clicking on these two buttons is huge. In Laiye RPA, you don't need to care about these differences at all, because Laiye RPA will handle the details for us. Therefore, this is how Laiye RPA achieves and balances the three indicators of an RPA platform: "powerful", "simple", and "fast".

Edit Target

We have learned in the previous section that the selector in Laiye RPA works automatically.As long as we move the mouse to the interface element that we want as the target, the selector will perfectly mask over the needed interface element and generate a piece of data to describe the element, which allows Laiye RPA to find the target again during runtime.

Naturally, any automatic tool will fail from time to time. When using the selector, there are some common problems:

•No matter how we move the mouse, we can't make the mask perfectly cover the intended interface element. (A common situation is that the mask is covering the whole window, instead of selecting the element you need, which might be smaller or inside that window.)

•The mask can perfectly cover the interface element, but when you try to find the target during runtime using the generated data, one of the following occurs:

• Incorrect selection: the interface element can be found, but it is not the one we originally selected.

• Missed selection: the interface element existed at the time of selection, but now it cannot be found.

We will address the first scenario, i.e., when the target cannot be covered, in detail in the next chapter. Here, we will focus on the second scenario, i.e., when the target can be perfectly covered, but Laiye RPA selects incorrectly or fails to find the target in runtime.

As we mentioned in the previous section, Laiye RPA generates a piece of data similar to the one below to describe a selected target:

Laiye RPA finds the target according to this piece of data when running the process. Therefore, when Laiye RPA misses or incorrectly selects a target, something is wrong with this data, and we need to modify it.

Locate the Property "Target" of the Command, where the piece of data is displayed in the corresponding input box. It is possible to directly edit the content of the field using the input box, but the box is too small for efficient editing. Therefore, we recommend you click on the button to the right of the input box, shown in red in Figure 37.

After you click the button, an "Editor" window pops up. The image on the top of the window is a thumbnail, which shows the general appearance of the target interface element. Once again, this is just for your own reference, and Laiye RPA will not use it when running the process. Right below is the "Attribute Filters", which displays the piece of data you saw earlier as a tree structure (Figure 38).

This tree structure displays the attributes of the interface element automatically chosen by the selector earlier. Each item is an attribute, and they have been automatically selected by Laiye RPA to describe the element. During runtime, Laiye RPA only selects an interface element if it contains all these attributes. Moreover, because the interface elements are nested with each other, Laiye RPA will not only record the attributes of the target interface element, but also that of parent interface elements of the target. Therefore, for an element to be selected, the attributes of each specified level must be satisfied.

Take the start menu button of Windows in the above figure as an example. The line 0: Object and the entries underneath it represent the attributes of the interface element that contains the start menu button (the Windows taskbar), while the line 1: Object and the entries underneath it represent the attributes of the start menu button itself. During runtime, Laiye RPA will search level by level. It will first find the Windows taskbar at the first level; then it will search for the start menu button within the child elements of the taskbar.

Such strict attribute matching naturally leads to situations when Laiye RPA is unable to find an element that fits. In this example, Windows assigns start menu a "title", with the value "start" (the title is usually hidden from users, but it exists underneath the hood).Laiye RPA takes this title as part of its attributes, because, in general, the title of the button does not change.However, if one day the title of this button changes, Laiye RPA would fail to find the button.

In this case, we'll need to modify our attribute matching process. How? We can see that there is a check box before every attribute that is checked by default. Uncheck the "title" field, and Laiye RPA will no longer use this attribute when searching for the target interface element. This way, even if the title changes, the target interface elementcan still be found.

However, if too many attributes are removed, though you are less likely to miss the intended target, you are more likely to end up with an incorrect selection.For example, if you uncheck both cls: "Start" and title: "** Start **" , then any interface element under the Windows taskbar (selected by 0: Object ) can meet the condition, and Laiye RPA will likely select the wrong element.

Therefore, if the interface elements are complicated, or if their attributes often change, then it takes some practice and skill to edit the target to accurately select the intended element.Unfortunately, there are no specific rules in this regard, so you need to learn from experience.Here are several general guidelines to help you get started:

• You may be unfamiliar with some attribute names for the time being, such as cls , aaname , etc.Don't worry about them.
• Make good use of the wildcard symbol * , which "matches anything".For example, if there is an interface element that has an attribute title with value "Name: James". The value "James" might change, but the part "Name: " will remain constant.Therefore, you can use title: "Name: *" as the attribute, rather than simply removing it.
• Be conservative when removing attributes.Although removing attributes can reduce missed selections, your attribute set could quickly become too general to avoid selecting the wrong elements.When the process is running, missed selections are generally easier to debug, while incorrect selections may not be found immediately.

Expanding on the last point, we note that when Laiye RPA is running a process, most Commands with target (except for Commands like "Determine If Any Element Exist") throw an exception when the described target element is not found and it would terminate the process immediately. (Unless the exception is handled by a Try…Catch statement, and more on this later.) As a result of this, missed selections are easy for you to notice. On the other hand, if Laiye RPA selects an incorrect target, it does not know that, so it will continue to run, which silently suppresses the error and makes it hard to notice.

Lastly, the same interface elements may have different attributes on different operating systems and different browsers. Especially for Internet Explorer and Chrome, the same elements on the same page usually have drastically different attributes. For example, if we select the search bar on www.google.com in IE and Chrome, we would get massively different element attributes (see Figure 39).

Therefore, when using the Laiye RPA to create a process to be run on other computers, please try to ensure that the development and production environments are consistent (i.e., same operating system, same browser, etc.) in order to reduce unnecessary runtime errors.

Interface Element Operation

The list of preset Commands interacting with interface elements is given in Figure 40.

Don't be overwhelmed by how long this list is. There is no need to learn all these Commands one by one. By learning some of the Commands and understanding the design of Laiye RPA, you can extrapolate problem solving principles that help you know what Commands and what even Properties for these Commands to use when you have a certain functionality in mind.

As we have learned, operations on interface elements are executed through Commands , and Properties are an indispensable part of the Command. In fact, a Command is complete only when its key Properties are clearly specified. Generally speaking, a Command consists of the following parts:

Command = (do what) + (to what) + (with what) + to (get what)

" Do what" refers to operation itself. In general, the types of operations that can be performed on a target depends on the type of the target. Once you have determined the target, you know what operations you can do to it. For example, you can use "click" on a button, while you generally can't click on texts. In a similar vein, there are special operations to checkboxes, including one determining whether a box is checked and one setting the status of a checkbox. There are also some common operations that are available in almost all interface elements, such as obtaining the size, position, and text of an interface element.

" To wha** t" refers to the object, or target**, of the command's operation. For interface elements, targets can be dropdowns, checkboxes, textboxes, and more. We have already learned how to select an interface element using Laiye RPA's Selector.

" With what" refers to additional modifiers the user needs to provide when invoking this command. This also depends on the type of the target. For example, the user needs to enter a Boolean value (true or false) when setting the status of a checkbox. However, there are also some Commands that can be executed without any additional information from the user, such as Get Element Text. As long as the user sets the target of this Command, the Command can get the text of the element without any more information. In this respect, "with what" is similar to parameters in programming. A Command that does not require additional information is just like a function without parameter.

" Get what" refers to the result returned by the Command. For example, invoking the Command Get Element Text returns a string whose value is the text of the element. Likewise, invoking the Command Get Element Selection against a checkbox returns an array of the selected contents (The user may have made multiple selections).

In essence, Laiye RPA simulates real human operations to computers. It encapsulates each operation into a Command, and the key Properties of these Commands are determined by its target.

Let us consider a test form to demonstrate how to operate interface elements. This interface includes text labels, radio boxes, checkboxes, single-line textboxes, multi-line text boxes, list boxes, dropdowns, etc., which are all common interface elements in forms.

Determine If Element Exists

As the name implies, the Command "Determine If Any Element Exists" checks if a specific interface element exists in the current screen and stores the result in a variable. When the interface element exists, the Command returns "True"; otherwise it returns "False".

If you need to check whether a desired interface has appeared on the screen, you can check if a key element on that interface exists using this Command. If this element exists, it indicates that the interface appears; otherwise it means that the interface does not exist.

For example, let's check whether the "submit" button on the test form above exists.

1. Drag and drop the Command "Determine If Any Element Exists" to the Visual View, as shown in Figure 42.

1. Click the button "Selector" on the Command Determine If Any Element Exists, switch to the test form, and hover your mouse over the button "submit", so that the blue translucent mask with red border perfectly covers the button. Left-click to confirm your selection.

2. The result of the Determine If Any Element Exists Command is saved in the variable bRet, as shown in Figure 42. We can use a conditional to check whether the result is true or false. True indicates that the element is found, and false indicates that the element is not found.

Set Element Selection / Get Selected Element

Commands Set Element Selection and Get Selected Element allow you to operate form elements, specifically radio boxes and checkboxes, automatically.

Let's use the test form again for demonstration.

1. Drag and drop the Command "Set Element Selection" to the Visual View. Click the button "Selector" to select the options "Music" and "Sport" under the "Hobby" section. Pay close attention to ensure you select the corresponding checkboxes, not the texts "Music" and "Sport" themselves.

2. Drag and drop the Get Selected Element Command to the Visual View. Select the options "Music" and "Sport" under the "Hobby" section.

3. Run the process (see Figure 43). You should see "Love music" and "Love sports", which indicates that both elements are checked.

1. On the test form, we can see that the checkboxes for "Music" and "Sport" are now checked. We have successfully filled the form using these two Commands.

In terms of interface element related operations, besides commands mentioned above, there are some other commands such as "Get Child Element", "Get Checked Element", "Get Element Area" and "Element Screenshot". You can look up in Command Manual for detailed information about the usage of these commands.

UI Analyzer

Before introducing UI Analyzer, let's review the content we've discussed so far in this chapter.

Firstly, in Laiye RPA, Commands with target should be preferred for higher accuracy.

Secondly, while using Laiye RPA, users need to select the target accurately to conduct operation simulations simply.

Thirdly, users can select targets automatically with Laiye RPA. And of course, to reduce wrong or missed selections, users are allowed to edit the target manually.

Although users can select targets automatically in Laiye RPA, selecting the specific target correctly is still a great challenge.

If you are familiar with the front end, you may understand the structure of the interface element easily, especially of those on webpages, which is nested tree structure. When the robot is sent to select the target element, it may make mistakes and locate its parent or child element since sometimes they look all the same as the target element on the interface. Let's look at a specific example. This is a screenshot of one shopping website, as shown in Figure 44.

Before obtaining the URL of this online product with Laiye RPA, we should know the URL is usually specified by the attribute "href" in the tag \<a\>. We can check this in the webpage source file.

Add a "Get Element Attribute" Command and click the "Selector" button. When a blue translucent mask with red border appears, move your mouse patiently until the mask covers the element you want perfectly. Fill "href" in the bar "Property name". Then add the Command "Output Debug Info" to output the obtained URL.

After those steps, run this robot to see the result. However, the URL did not be output as expected. Why?

It's because that the target we want has not been accurately selected. Then how to ensure accurate selection? It is UI Analyzer who can help at this moment. UI Analyzer is an independent application and you can find its start button in the toolbar of RPA Creator.

Click the button to open UI Analyzer. When started for the first time, it's empty in this interface.

Click the button "Selector" on the UI Analyzer interface (The "Selector" here has the same function as the "Selector" in Commands with target). Select the same target and its detailed information such as features and the visual tree will be demonstrated clearly on this interface.

Please pay close attention to its visual tree, in which UI Analyzer highlights the tagName of the current target. It's shown in Figure 50 that the tag \<EM\> is selected. However, the target we want is tag \<a\>, the parent node of tag \<EM\>. Right-click the tag \<a\> and select "Set current target". Then the tag \<a\> will be highlighted as the current target.

Click the "Copy to clipboard" button at the bottom right corner of the visual tree to copy the features of the current target. Then switch to the Visual View of RPA Creator and edit the target by pasting those features into the Property bar. To ensure successful editing, we can click the Editor button of the target to check its detailed information.

Run this robot again and it can output the right result this time.

Install Extension

When selecting a target with the Selector, we often run into situations where the mask only covers the entire window, and no matter what you do, the mask refuses to select the interface elements within the window. As shown in Figure 53, the mask only selects the whole Chrome tab, but it cannot select the input boxes and buttons inside the tab.

There are two possible reasons for this situation. One possibility is that there is no possible way to select this interface element (more on this later). The other is that the interface is actually selectable, but it requires additional support from Laiye RPA. At the time of writing, Laiye RPA provides extensions for Chrome, Firefox, and Java programs. Installing the corresponding extensions allows us to select page elements in Chrome, page elements in Firefox, and Java interface elements like Swing, AWT, JNPL, Applet, etc.

Let's take the Chrome browser as an example to see how to install these extensions.

Chrome Browser

Chrome requires us to both install and enable the Laiye RPA extension to properly select elements inside a webpage. After you have installed the extension, you will see the icon in Figure 54, located to the right of the address bar. When you hover over the icon, a tooltip with the text "Laiye RPA Native Message Plugin" should appear.

If you have not installed the extension, follow these steps to install it:

1. Close Chrome.

2. Open any process in Laiye RPA Creator. Select "Help" -> "Install plug-in" in the menu, and then select "Chrome plug-in".

3. Open Chrome. The browser should prompt you that a new extension has been added (Figure 55). Click "Enable Extension”. You should be all set.

4. If you are still experiencing difficulties selecting Chrome interface elements, please follow Figure 56 to enable Laiye RPA Native Message Plugin.

In addition to interface elements on IE Browser and Chrome Browser, interface elements on some other browsers who adopt Chrome kernel or IE kernel can also be recognized by Laiye RPA. However, since the settings of these browsers are different and are change frequently, IE Browser and Chrome Browser are recommended in Laiye RPA.

SAP

Functions of Laiye RPA products such as recording manual operations and simulating those operations automatically are available when using SAP. But you need to set SAP up first so that Laiye RPA can identify and operate SAP controls.

Enable SAP GUI Scripting (Server)

1. Log in to SAP and perform the transaction "RZ11".

2. Enter the parameter "sapgui/user_scripting" and click the button "Display".

3. Click the button "Change Value" and enter "TRUE" in "New Value".

Enable SAP GUI Scripting (Client)

1. Log in to SAP, click the button "Customize Local Layout (Alt+F12)" and click the icon "Options".

2. Click to open "Scripting" under the category "Accessibility & Scripting" and check "Enable scripting" in "User Settings". Do not forget to cancel the checks below to block other notifications.

Dialog (modal) in F4 Help

1. Log in to SAP, click the button "Help" and choose "Settings".

2. Switch to "F4 Help" and check "Dialog (modal)" in "Display".