Keyboard Operation on RPA

I'm playing with RPA using Sikulix on my Cloud Computer, and the basic concept of Sikulix is Image Matching. RPA is to save a button, field, or message in an image and write the logic to match it.
However, this is a little bit SLOW. The search is done on the screen to find a match with the corresponding image, so it is a reasonable process, and it can be said to be unavoidable. In this case, it is necessary to think about what can be done with scripts and keyboard operations. I have previously introduced the use of RPA notifications via e-mail or LINE.
It is useful to know that Windows can be operated with the keyboard alone more than expecting, which improves the efficiency of RPA.

1. Alt key family: Window size change, etc.
2. WinKeys family: Windows standard shortcut keys
3. Function family: Shift + F10, etc. Substitute for right button menu of mouse.
4. Tab key: Move between fields, links in the browser, etc.

When I actually tried this with Sikulix, I also found that it requires a lot of fine tuning at the execution stage. It takes quite a while for the app to accept the next key after starting up or transitioning menus. Of course, it takes a few seconds at most, but if the RPA executes type() without a pause, it usually results in an error. Naturally, the time varies depending on the performance of the machine and the load, so we have a function that waits for a certain amount of time after a key is entered in the type() function, and we fine-tune the waiting time according to the condition of the execution, but if the function is executed for a long time, the time still does not match up and an error occurs. This is a point of another difficulty that has not existed in previous command-line programs.

Notification from RPA by LINE

Notification from RPA by LINE

My Cloud Computer has recently become an RPA dedicated execution machine called Sikulix, and it runs more comfortably and crisply than my home computer (my personal opinion). After all, it is physically located in a data center, which is (probably) closer network-wise to the target access server, and best of all, it is not affected by the congestion and failures of service providers. Even I don't do stock or crypto assetes at all, but I imagine that the popularity of cloud use among those people is partly because they are playing a more serious game in this area.
I used to use Gmail to notify when RPA was being executed or finished, but I decided that nowadays it is not necessary to use e-mail, so I decided to use LINE, which is popular mainly in Japan and some other Asian countries.
Although we use LINE casually, various APIs are available to application developers. In this case, I just want to send a message, so I'll use LINE Notify to make it as easy as possible. From the LINE Notify page, create a token, which is very popular nowadays. Token is a new word, but it is not so wrong to think of it as just a long password. Copy and paste this into a file on your Cloud Computer and save it. After that, all you have to do is access the dedicated LINE Notify page with your message. It's that easy. You can also send the message by typing in the URL in your browser, but it would be better to use a script since it would take a lot of time. In this example, we put the following contents in the file sendline.sh, and then type

#! /bin/bash -f
curl -X POST -H 'Authorization: Bearer '`cat linetoken` -F 'message="'$1'"'' https://notify-api.line.me/api/notify

#! /bin/bash -f
curl -X POST -H 'Authorization: Bearer '`cat linetoken` -F 'message="'$1'"'' https://notify-api.line.me/api/notify

Add execution rights.

$ chmod +x sendline.sh

$ chmod +x sendline.sh

Note that in the above example, the tokens are stored in a file called linetoken. The cat linetoken` is a symbol to tell you to extract the contents of the file linetoken on the command line. The $1 means to set the value written at the back (called argument) when executing. When you run the command, it looks like this.

$ . /sendline.sh "line message from Cloud PC"

$ . /sendline.sh "line message from Cloud PC"

It seems that the destination of the line is not written, but that information is contained in the token issued. I wrote above that it is just a long password, but the token seems to have a slightly different role. Sikulix, which is an RPA tool, uses python, so when you call it from there, it looks like this

# Send a message from Sikulix Python to LINE
m = "Message from Sikulix"
subprocess.call(['linesend.sh', m])
```

As a result, I received a message from **LINE Notify** without any problems (:-). This could be used in many ways.

# Send a message from Sikulix Python to LINE
m = "Message from Sikulix"
subprocess.call(['linesend.sh', m])
```

As a result, I received a message from **LINE Notify** without any problems (:-). This could be used in many ways.

Notification from RPA by GMail

Notification from RPA #1 GMail

I have Sikullix in my Cloud Computer and I am playing with RPA and other things, but I can't watch it all the time after it starts because I let it run for a good long time, but I don't know when it finishes or it may have ended up with an error for some reason in the middle of the process. So, even very simple idea, but I'm going to use Gmail to notify me when the process is finished or when there is an error.
Of course, since I am dealing with RPA so far, we could use that to send it via webmail from a browser, but it would be a lot of work to start a browser, access the Gmail page, go to .... and as anyone who has ever made RPA knows, it doesn't work quite as well as I'd like (;-s). Especially this time, I want to send an email even when an error occurs while letting the user surf the web using a browser, but it is not a good idea to send an email through the browser because the error may be caused by the browser. Also, since I am a command line lover (:-), I will make this part with a script. However, my current theme is a Cloud Computer, so I don't mean to make it a mail server. It is just a normal Computer that can be used to send mail.
In the case of Ubuntu, it is under /etc/postfix. However, since security is getting stricter these days, you need to issue an application password on the Gmail side and set it up on your cloud computer as well.

On the Gmail side, go to Account ⇒ Security ⇒ Application Password.

Write this password in a file named sasl_passwd, excluding spaces. When ready, create transport.db and sasl_passwd.db and restart postfix. It looks like this.

$ sudo postmap /etc/postfix/transport
$ sudo postmap /etc/postfix/sasl_passwd
$ sudo systemctl restart postfix

$ sudo postmap /etc/postfix/transport
$ sudo postmap /etc/postfix/sasl_passwd
$ sudo systemctl restart postfix

Now all you have to do is send mail as usual.

$ echo "body of mail" | mail -s "subject of mail" mail to (xxxxx@xxx.com)

$ echo "body of mail" | mail -s "subject of mail" mail to (xxxxx@xxx.com)

The command line world is easier to understand than RPA after all (:-), and there are many cases where it is better to use scripts to do things that can be done with RPA. If you try to do everything with RPA, you will see a tough world a bit (:-).

RPA on Cloud Computer #2

I started using Cloud Computer, a local but NTT-affiliated service in Japan, with 2 CPUs and 2 gigabytes of memory for a reasonable (I think) maximum of 700 yen/month. The Dynabook N40 at home has taken on a second life (lol) as a Thin Client.
There is one thing I would like to confirm before using RPA with Cloud Computer this time. What is the resolution of the screen when not connected to Cloud Computer via VNC? ** I don't know what you are talking about. I know you don't know what I'm talking about, but I don't know much either (lol)... When I was still an active engineer about 20 years ago, most of my work and geeky activities were done on the command line. Of course, I used X-Window and Windows, but I used them only as it is, with no customization. So I am not good at this area at all. By the way, command line is what is called CUI or shell, and it is the world of computers without using a mouse. For some person, it is more understandable, let's say , it is just like MS-DOS.
RPA is a part of the mouse-using culture. In a world without a mouse, there are only commands, so there is no need to bring in the concept of RPA, and all you have to do is to write a list of commands and execute them in order from the top (with some logic, of course). Before the advent of RPA, I remember that even when surfing the Internet, I had to acquire information from scripts and programs via http and then analyze the tags. However, with the development of CSS and other technologies in addition to images and videos, information processing by tag analysis is complicated and difficult, so it has become simpler and quicker to automate the world as it is, which is what RPA is. I think this is what RPA is all about.
In RPA, of course, if the resolution changes, the images of buttons and input fields may not match the images prepared in advance, so it is necessary to know the resolution. Therefore, it is necessary to know the resolution of the screen. However, while RPA is being executed on a cloud Computer, it is left unconnected from the home PC, so the screen is not physically output anywhere during that time. However, the RPA is working by properly understanding the location and contents of the Window of the application, which is not visible to humans. What is the screen resolution at that time? is my question.
At the result , I understand that, when the RPA is running on a Cloud Computer, the monitor is connected even if the screen is not physically output anywhere. When I ran it with no VNC connection and only an ssh connection, I got the following message

$ xrandr
Screen 0: minimum 32 x 32, current 1920 x 1200, maximum 32768 x 32768
VNC-0 connected primary 1920x1200+0+0 0mm x 0mm

$ xrandr
Screen 0: minimum 32 x 32, current 1920 x 1200, maximum 32768 x 32768
VNC-0 connected primary 1920x1200+0+0 0mm x 0mm

On the Cloud Computer side, the VNC application (vncserver) is launched, so it is the monitor. In other words, when the vncserver was started, the monitor was already connected from the OS point of view, and whether or not the VNC app was viewing it from the home computer was not important to either the OS or the RPA. So if you want to run RPA from your home PC on a Cloud Computer, you don't necessarily need to connect via VNC, just an ssh connection.

$ xrandr --output VNC-0 mode=1920x1080
java sikulix ...

$ xrandr --output VNC-0 mode=1920x1080
java sikulix ...

I found no problem if I specify the resolution first and then run the program. To be honest, it was getting a little hard to start up an application for mouse (so-called desktop) on my Dynabook N40, which is my home thin client, but now that I know I can run RPA from just the command line, I'll let it work a little longer (lol).

WebARENA Indigo

I had been using AWS Workspaces as part of a project to promote the use of Personal Cloud Computer (lol), but I immediately stumbled on Sikulix, an RPA tool (;-o). I was thinking of various workarounds, but since AWS Workspaces was originally a complete package in a sense, there is not much room to tinker with it, and while it is not impossible, I do not think it should be so. In addition I would like to do various things (:0) on it, I would like to choose something from a different service.
There are a lot of Cloud providers offering various services nowadays, and among them, there are a lot of free slots for individuals, which is very appealing. The AWS Workspaces I mentioned last time is also what I am using. However, the competition for free space is naturally fierce, and it seems to happen frequently that the service is still available but there are no more spaces available at the moment.
After much consideration, I decided to try WebARENA Indigo for my next Cloud Computer. This service seems to be mostly local to Japan, but since it is run by an NTT group, which is Japan big tel-communication company liks AT&T , I decided to sign up for it right away. Thanks to the proliferation of video services such as Netflix over the past few years, and the fact that I have done a lot of free viewing myself (lol), I am not so reluctant to register a credit card and create an account. Once you have an account, the next step is to create an instance (that’s what they call it). Here, as with AWS, the number of CPUs and the size of memory are selected. Since the purpose of use is Linux, minimum option like 1 CPU with 700M of memory is OK for my purpose, but since I will be using a desktop, I selected 2 CPUs and 2G of memory. The monthly cost is about $5 at the maximum. Windows OS is also available, which is a little more expensive than Linux, but not too expensive, I personally feel (:-).

Run Sikulix Scripts

RPA gives the impression of being a business improvement tool for today's companies, but it is also one of the so called "robots" in the world of the Internet. Sikulix is a great tool that is easy to use even for beginners, but please be careful how you use it with somehow unexpectedly heavy load. I downloaded the Sikulix java file and tried to run a simple script, but it is very annoy to launch the completed script from the integrated environment every time, and the editor is hidden during the execution, so you have to wait and see if the execution is going well or not, without even seeing the log. So, it is better to run the script from the command line.

$ java -jar sikulixide-2.0.5.jar -r source location

When I run it, I place browser and term like this: Um, that's too easy work!, but isn't it enough? (:-)