RPA Challenge - code design and optimisation to run 20-30X faster

[pierrekublik]

Hey,

this is not really an issue, more a proposal. If this shouldn't be here, I am sorry.
My "issue":
I finished the RPA Challenge (www.rpachallenge.com) with Python RPA and I thought it could be interesting for many People (who are relatively new to coding and XPath) because you have to use relative XPath Code.

My score is not the fastest and Maybe someone has interesting ideas how to improve the time.
I added the Code and the snapshot of the time.
I do not know how to upload Jupyter Notebooks.

And Maybe this RPA Challenge could also be included in the example package for newbies to test their understanding of XPath.
Just a few thoughts..

Best regards

Pierre

[kensoh]

Hi Pierre, thanks for posting and sharing this! Yes, this post should be here and I appreciate it :)

Can you paste the code here? Either copy and paste (you can put code blocks here by putting 3 backticks ``` at the top and below of your code block). Or you can drag and drop your notebook file into the text box here to attach your notebook.

# sample

Although this tool is designed to simulate a user action, including the usual user speed in using websites, I'll want to take a look at how could the code be optimised to run this at the fastest speed, and perhaps suggest some hacks that could push the limits further, for the fun of it.

And yes, this makes good interesting example to new users who found this RPA package. Will figure out a way to add this example into the readme after we see how it can be optimised.

[pierrekublik]

Hey kensoh,

this is the code block.
Just to be safe, I also added the txt file.

import pandas as pd
df = pd.read_excel("challenge.xlsx")
df['Phone Number'] = df['Phone Number'].astype(str)
r.init(True,True)
r.keyboard("[alt][space]")
r.keyboard("x")
r.url("http://rpachallenge.com")
r.click("/html/body/app-root/div[2]/app-rpa1/div/div[1]/div[6]/button")

for i in range(len(df.axes[0])):
    r.type("//LABEL[@_ngcontent-c2=''][text()='First Name']/following-sibling::INPUT", df["First Name"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Last Name']/following-sibling::INPUT", df["Last Name"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Company Name']/following-sibling::INPUT", df["Company Name"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Role in Company']/following-sibling::INPUT", df["Role in Company"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Address']/following-sibling::INPUT", df["Address"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Email']/following-sibling::INPUT", df["Email"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Phone Number']/following-sibling::INPUT", df["Phone Number"][i])
    r.click("/html/body/app-root/div[2]/app-rpa1/div/div[2]/form/input")

r.snap("/html/body/app-root/div[2]/app-rpa1/div", "score.png")

r.wait(10)
r.close()

The most time consuming part is the typing. Like you said it simulates user action. One possibility which came to my mind is to "copy paste" every single entry, thus we dont do typing. But maybe hard to code... not sure...

RPA_challenge.txt

[kensoh]

Thanks Pierre, I'll have a look!

Copy and paste is easy, something like below would work, just need to make sure the element is in focus. I'll be trying that out. Also, the default design is to simulate normal user reaction time, thus the cycle time for TagUI is between 1-2 seconds per action when communicating with the web browser or using visual automation. This cycle time can be hacked to be set to 0 and may be able to speed things up much faster, depending on the automation scenario.

r.clipboard('text to type')
r.keyboard('[ctrl]v')

[pierrekublik]

Reducing the cycle time sounds really interesting.
Is that something what only can be achieved by adjusting the source code or can I do this too?

Copy pasting would only be faster (I guess) if we can copy to clipboard without focussing on the excel sheet. Whats in my mind: read pandas --> entry i --> clipboard entry i --> click xpath xy --> paste entry i
But not sure if this is possible and if that would be faster at all.

[kensoh]

The source files for TagUI has to be hacked. Will see if that is possible to do the hacking of source files as part of the Python code. But before that, some work-in-progress code -

import rpa as r
import pandas as pd
df = pd.read_excel("challenge.xlsx")
df['Phone Number'] = df['Phone Number'].astype(str)

def paste(element_identifier, text_to_paste):
    r.click(element_identifier)
    r.clipboard(text_to_paste)
    r.keyboard('[cmd]v')
r.paste = paste

r.init()
r.url("http://rpachallenge.com")
r.click('//*[text()="Start"]')

for i in range(len(df.axes[0])):
    r.paste('//*[@ng-reflect-name="labelPhone"]', df["Phone Number"][i])

    r.type("//LABEL[@_ngcontent-c2=''][text()='First Name']/following-sibling::INPUT", df["First Name"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Last Name']/following-sibling::INPUT", df["Last Name"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Company Name']/following-sibling::INPUT", df["Company Name"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Role in Company']/following-sibling::INPUT", df["Role in Company"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Address']/following-sibling::INPUT", df["Address"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Email']/following-sibling::INPUT", df["Email"][i])
    r.type("//LABEL[@_ngcontent-c2=''][text()='Phone Number']/following-sibling::INPUT", df["Phone Number"][i])
    r.click('//*[@value="Submit"]')

r.snap("/html/body/app-root/div[2]/app-rpa1/div", "score.png")

r.wait(10)
r.close()

 
The fastest way will be to manipulate the HTML DOM itself using JavaScript. That is possible using dom() function to modify the element text directly. But I will not go there in order to achieve fastest, because that is not even RPA anymore, it's manipulating webpage directly using JavaScript code.

[pierrekublik]

Ok I tried it and unfortunately it doesnt fill in the fields when I execute it.
It just runs over the fields but does not type anything in; at least in my case.
Already tried to switch from cmd to ctrl but did not help.
Best regards

[kensoh]

Oh no, it is not tested code, still work in progress. Above should fail, because init() has to be init(True) in order to use the low-level keyboard and mouse automation.

I'll update again when I have a good solution. If I can, I'll post a standard solution for this tool, and another solution for optimised timings 😄

[pierrekublik]

Ouh yes I also tried to turn it on, otherwise it gave an error like you said.

Ok I am really excited to see the code!! :)

[kensoh]

Below example solves the original RPA Challenge at https://rpachallenge.com

# to use Pandas to read Excel, pip install pandas -> pip install xlrd -> pip install openpyxl
import rpa as r
import openpyxl
import pandas as pd

r.init(turbo_mode = True)
r.url('https://rpachallenge.com')

# click to download challenge spreadsheet
r.click('Download Excel')
r.wait()

# load and prepare all data to string
df = pd.read_excel('challenge.xlsx', engine='openpyxl')
df['Phone Number'] = df['Phone Number'].astype(str)

# timer starts after running this step
r.click('//*[text()="Start"]')

# loop through and fill in all fields
for i in range(len(df.axes[0])):
    r.type('//*[@ng-reflect-name="labelFirstName"]', df['First Name'][i])
    r.type('//*[@ng-reflect-name="labelLastName"]', df['Last Name '][i])
    r.type('//*[@ng-reflect-name="labelCompanyName"]', df['Company Name'][i])
    r.type('//*[@ng-reflect-name="labelRole"]', df['Role in Company'][i])
    r.type('//*[@ng-reflect-name="labelAddress"]', df['Address'][i])
    r.type('//*[@ng-reflect-name="labelEmail"]', df['Email'][i])
    r.type('//*[@ng-reflect-name="labelPhone"]', df['Phone Number'][i])
    r.click('//*[@value="Submit"]')

# page as identifier means the webpage
r.snap('page', 'score.png')
r.wait(10)
r.close()

Below example solves Automation Anywhere BotGames Week 2 Challenge

# you need to click "Accept All Cookies" button on the popup page for the first run
import pandas as pd
import rpa as r

r.init(turbo_mode = True)
r.url('https://developer.automationanywhere.com/challenges/automationanywherelabs-supplychainmanagement.html')
r.click('Download Agent Territory Spreadsheet')
r.wait()
df = pd.read_excel('StateAssignments.xlsx')
po_numbers = []
for n in range (7):
    po_numbers.append(r.read('#PONumber' + str(n+1)))

r.dom('window.open("https://developer.automationanywhere.com/challenges/AutomationAnywhereLabs-POTrackingLogin.html")')
r.popup('POTracking')
r.type('inputEmail', '[clear]admin@procurementanywhere.com')
r.type('inputPassword', '[clear]paypacksh!p')
r.click('(//button)[1]')

orders_list = []
for n in range(7):
    r.type('//input[@type = "search"]', '[clear]' + po_numbers[n])
    state = r.read('(//table//td)[5]')
    ship_date = r.read('(//table//td)[7]')
    order_total = r.read('(//table//td)[8]')
    orders_list.append([state, ship_date, order_total])

r.popup('supplychainmanagement')
for order in range(7):
    r.type('#shipDate' + str(order+1), orders_list[order][1])
    r.type('#orderTotal' + str(order+1), orders_list[order][2][1:])
    agent_name = df.loc[df['State'] == orders_list[order][0]].iloc[0]['Full Name']
    r.select('#agent' + str(order+1), agent_name)

r.click('#submitbutton')
r.wait()
r.close()

[kensoh]

Gets the following result - 117 seconds
(before tuning for faster execution instead of normal human action time)

[kensoh]

It looks not viable to use clipboard to paste, because it requires clicking visually on the textbox. That takes time to click onto the text box. Let's see for above solution, after tuning how long it would take.

[kensoh]

By tuning the default delay time between the tool and Chrome from 200ms to 0, the result becomes 12.6 seconds. Typing is still character by character but type very fast -