topic RE: Unreliably reading std out with Utility - Environment action in Product Forum

Unreliably reading std out with Utility - Environment action

JaimeSalazar — Mon, 18 Apr 2022 11:28:00 GMT

Hi,

I've created an action under Utility - Environment which runs a process, waits for it to finish, and reads its standard output. This custom action works fine most of the time, except that 20-30% of the time the stdout comes out empty. I know that the process itself always outputs to stdout, it just seems that sometimes BP is unable to read it properly.

My custom action is a combination of 2 existing actions in Utility - Environment: "Start Process Read StdErr and StdOut" and "Run Process Until Ended". I had to create my own custom action because (1) "Start Process Read StdErr and StdOut" doesn't have a 'Working Folder' or a 'Timeout' parameter and (2) it is not clear to me if this action actually waits for the process to end (I may be wrong about this last point). On the other hand, obviously Run Process Until Ended explicitly appears to wait for the process to end, but it does not read stdout or stderr. Therefore, I was able to combine both actions into one, which (usually) does both things I need, that is, is waits for the process to end or a timeout AND also reads the stdout. This is the code for the action I created:

timedOut = False
Dim startTime as Date = Date.Now
Dim processName As String = appn
Dim proc As New Process() With {
    .StartInfo = New ProcessStartInfo() With {
        .FileName = processName,
        .Arguments = args,
        .WorkingDirectory = dir,
        .UseShellExecute = False,
        .CreateNoWindow = False,
        .RedirectStandardError = True,
        .RedirectStandardOutput = True
    }
}

Using proc
    proc.Start()

    Dim standardOutput = proc.StandardOutput.ReadToEndAsync()
    Dim standardError = proc.StandardError.ReadToEndAsync()

    timedOut = Not proc.WaitForExit(CInt(timeout.TotalMilliseconds))

    If standardOutput.isCompleted Then Standard_Output = standardOutput.Result
    If standardError.isCompleted Then Standard_Error = standardError.Result
End Using

As mentioned, this action typically works except for when stdout comes out empty, as evidenced by the logs. The timeout is set to a pretty long time and my action is not raising its built-in timeout exception, so that's probably not it. I'm tempted to think that BP is simply unreliably reading this action's output, but I don't know how to improve my code (the line If standardOutput.isCompleted Then Standard_Output = standardOutput.Resultis pretty clear).

Some forum posts talk about adding delays or retries, or checking your data items, but that doesn't seem to apply here. Others recommend having BP just read the log.txt that my process creates (which would add some complexity) or provide alternative code, but I don't really think it should be necessary. I mean, the above code is pretty straight-forward no?

Any ideas? Thanks

PS. For context, the process I'm running is a Python script, but I don't believe the specific type of process should matter too much for this question.

------------------------------
Jaime Salazar
------------------------------

RE: Unreliably reading std out with Utility - Environment action

PabloSarabia — Mon, 18 Apr 2022 12:27:00 GMT

Hi @JaimeSalazar

Can you try putting these two sentence after the Process Start (proc.Start())?

proc.WaitForInputIdle()
proc.WaitForExit(CInt(timeout.TotalMilliseconds))

Then you can use the StreamReader with the method ReadToEnd (against ReadToEndAsync, because the process should already be finished at this point)

Hope this helps you!

Hasta luego 🙂

------------------------------
Pablo Sarabia
Architect
Altamira Assets Management
Madrid
------------------------------

RE: Unreliably reading std out with Utility - Environment action

devneetmohanty07 — Mon, 18 Apr 2022 12:39:00 GMT

Hi Jamie,

Have you tried by tweaking the condition a bit like this:

If Not String.IsNullOrEmpty(standardOutput.Result) Then Standard_Output = standardOutput.Result

------------------------------
----------------------------------
Hope it helps you out and if my solution resolves your query, then please mark it as the 'Best Answer' so that the others members in the community having similar problem statement can track the answer easily in future

Regards,
Devneet Mohanty
Intelligent Process Automation Consultant | Sr. Consultant - Automation Developer,
Wonderbotz India Pvt. Ltd.
Blue Prism Community MVP | Blue Prism 7x Certified Professional
Website: https://devneet.github.io/
Email: devneetmohanty07@gmail.com

----------------------------------
------------------------------

RE: Unreliably reading std out with Utility - Environment action

JaimeSalazar — Mon, 18 Apr 2022 13:14:00 GMT

Gracias Pablo, I'm assuming this part should therefore look like this:

Using proc
proc.Start()

proc.WaitForInputIdle()
timedOut = Not proc.WaitForExit(CInt(timeout.TotalMilliseconds))

Standard_Output = proc.StandardOutput.ReadToEnd()
Standard_Error = proc.StandardError.ReadToEnd()
End Using

I'll test it a bit and post my results back to this thread.

EDIT: SinceReadToEnd() returns a String instead of ReadToEndAsync()'s Task(Of String) , the original standardOutput.isCompleted is not applicable anymore (nor does it appear to be necessary anymore thanks to this restructuring of the code).

------------------------------
Jaime Salazar
------------------------------

RE: Unreliably reading std out with Utility - Environment action

JaimeSalazar — Mon, 18 Apr 2022 13:16:00 GMT

Thanks for the tip Devneet, I'm going to try Pablo's suggestions first but I will definitely keep in mind the use of String.IsNullOrEmpty(). Will post back with results.

------------------------------
Jaime Salazar
------------------------------

RE: Unreliably reading std out with Utility - Environment action

JaimeSalazar — Fri, 22 Apr 2022 07:49:00 GMT

Thanks for the tip @PabloSarabia, although it turns out that WaitForInputIdle() fails. According to BP, it is likely "because the process does not have a user interface". Sure enough, the docs mention that "This overload applies only to processes with a user interface and, therefore, a message loop." Although my bots are all Selenium (Python) scripts, I suppose they are not considered to have a user interface.

Once I removed that line, the code was able to at least run. However, throughout the next few days I noticed my Python programs started freezing at various times. I can't say for sure, but I think I really do need to go back to ReadToEndAsync() instead of ReadToEnd(). I have in fact gone back to my original code and Python is back to running smoothly.

However, the issue remains where BP sometimes reads an empty stdout. I will likely be trying @devneetmohanty07's suggestion in the next few days.

------------------------------
Jaime Salazar
------------------------------

RE: Unreliably reading std out with Utility - Environment action

JaimeSalazar — Mon, 06 Jun 2022 07:26:00 GMT

To summarize, it looks like what has ended up working is @devneetmohanty07's solution regarding If Not String.IsNullOrEmpty(standardOutput.Result). In particular, I decided to try with a for loop, which may or may not be required, I just know that I haven't read an empty stdout since I implemented it. Here is the full code in the customized action I created in the Utility - Environment VBO:

timedOut = False
Dim startTime as Date = Date.Now

Dim processName As String = appn

Dim proc As New Process() With {
    .StartInfo = New ProcessStartInfo() With {
        .FileName = processName,
        .Arguments = args,
        .WorkingDirectory = dir,
        .UseShellExecute = False,
        .CreateNoWindow = False,
        .RedirectStandardError = True,
        .RedirectStandardOutput = True
    }
}

Using proc

    proc.Start()

    Dim standardOutput = proc.StandardOutput.ReadToEndAsync()
    Dim standardError = proc.StandardError.ReadToEndAsync()

    timedOut = Not proc.WaitForExit(CInt(timeout.TotalMilliseconds))

    For attempt As Integer = 0 To 2
        If standardOutput.isCompleted And Not String.IsNullOrEmpty(standardOutput.Result)Then
            Standard_Output = standardOutput.Result
            Exit For
        End If
        Threading.Thread.Sleep(5000) '5 seconds
Next

    For attempt As Integer = 0 To 2
        If standardError.isCompleted And Not String.IsNullOrEmpty(standardError.Result)Then
            Standard_Output = standardOutput.Result
            Exit For
        End If
        Threading.Thread.Sleep(5000) '5 seconds
    Next

End Using

Thanks for the help!

------------------------------
Jaime Salazar
------------------------------

RE: Unreliably reading std out with Utility - Environment action

devneetmohanty07 — Mon, 06 Jun 2022 08:14:00 GMT

Thanks a lot Jamie to share the results with us and glad to see the solution working for you as well.

------------------------------
----------------------------------
Hope it helps you out and if my solution resolves your query, then please mark it as the 'Best Answer' so that the others members in the community having similar problem statement can track the answer easily in future

Regards,
Devneet Mohanty
Intelligent Process Automation Consultant | Sr. Consultant - Automation Developer,
WonderBotz India Pvt. Ltd.
Blue Prism Community MVP | Blue Prism 7x Certified Professional
Website: https://devneet.github.io/
Email: devneetmohanty07@gmail.com

----------------------------------
------------------------------