Passing data between QM exe's - Stdin, Stdout?

[stupomer]

Hi Gintaras,
I am interested in passing data (str, array, int values) between QM-compiled executable applications and different programs running on the same PC - 2 scenarios:

1) Between two separate QM-compiled executables:
I know how to pass arguments via run command line using ExeParseCommandLine but not how to pass information in an I/O stream or pipeline between two running exe's. So currently, I do this all by setfile and getfile to local txt files. I think this must be slower and also sometimes is a problem when txt file gets locked for some reason.

2) Between tcl script in a application that uses a tcl compiler as it's API scripting lanugage and concurrently running QM-compiled executable.

I think Stdout Stdin may apply but not sure:

https://en.wikipedia.org/wiki/Standard_streams

https://msdn.microsoft.com/en-us/library/3x292kth.aspx

Also looked at these QM forum posts:

http://www.quickmacros.com/forum/showthread.php?tid=982

http://www.quickmacros.com/forum/showthr...p?tid=2872

and more relevant to Stderr

http://www.quickmacros.com/forum/showthr...p?tid=3905

Thanks for any ideas,
Thanks, S

[Gintaras]

Stdin etc can be used only when exe1 launches exe2 and waits until it exits. Function RunConsole2 uses it.

Other inter-process communication (IPC) methods that can be used in QM:

1. SendMessage WM_COPYDATA. Easy, fast. Useful when the receiver thread has windows.

2. Pipes. Not so easy. Fast. The receiver thread should not have windows. There are no QM classes for it, but I have some test code somewhere.

3. Store data in shared memory (like in files). Not very easy. Fast. Useful when you want to store shared data for a longer time, not just to send-receive like with 1 or 2. To allocate shared memory can be used QM class __SharedMemory.

4. Store data in registry (like in files). Easy, slow.

5. Mailslots. Not very easy. Useful when need asynchronous notifications (rarely). One receiver, multiple senders.

6. TCP/IP (WinSock). Quite easy with QM class TcpSocket from Archive.qml. Probably slower than 1, 2 and 3, but need to test to confirm it. The two programs can even run on different computers on local network or internet.

http://www.codeproject.com/Articles/13724/Windows-IPC

What IPC methods can be used in TCL scripts?

[stupomer]

Thanks so much Gintaras - sorry I didn't see the WM_COPYDATA threads in the forum and the shared memory examples in QM.

This is very helpful. I have some experience with WinSock so this is a good place to start but will look at the other two for speed.

As far as Tcl IPC

see:

https://www.tcl.tk/man/tcl8.5/tutorial/Tcl26.html

Running other programs from Tcl - exec, open

Previous lesson | Index | Next lesson
So far the lessons have dealt with programming within the Tcl interpreter. However, Tcl is also useful as a scripting language to tie other packages or programs together. To accomplish this function, Tcl has two ways to start another program:
open ...... run a new program with I/O connected to a file descriptor
exec ...... run a new program as a subprocess
The open call is the same call that is used to open a file. If the first character in the file name argument is a "pipe" symbol (|), then open will treat the rest of the argument as a program name, and will run that program with the standard input or output connected to a file descriptor. This "pipe" connection can be used to read the output from that other program or to write fresh input data to it or both.

If the "pipe" is opened for both reading and writing you must be aware that the pipes are buffered. The output from a puts command will be saved in an I/O buffer until the buffer is full, or until you execute a flush command to force it to be transmitted to the other program. The output of this other program will not be available to a read or gets until its output buffer is filled up or flushed explicitly.

(Note: as this is internal to this other program, there is no way that your Tcl script can influence that. The other program simply must cooperate. Well, that is not entirely true: the expect extension actually works around this limitation by exploiting deep system features.)

The exec call is similar to invoking a program (or a set of programs piped together) from the prompt in an interactive shell or a DOS-box or in a UNIX/Linux shell script. It supports several styles of output redirection, or it can return the output of the other program(s) as the return value of the exec call.

open |progName ?access?
Returns a file descriptor for the pipe. The progName argument must start with the pipe symbol. If progName is enclosed in quotes or braces, it can include arguments to the subprocess.
exec ?switches? arg1 ?arg2? ... ?argN?
exec treats its arguments as the names and arguments for a set of programs to run. If the first args start with a "-", then they are treated as switches to the exec command, instead of being invoked as subprocesses or subprocess options.
switches are:

-keepnewline
Retains a trailing newline in the pipeline's output. Normally a trailing newline will be deleted.
--
Marks the end of the switches. The next string will be treated as arg1, even if it starts with a "-"
arg1 ... argN can be one of:
the name of a program to execute
a command line argument for the subprocess
an I/O redirection instruction.
an instruction to put the new program in the background:
exec myprog &
will start the program myprog in the background, and return immediately. There is no connection between that program and the Tcl script, both can run on independently.
The & must be the last argument - you can use all other types of arguments in front of it.
[NOTE: add information on how to wait for the program to finish?]

There are many I/O redirection commands. The main subset of these commands is:
|
Pipes the standard output of the command preceding the pipe symbol into the standard input of the command following the pipe symbol.
< fileName
The first program in the pipe will read input from fileName.
<@ fileID
The first program in the pipe will read input from the Tcl descriptor fileID. fileID is the value returned from an open ... "r" command.
<< value
The first program in the pipe will read value as its input.
> fileName
The output of the last program in the pipe will be sent to fileName. Any previous contents of fileName will be lost.
>> fileName
The output of the last program in the pipe will be appended to fileName.
2> fileName
The standard error from all the programs in the pipe will be sent to fileName. Any previous contents of fileName will be lost.
2>> fileName
The standard error from all the programs in the pipe will be appended to fileName.
>@ fileID
The output from the last program in the pipe will be written to fileID. fileID is the value returned from an open ... "w" command.
If you are familiar with shell programming, there are a few differences to be aware of when you are writing Tcl scripts that use the exec and open calls.

You don't need the quotes that you would put around arguments to escape them from the shell expanding them. In the example, the argument to the sed command is not put in quotes. If it were put in quotes, the quotes would be passed to sed, instead of being stripped off (as the shell does), and sed would report an error.
If you use the open |cmd "r+" construct, you must follow each puts with a flush to force Tcl to send the command from its buffer to the program. The output from the program itself may be buffered in its output buffer.
You can sometimes force the output from the external program to flush by sending an exit command to the process.

You can also use the fconfigure command to make a connection (channel) unbuffered.

As already mentioned, expect extension to Tcl provides a much better interface to other programs, which in particular handles the buffering problem.
[NOTE: add good reference to expect]
If one of the commands in an open |cmd fails the open does not return an error. However, attempting to read input from the file descriptor with gets $file will return an empty string. Using the gets $file input construct will return a character count of -1.
Tcl does not expand file names like the UNIX/Linux shells do. So:
exec ls *.tcl
will fail - there is most probably no file with the literal name "*.tcl".
If you need such an expansion, you should use the glob command:

eval exec ls [glob *.tcl]
or, from Tcl 8.5 onwards:
exec ls {*}[glob *.tcl]
where the {*} prefix is used to force the list to become individual arguments.
If one of the commands in an exec call fails to execute, the exec will return an error, and the error output will include the last line describing the error.
The exec treats any output to standard error to be an indication that the external program failed. This is simply a conservative assumption: many programs behave that way and they are sloppy in setting return codes.

Some programs however write to standard error without intending this as an indication of an error. You can guard against this from upsetting your script by using the catch command:

if { [catch { exec ls *.tcl } msg] } {
puts "Something seems to have gone wrong but we will ignore it"
}
To inspect the return code from a program and the possible reason for failure, you can use the global errorInfo variable:

if { [catch { exec ls *.tcl } msg] } {
puts "Something seems to have gone wrong:"
puts "Information about it: $::errorInfo"
}
Example

#
# Write a Tcl script to get a platform-independent program:
#
# Create a unique (mostly) file name for a Tcl program
set TMPDIR "/tmp"
if { [info exists ::env(TMP)] } {
set TMPDIR $::env(TMP)
}
set tempFileName "$TMPDIR/invert_[pid].tcl"

# Open the output file, and
# write the program to it

set outfl [open $tempFileName w]

puts $outfl {
set len [gets stdin line]
if {$len < 5} {exit -1}

for {set i [expr {$len-1}]} {$i >= 0} {incr i -1} {
append l2 [string range $line $i $i]
}
puts $l2
exit 0
}

# Flush and close the file
flush $outfl
close $outfl

#
# Run the new Tcl script:
#
# Open a pipe to the program (for both reading and writing: r+)
#
set io [open "|[info nameofexecutable] $tempFileName" r+]

#
# send a string to the new program
# *MUST FLUSH*
puts $io "This will come back backwards."
flush $io

# Get the reply, and display it.
set len [gets $io line]

puts "To reverse: 'This will come back backwards.'"
puts "Reversed is: $line"
puts "The line is $len characters long"

# Run the program with input defined in an exec call

set invert [exec [info nameofexecutable] $tempFileName << \
"ABLE WAS I ERE I SAW ELBA"]

# display the results
puts "The inversion of 'ABLE WAS I ERE I SAW ELBA' is \n $invert"

# Clean up
file delete $tempFileName
Previous lesson | Index | Next lesson

[stupomer]

also tcl chan command:

http://www.tcl.tk/man/tcl/TclCmd/chan.htm

This command provides several operations for reading from, writing to and otherwise manipulating open channels (such as have been created with the open and socket commands, or the default named channels stdin, stdout or stderr which correspond to the process's standard input, output and error streams respectively).