Launcher.__call__ should return information on the output

[prabhuramachandran]

1. This would make it easy to get back the stdout/stderr along with any files generated without having to write a lot of code.
2. The returned information should essentially be a combination of what Log and FilePattern provide.

For example:

launcher = Launcher(...)
output = launcher()
results = [float(stdout.read()) for stdout in output.stdout] 

Ideally the output should contain enough information that it should be very easy to generate HoloMaps or pandas dataframes from this output without having to write a lot of code.

[melver]

While in principle, this would be quite useful, it would break the interface that both Launcher and QLauncher present equally.

Implementing this feature request for anything other than Launcher is certainly not always possible. In the case of SGE (QLauncher), the call to the Launcher will return before all jobs have finished.

Changing the interface of Launcher instances to be different is not something I would endorse. The goal is always to be able to write Launcher-agnostic code, therefore we have Log and FilePattern.

Even providing an undocumented interface, and adding this hidden feature is questionable to me.

[jlstevens]

Implementing this feature request for anything other than Launcher is certainly not always possible. In the case of SGE (QLauncher), the call to the Launcher will return before all jobs have finished.

This seems ok to me as we know where the output is supposed to appear - think of the return value as a deferred (i.e expected) specification that can be resolved to the actual results once the jobs are actually finished. Obviously something that can be done immediately for Launcher but you may have to wait for with QLauncher.

Changing the interface of Launcher instances to be different is not something I would endorse. The goal is always to be able to write Launcher-agnostic code, therefore we have Log and FilePattern.

I also agree with the idea of that you should always be able to write Launcher-agnostic code. That said, I don't see why we can't make all Launchers behave in the same way. The directory structure we have decided to use with Lancet is designed to have the standard output in one place (the streams directory) and then files should appear in the appropriate root directory.

I think we will always know where the standard output will go as this should certainly be an option for any reasonable batch job system.

Of course, any process executed is out of our control and may not follow any of our suggested conventions. That said, this is always an unavoidable problem and I think having an easy handle on standard output/error and the output directory for a particular launch based on where the files should be would be very useful feature.

In short, I am in favor of this idea in principle. I would want to see a nice, short and elegant implementation before I would want to include such a feature in practice!

[jlstevens]

Since commit 276bff9, I would now also consider returning a FileInfo object instead of a FilePattern. The issue then is to select a filetype appropriately which may be tricky, especially if many different filetypes are output by the run.

I am now wondering if there could be some sort of completely generic FileType object that could be set by default that simply outputs the file contents (whatever they may be) from the data method and maybe some generic metadata (e.g the file modification time) for the metadata method. Then maybe FileInfo could somehow be treated as a strictly more useful class than FilePattern.

Anyway, I will soon be using Lancet again for my own work which will help me decide what I want to do!

[prabhuramachandran]

I have a very simple implementation that should please @melver as it does not modify anything that currently exists and makes it easy to get the generated outputs.

#12