Feature Slate Raptor

Contents

Introduction

This document is designed to briefly detail new features to which we can choose to commit in Dawn Raptor. A discussion will follow as which features should be worked on in Dawn Raptor and features added to Jira sprints as decided later in the development cycle. The estimated development times quoted are not elapsed time, just hours coding; they do not allow for meetings etc.

* Completed in Raptor

* Not planning to do

** Carried forward to future releases.

Detailed Design

Large Changes

Harmonizing Workflow Technology

The repositories on github are based on old passerelle versions as are the Dawn actors. Dawn should be upgraded to use the new release of passerelle (the build will be partly done from these new repos instead). Ideally a github fork will exist to allow any developer of Dawn to check in a fix which can have a pull request to passerelle.

Estimate:unknown Status: **Refectoring **Collaborator: **ESRF

Workflow Upgrade

The workflow tool could be upgraded to be more user friendly. At the moment the log file is used to develop workflows and once they are running, it can be hard to diagnose failure. Recent experience with the ARPES project has shown that some expected features are missing from workflow editing.

Debug or ‘Stepping Mode’ *

Ability to ‘debug’ workflow in the UI. This would mean coupling the Passerelle debug (available with the event director model) with the Eclipse RCP debug framework. Using the RCP debug extensions would mean pydev and Passerelle debugging would be linked i.e. break points would be hit in either system during the debugging of one workflow.

Estimate: 24h Status: Underway Collaborator: Diamond

Dry run ability.

This is having a more user friendly/flexible ‘mock’ system than currently available. For instance taking actors out of the loop, they grey out. Disconnecting and reconnecting tango motors, data collection scripts run in a test or real environment. Validation of spec scripts and GDA jython scripts.

Estimate: 8-24h (task needs better definition) Status: New feature Collaborator: ESRF/Passerelle

User friendly workflow development *

With better error handling. At the moment when developing a workflow you have to use the log file. Instead the user should see and actor go red and a user understandable message be shown with an additional link to the right part of the log stack for experts. There should be a tighter integration with eclipse parts – for instance the eclipse error log. This is linked to debug as well in that it makes it easier and more precise in feel when you develop a workflow.

Estimate: 8-24h (task needs better definition) Status: New feature Collaborator: ESRF/Passerelle

Traffic Light Colour Scheme

A traffic light colour methodology applied everywhere, green running, amber stalled, red error. So actors when selected and running would be green, user interface actors in JMX would be amber while waiting for input, actors which fail would be red. Actions would also be colour coded this way.

Estimate: 6-8h Status: New feature Collaborator: ESRF/Passerelle

Plotting API Upgrade *

The plotting API currently does not allow 3D. If it was extended to 3D the elegant API design which we have the benefit of for 1 and 2D plotting today will allow new an interesting parts to be created using a rational API. The prototype API also fixes a number of defects in the PlotView/PlotServer interaction and will simplify Dawn code greatly at the point where the plot server connects to the user interface.

Estimate: 5d Status: New feature Collaborator: Diamond

Dawn Web Interface (DWI) Using RAP * (Prototype made)

The new plotting is dawn using draw2d which, with some modifications to existing Dawn source code, is available in the RAP the web interface. The DWI feature would allow remote log in back to the visit space using fedid and password followed by the viewing of data collected. The RAP application would support the project explorer for the remote file space (the visit) and allow the editor parts used to visualise the data to run correctly (double click and open image). This has an advantage that in order to be viewed the data does not have to be downloaded only the current image created from it. In later Dawn releases, DWI could be extended to support more of the functionality of Dawn, tools and ipython for instance. A server (hardware) would be required to support the RAP application and the feature would be supported in a ‘Pre-release’ prototype format for Dawn 1.1.

Estimate: 80h Status: New feature Collaborator: Diamond

Small / Medium changes

Interactive Plot Actor *

This is an actor (to be used in the ARPES pipeline but it is generic) to simplify the workflow and make it even easier to use. It will be a popup form similar to the review actor which shows a plot of the incoming data in 1D or 2D (and 3D when the plotting API is upgraded). The data may be interacted with tools and the resulting ROIs and other data (such as fit traces and smoothing) will be forwarded in the workflow message.

Estimate: 2d Status: New feature Collaborator: Diamond

Smoothing for 1D

1D plots can be smoothed using an algorithm already developed by DLS. The plotting should be extended to have UI to drive the smoothing such that any plot may be smoothed

Estimate: 4h Status: New feature Collaborator: Diamond

Harmonizing DExplore and Data Browsing perspective

A ticket has already been created for this: https://jira.diamond.ac.uk/browse/DAWNSCI for those with access.

Pydev Actor *

Similar to the current python actor, this will allow python scripts to be run in the current pydev configured interpreter. Breakpoints added with pydev will ideally be supported.

Estimate: 1-3d Status: New feature Collaborator: Diamond (Contractor)

New Workflow Actors

New actors available in the workflow. As follows:

Tool Action *

Ability to drive any 2D or 1D tool, for instance azimuthal integration, box profile etc. from the workflow using a specific actor.

Estimate: 4-6h/tool Status: New Feature Collaborator: Diamond

Improved System Command Actor(s)

Actors to complete substitution into one or more files, run a process and then complete extraction of one or more result files. This would allow arbitrary system processes to be run, for instance Alun batch script jobs to be easily. The replace and extract actors would be general purpose and come with a custom editor similar to an edna actor.

Estimate: 20h Status: New Feature Collaborator: Diamond

R-script Actor

Ability to call R-scripts from workflows and reuse existing mathematics in that language. Java integration with R is claimed to be good and products like KNIME already support this (could also support KNIME actors but these have the actor level hard coded into Swing).

Estimate: 8h Status: New Feature Collaborator: Diamond

Taverna Support

Ability to call taverna workflows in a similar way to how composite actors work. If the RCP plugins for taverna are ready when this feature is coded it may be possible to have a taverna editor and good integration.

http://dev.mygrid.org.uk/wiki/display/TNG/Preview+1

Estimate: 8h Status: New Feature Collaborator: Diamond

Deploy Workflows as Actions in Editors and Workbench

This feature allows any .moml file to be reused as an action in a special menu on any data file. When the data file is opened the user may run the algorithm without having to open and run the Passerelle user file. There will be a few deployed by default such as background image subtraction, medium of flat field images, azimuthal integration, packing of image folders into hdf5 files etc. as available.

Creators of a workflow will be able to right click on the file and open a wizard to choose how the action should be deployed. The workflow runs without UI interaction as the editor provides the data. It has also been suggested that there be an option to automatically apply the workflow when an image is opened or viewed in the image viewer. This is a general feature and has wide applications in image and 1D data manipulation.

Estimate: 32h Status: New feature Collaborator: Diamond

Rotation ability for selections

Add ability for regions of interest to rotate fully (for instance rectangle). Perhaps a tool for manipulating rotation.

Estimate: 8h Status: New Feature Collaborator: Diamond

Macintosh Support (first steps)

The first part of the work required to support MacOS - compiling CBF libraries. This will allow most of dawn features to work on Mac as HDF5 is already working. The 3D viewer (jogl) will likely not be included in this step. Graeme Winter has offered to compile the JNI C code on Mac.

Estimate: 8h Status: Underway Collaborator: Diamond

The plotting toolbar is long, for instance the zoom actions and the image tools are many. It would be good to allow coolbar docking around plots meaning users can click and drag toolbars to the side of the plot. For instance the image tools may automatically appear to the side of the plot unless switched off.

Estimate: ~10h Status: New Feature Collaborator: Diamond

Open File Loading

One of the neat features of Fable was that you can write your own loaders in python. This means any ascii or unusual image format is supported. This feature would allow the LoaderFactory to read Eclipse extension points for user defined loaders. This would mean Java and python loaders could be plugged into it. The python loaders would be called using rpc, work in cpython and conform to a given interface. The Java loaders already have a Java interface they must implement.

Estimate: 6h Status: First version in repository, not tested Collaborator: Diamond

Colour Scale for Intensity *

This feature was discussed for Dawn 1.0. There was hope that the scale already existing in SWT XY-Graph would be available in time for Dawn 1.0. The feature is to provide a third axis apart from the positional ones correlating colour used with intensity value.

Estimate: 8h Status: New feature (maybe there will be a free version of this) Collaborator: Diamond

Graph Legend Move

Optionally place the graph legend on the left instead of at the bottom. API Change to plotting.

Estimate: 4h Status: New Feature Collaborator: Diamond

Masking Upgrade *

This idea is to allow masks to be saveable and choosable from a list so that a user can create a mask and apply it to many images using HDF5. The first version will allow masks to be saved to file. Later the concept of an active mask may be introduced so that and image loaded of the correct size will have the mask applied automatically.

In addition masking will have improvements for how the user interacts with the drawing of regions. A new method of masking will be introduced to allow single pixel tools to be created. The tools will require the use to zoom until no-downsampling is done so that individual pixels can be edited without the use of regions which are less precise.

https://jira.diamond.ac.uk/browse/SCI-384

https://jira.diamond.ac.uk/browse/SCATTER-6

https://jira.diamond.ac.uk/browse/DAWNSCI-183

https://jira.diamond.ac.uk/browse/DAWNSCI-184

Estimate: 6d Status: New Feature Collaborator: Diamond

Maxima, Minima, Points of Inflection finding

There is currently a peak fitting tool which fits a function to a peak. In fact very often required is an algorithm to mark maxima, minima and points of inflection in 1D data (the first of these being the most useful). This does not fit a function and actually puts an annotation on the real data at the point of a local maximum (for instance). This work should take the form of:

1. An algorithm which can be driven separately to the UI (from one of the maths team, Peter or Mark)

2. A UI tool similar to the fitting tool

Estimate: 8h UI, 8-16h algorithm Status: New Feature

P2 Update Site

Get the auto update working properly from P2 and set up the appropriate locations as default.

Estimate: 5 d Status: New Feature Collaborator: Diamond

*Image Stack Loading **

Allow the loading of a stack/directory of images as a volume, and displayed as such with the slicing tools.

Estimate: 1d Status: New Feature Collaborator: Diamond

Integrate simple ITK into Java dataset

Bring in this jar, and link functionality into the dataset objects

http://www.itk.org/Wiki/A_visual_guide_to_SimpleITK_in_Java

Estimate: 5d Status: New Feature Collaborator: Diamond

Integrate simple ITK into scisoftpy

This can use the python bindings for python and the java bindings for Jython.

Estimate: 5d Status: New Feature Collaborator: Diamond

Color Mapping Update *

Update colour mapping tool to deal correctly with RGB images

Estimate: 3 days Status: Underway Collaborator: Diamond

Update image plotting

Update image plotting and colour mapping to deal with Composite datasets (for mapping experiments)

Estimate: 5 days Status: New Feature Collaborator: Diamond

Java7 Support *

A new Java7 based build of Dawn will be produced and tested in the course of Raptor.

Estimate: 1 day Status: New Feature Collaborator: Diamond

Feature Slate Raptor

Introduction

* Completed in Raptor

* Not planning to do

** Carried forward to future releases.

Detailed Design

Harmonizing Workflow Technology

Workflow Upgrade

Debug or ‘Stepping Mode’ *

Dry run ability.

User friendly workflow development *

Traffic Light Colour Scheme

Plotting API Upgrade *

Dawn Web Interface (DWI) Using RAP * (Prototype made)

Interactive Plot Actor *

Smoothing for 1D

Harmonizing DExplore and Data Browsing perspective

Pydev Actor *

New Workflow Actors

Tool Action *

Improved System Command Actor(s)

R-script Actor

Taverna Support

Deploy Workflows as Actions in Editors and Workbench

Rotation ability for selections

Macintosh Support (first steps)

Toolbar Position *

Open File Loading

Colour Scale for Intensity *

Graph Legend Move

Masking Upgrade *

Maxima, Minima, Points of Inflection finding

P2 Update Site

*Image Stack Loading **

Integrate simple ITK into Java dataset

Integrate simple ITK into scisoftpy

Color Mapping Update *

Update image plotting

Java7 Support *