Merging individual ALS 9.0.1 exposures for one 2D view: Henry Chapman's merger

merger is an IDL graphical user interface (GUI) program to combine measured diffractograms of different exposure times into one data array. It subtracts CCD background, removes saturated pixels, and masks out the beamstop. There are several ways of starting and getting data into merger:

1. Start the program with the IDL command merger. Read a .nc file (or other compatible format) from the Intensity data$\rightarrow$Read frame menu.

2. Start the program by typing the IDL command merger. Read a .mg file from the File$\rightarrow$Open menu item.

3. Start the program with the IDL command merger, info=info. Insert an array d with the command merger_new, info, alt_data=d.

4. Start the program with the IDL command merger, file.mg where file.mg is a .mg file that is simply a text file that lists the datafiles and various parameters.

5. Start merger_manager with the IDL command merger_manager, dir=directory, where directory is the directory of datafiles. Then select the files (control-click to select discontinuous lists of files) and click on Send to merger.

There are no limits to the number of merger windows you can have open.

The main merger window lets you browse through the various ``frames'' of data that will be eventaully merged into one array. Select a frame with the slider near the bottom of the window. You can choose whether to display combinations of the intensity file, the background file, background-subtracted intensity, with or without masking of the beamstop mask or saturation. These choices are made from the Display panel on the bottom left. Click and drag in the image window to make a rectangular selection. You can then use the Zoom In, Zoom Out, or Auto buttons (the latter displays the image with an intensity scale that spans the minimum and maximum values of that selection). When there is no rectangular selection (a click without a drag), then these buttons act on the entire image. You can do similar things from the View menu.

Use the Background menu to load in background files. The selection of Background$\rightarrow$Auto-background files$\ldots$ gives a form that will allow newly-read intensity data to automatically choose a background file based on a critereon of matching a value of a specific dt_par tag (most likely IMAGE_SECONDS). There is a similar menu and form for Mask files (you could link a mask file to a value of bsystg_m, for example).

Once you've loaded in all data, background, and masks, you can select Analyse$\rightarrow$Merge all frames. The result of the merge will appear in a new merger window.

You can save to a .mg file at any time by choosing File$\rightarrow$Save As$\ldots$ or File$\rightarrow$Save. This will produce a text file that you can edit. The file can also be read by some routines. For example,

 IDL> d = mg_merge('file.mg')

will return the merged result of the set saved in file.mg.

The merger program might seem a bit slow - most of the time is spent on saturation detection, but some of the overhead is on the funny way the program is designed. It is based on another program for doing alignment of optical instruments based on many interferograms, which tried to avoid having too much data in memory and re-reading files when needed.

There is a nice utility called merger_manager that gives a database to preview files and list their parameters. From this you can select single or multiple files and send them to merger with the Send to Merger button. However, I'll first describe a typical session (for a single view) without using merger_manager. The session of merging might go something like this:

1. If you haven't already done so, in IDL, type merger.

2. Import some data: Choose Intensity data$\rightarrow$Read Frame$\ldots$ and then type in a file or click on the Browse button. You may choose more than one file from the list, but for this example just choose one of the files with diffraction data. If it is heavily saturated it will take awhile to load, since it is doing saturation detection.

3. Click on the Log button in the lower left to get a better view of the diffraction data.

4. Now set up the background (dark noise) and mask files. For the background, choose Background$\rightarrow$Auto-background files. This allows you to set up a table so that the right background will automatically be subtracted when a new Intensity frame is read in. Since the background is dependent on exposure time, choose IMAGE_SECONDS from the Meta-tag drop-down list. Click on the Browse button to choose a background file. Then click on Grab selected which will insert the value of IMAGE_SECONDS for the selected background file into the Value field. Next, click on Update table to enter this into the Auto-background table.

At this point, you can easily set up the other files for other exposures. Click in the table on the second row, since that's where you will want the next entry to go. Edit the filename in the Filename: field and then click Grab selected. Then click Update table. Repeat for the next background exposure: click in the table, edit the filename, click grab selected and click update table.

Sometimes you might have acquired data files at the same exposure time but with some difference that requires a different background subtraction. For example some of the exposures may be at a different ROI or binning. In this case, using an auto-background based on exposure time only won't be able to distinguish between the two cases. You can add additional conditions for the same background file simply by entering that filename into another row of the auto-background table, and set the tag to N_ROWS for example. A background file will only be subtracted from an intensity file only if its metadata satisfies all conditions (ANDed) for that background file.

If you like, click on Save table to save the table to a file (useful if this background is the same for another GMR angle). Click on OK.

The merger program will now chug away for too long (for some dumb reason it recomputes the saturation) and will subtract the background files from all the intensity frames that you have read in (only one, in this example). Note that it is possible to individually select a background for a particular frame (Background$\rightarrow$Read background file); this will always take precedence over the auto-background table.

You will see that the background is subtracted since the listed filename in the main merger window (bottom right) has [B] after it. Also, the Background button in the left Display panel is selected. You can click on that button to see the data with background on or off. Or, you can click off the Data button to see the background alone.

To scale the greyscale of the data, select a region by clicking in the image window and dragging out a box. Then click the Auto button near the bottom.

You can now read in the rest of the intensity data files. Choose Intensity data$\rightarrow$Read frame$\ldots$ and then browse to the correct directory etc, highlight the files of interest, and click on OK.

Now you want to make some beamstop masks. I do this by choosing the frame that has the longest exposure, going to Log mode, turn off saturation, and scaling the data to be able to see the beamstop. Make sure you have a zoom of 1; by default the zoom is 0.5, so click in the middle of the image (but don't drag a box) and then click Zoom In. You should see the words Zoom: 1.000 in the bottom status panel; if not, click Zoom Out until you do. Next select File$\rightarrow$Save picture$\ldots$ and then choose a filename to save to and a format such as TIFF. I then read this into Photoshop. The picture will be upside-down, but don't worry about it. Then do the following in Photoshop:

• Image $\rightarrow$ Mode $\rightarrow$ Greyscale, then OK

• Layer $\rightarrow$ New $\rightarrow$ Layer then OK

Select polygon lasso tool and use it to outline the beamstop Select a paint brush, choose a really big brush size, black color, and paint inside the selection so it is all black inside the beamstop. In the layers palette, click on the background layer and then choose Layer $\rightarrow$ Delete $\rightarrow$ Background Layer Layer $\rightarrow$ Flatten Image You will now have a black and white mask (black inside the beamstop). File $\rightarrow$ Save As to save as a tiff file.

Repeat for all beamstop positions.

Now you can set up an auto-load table for the mask as you did for the backgrounds. In merger, choose Mask$\rightarrow$Auto-mask files$\ldots$. You get a similar window as for backgrounds. This time, you want to tag the files not to IMAGE_SECONDS but to BSYSTG_M or whatever motor distinguishes the beamstop positions. Choose one of the mask files and then click on Grab current (the tiff file doesn't know anything about the beam-stop stage, so there's no point choosing Grab selected). You need to make sure that the mask you read is the right one for the current frame; the merger window is still responsive, so you can use the slider to change to the right frame. Once you've filled in the table, save it to a file and click on OK.

Now the Display in merger should be showing Data, Background, Mask, and Saturation all selected. You are now ready to merge the data, which you accomplish by selecting Analyse $\rightarrow$ Merge all frames.

Phew!

You will get another merger window with the result. This window has two frames: the first (frame 0) is the merged data and the second (frame 1) is the exposure mask (accumulated exposure for each pixel; the units are seconds at 400 mA, that is a value of 1 is equal to 400 mA seconds). You can save these frames individually to datafiles with File$\rightarrow$Save data$\ldots$ or save to a picture with File$\rightarrow$Save picture$\ldots$. To save all your hard work in assembling the files, go back to the merger window with all the intensity frames, and choose File$\rightarrow$Save. Type in a name of a file, for example gmr00.mg. This a .mg file, or a merger file. It is a text file that you can look at (and edit) that has all the appropriate files.

An easy way to merge another similar set of data (e.g., another GMR angle) is simply to copy gmr00.mg to gmr01.mg (for GMR=1 degree) and change the names in the [Intensity data] section to the new ones. The beamstop masks and backgrounds should still be valid, but if not, edit accordingly. Then, back in merger, choose File$\rightarrow$Open to read in that set, and the Analyse$\rightarrow$Merge all frames to get the merged result.