Volume 9, Number 2, July 2006
Image Restoration In Horizontal Surveillance By Telescope
- 1 School of Information Technology & Electrical Engineering, UNSW, Australian Defence Force Academy, Canberra, ACT, 2600.
- 2 EWD, Defence Science and Technology Organisation, PO Box 1500, Edinburgh, SA, 5111, Australia.
Abstract
A new method for restoring images degraded by atmospheric turbulence where the resulting point spread function varies across the field of view is discussed. Such cases occur in horizontal imaging by telescope close to the ground, especially during daytime when convective turbulence is worst. Each image frame of a captured movie sequence is exposed for a time short enough to freeze the effects of the turbulence, resulting in a random wobble and blurring of the image that is position and time dependent. Registration of each frame to a reference image is achieved either by a moving region-of-interest correlation or by a gradient-based optical flow method. In this paper, we discuss a new method, replacing correlation by a moving region-of-interest Wiener filter that came from experiments visualising turbulence in jet plumes. The resulting shift information is used to dewarp each frame of the sequence before averaging to provide a result corrected for motion-blur. Further deblurring is carried out by a variety of deconvolution techniques. The shift or blur information can also be used to visualise the intervening atmospheric turbulence.
Introduction
A new method for restoring images degraded by atmospheric turbulence where the resulting point spread function (PSF) varies across the field of view is discussed. Such cases occur in horizontal imaging by telescope near the ground [1–14]. With a portable telescope (such as the one shown in Figure 1), we capture a scene as a movie in which each of a large number of image frames is exposed during a short time to freeze the effects of the turbulence. The effects observed are a random shift or wobble with time of regions of the image and a position and time-dependent blurring. The two effects taken together correspond to a position-varying PSF.
The research builds on previous work (see [13], for example) and general progress in astronomical image restoration [16–24]. So far, we remove the effects of turbulence consecutively and independently [5–8,10–14]. To determine the spatially varying shift vectors, each frame is registered by a moving region-of-interest correlation to a reference image. An alternative registration method, based on a gradient-based optical flow approach [25], has been used [11].
The shift vectors provide the key to dewarping each image in the time sequence before averaging to a result that is free from motion blur. In a subsequent stage [2, 4–7], the motion-corrected result is further restored by application of a blind deconvolution algorithm [19–21], as in Figure 2.
A recent variant of this approach to registration is to replace cross correlation by detection of a local, space-varying point spread function (PSF). The PSF is found by computing a Wiener filter over a windowed region-of-interest (ROI) that is scanned across each image frame of the time sequence and a reference image [2,4]. As in previous work, the reference image is taken initially to be the average of the raw time sequence, and therefore includes motion blur, but other methods are also being tried [1]. It is updated to reduce motion blur after each motion-compensating pass of the complete time sequence.
In both the windowed cross-correlation method and the PSF method, a peak is found at a position that depends on the x and y disparity in the grey-scale patterns seen through the reference and test windows. However, we find that the relation between the actual disparity and the position of the PSF peak appears to be more robust in this method than that in the cross-correlation method. This is probably is due to the fact that the Wiener requires a noise-to-signal ratio factor which, if chosen correctly, leads to the more robust results.
In addition, the local PSF detected contains other useful information about the relationship between the windowed ROIs, not available with cross-correlation. In the simple case where one image is sharp and the other blurred (by a space-varying blurring function), the local PSF provides a good estimate of the local blur. In our applications, the reference image is initially more blurred than each individual frame, due to the averaged motion blur mentioned before. However, each individual frame includes space-varying blur due to higher-order turbulence effects, and the local PSF detected shows only the relative, additional blur due to motion averaging. The reference image is improved on each iteration of the complete registration and dewarping algorithm until it becomes a best motion deblurred result.
Wiener filter vs correlation
The PSF derived by a Wiener filter within a ROI shows not only the degree of blurring by its shape and size, but also any tip-tilt induced local position shift, as discussed. The method can be used in place of cross-correlation for registration purposes. It is more robust in simulated tests, because the Wiener filter takes into account the effect of noise. Mathematically, the Wiener filter and the cross correlation, computed in the Fourier domain, have similarities.
Suppose we have windowed regions of interest and of the same size extracted from a reference image and a test image. Then, the cross-correlation, r, is
where F and F-1 are the forward and inverse discrete Fourier transforms, respectively, and * indicates the complex conjugate. The position of the peak value in r gives the x and y disparity in the patterns as seen through the two windows. The peak may be the pixel of maximum value in r, or the centroid of a region of pixels around the maximum.
In the new method, we derive a local PSF, h, through a region-of-interest Wiener filter
where is the noise-to-signal ratio. Once again, the position of the peak in h gives the x and y disparity in the patterns as seen through the two windows. In addition, the shape of h provides information on the local blurring not directly available with the cross correlation method.
Simulations
Experiments were carried out with simulated warped and blurred images to test the ROI Wiener filter derived PSFs as a method of registration. Starting with four test images: (a) unwarped and sharp image, (b) unwarped and blurred image, (c) warped and sharp image, and (d) warped and blurred image, we have examined the derived PSF method for several combinations of these.
The simulated atmospherically warped image on the right of Figure 3 is obtained by warping the reference image on the left of Figure 3 according to the “true” x and y shift maps on the left of Figure 4. The size of the original images is 256 × 256 pixels, and the original image is of the lunar crater Theophilus, previously restored by us [13]. In this case, both the reference and warped images are sharp and not blurred.
An example of a ROI centred on a small crater is shown in Figure 3, and the resulting PSF derived through Equation (2) is shown in grey scale and as a surface in Figure 3. In the example, the ROI window size is 32 × 32 pixels, and it is shown overlaid on the original images to indicate its position (it is not overlaid in practice).
As expected, the derived PSF shows a sharp peak, since both the images themselves are sharp. In the case that the test image is blurred as well, the peak is wider indicating the blurring. However, the peak in both cases is offset from the origin at the ROI centre by an amount corresponding to the x and y disparity between the images. By stepping the ROI over both images, the disparities so found are used to build the derived shift maps on the right of Figure 4. (Note that, in these and other shift map images, the grey-level indicates the degree of shift, white positive and black negative.)
To cover a broad range of scales in x and y disparity, the ROI window size and scanning step size is altered in a hierarchy of scanning passes. In the example shown, the ROI window size on the first pass is 32 × 32 pixels with a span step of 4 pixels in each dimension, changing to 16 × 16 with a span step of 2 on a second pass, and to 12 × 12 with a span step of 1 on a final pass.
After each pass, intermediate shift maps of size that is inversely related to the span step are obtained. After removing outliers with a median filter, the resulting shift maps are interpolated back to the size of the original images. This allows the position of the test ROI on the next pass in the hierarchy to be offset according to the coarser shifts found at the previous level of the hierarchy. Thus, apart from the initial pass in the hierarchy, the shifts found on each pass are relative and are summed to form a running total. The derived shift maps agree well with the true shift maps and can be used in practice to inverse warp each image in a turbulence distorted time sequence [2, 4].
Note the vignetting of the ROI windows. These are modified by an edge-smoothing cosine-bell function to bleed the pixel values in the edge 25% of the window down to the mean of the ROI window at the extreme edges. This method has been found to help reduce edge effects as the ROI is scanned over the images. In a similar way, the input images have 10% of their edges bled down to their mean at their edges, which are further extended by mean values to avoid problems when the ROI scans to the edges of the images.
To provide sub-pixel accuracy in disparity estimation, the input images are interpolated to a larger size, and a factor of 2:1 is used in these test examples.
Turbulence visualisation
A useful by-product of these methods is the visualisation of the intervening turbulence. For example, the x and y shift maps obtained during registration of the “House on a Hill” sequence mentioned previously are shown in Figure 5. This indicates the local tip-tilt behaviour of the wavefront. When the shift maps are viewed as a movie, it is as though one is watching the intervening turbulence itself “boiling” past. In fact, in this particular case, the changing pattern appears to move across from right to left at a steady rate.
It seems reasonable to assume that we are visualising the prevailing wind at the time, in this case, which would tend to be horizontal due to the proximity of the ground. Previously, we had seen a similar effect when restoring images of the crater Theophilus on the moon, in which the transport of the turbulence is at approximately 45° to the image frame [13].
We expected that the same approach would be applicable to visualising the wake behind a jet aircraft. A movie was made looking at a background of trees and other objects close to a runway as an aircraft accelerated past for take-off. A region-of-interest registration of each frame to a reference frame (the background just before passage of the aircraft) failed to produce useful results. The reason appears to be due to the extreme blurring of the scene by the extreme turbulence at the time, making meaningful registration very difficult. (There need to be recognisable patterns within the windows to provide a “lock on” for registration.)
Although disappointing at first, it did point to the different approach to both the visualisation and registration discussed in this paper, i.e., the moving window, region-of-interest Wiener filter. The windowed Wiener filter is applied between the reference frame and each subsequent frame following the passage of the aircraft, looking at the background scenery, as in Figure 6. The reference frame is of the same background, but the detail is sharp as it is taken just prior to the passage of the aircraft and its turbulent wake.
The relative “width” of the PSF derived through a windowed Wiener filter in this way has been converted to a grey scale value and displayed in Figure 7. Here we can clearly visualise the turbulent wake, due to the relative blurring of the background image. To provide an indication of position, an edge-enhanced version of the background scene has been superimposed on the image.
A sequence of these images, played as a movie, provides a striking visualisation of the moving turbulent wake as it decays over time.
Conclusion
A general overview of image restoration for surveillance by telescope has been described, including a new approach to image registration using a region-of-interest Wiener filter. Testing with simulated warped and blurred images suggests that it will provide a robust method of obtaining x and y disparity information while at the same time providing additional information on any blurring that is also likely to be position-dependent.
Our interest in this area results from our previous research in wide-area restoration of images degraded by atmospheric turbulence effects, as in astronomy or horizontal imaging by telescope.
Acknowledgements
We gratefully acknowledge the Australian Research Council, which funded the earlier wide area imaging research, DSTO, who provided a contract that led to the idea of the Wiener filter approach discussed in this paper, and the University of New South Wales for its overall support of our research. We also gratefully acknowledge the assistance of M. Resa Sayyah Jahromi, David Clyde and Warren Streitberg for their work in capturing, dewarping, and deconvolving the images.
References
[1] M. Tahtali, D. Fraser and A. Lambert, “Restoration of Non-Uniformly Warped Images Using a Typical Frame as Prototype”, IEEE Tencon’05, ISBN: 0-7803-9312-0, paper no. 1568965491, Melbourne, 21-24 November 2005.
[2] D. Fraser and A. Lambert, “Information Retrieval From a Position-Varying Point Spread Function”, ACIVS 2004—Advanced Concepts for Intelligent Vision Systems, Brussels, Belgium, pp. 415–9, 31 August–3 September 2004.
[3] A. Lambert and D. Fraser, “Super-resolution in Imagery Arising From Observation Through An-isoplanatic Distortion”, Image Reconstruction from Incomplete Data III (AM320), SPIE 49th Annual Meeting, Denver, Colorado, USA, 5562-8, 2–6 August 2004.
[4] D. Fraser, A. Lambert, and M. Reza Sayyah Jahromi, “Position-Varying Tip-Tilt Estimation and Region-of-Interest PSF Derivation by Wiener-Filter”, Image Reconstruction from Incomplete Data III (AM320), SPIE 49th Annual Meeting, 2-6 August 2004, Denver, Colorado, USA.5562-6.
[5] D. Fraser, A. Lambert, M. Reza Sayyah Jahromi, M. Tahtali, and D. Clyde, “Wide Field-of-View Image Correction with Turbulence Tip-Tilt Visualization”, Invited Paper in Chapter 3, High Performance Computing, 2003 AMOS Technical Conference Proceedings (CDROM, (US) Air Force Research Laboratory), 2004.
[6] D. Fraser, A. Lambert, M. Reza Sayyah Jahromi, M. Tahtali, and D. Clyde, “Anisoplanatic Image Restoration at ADFA”, Digital Image Computing: Techniques and Applications, Proc the VIIth Biennial Australian Pattern Recognition Society Conference — DICTA 2003, Vol, 1, CSIRO, ISBN 0 643 09039 8, pp. 19–28, 2003.
[7] D. Fraser, A. Lambert, M. Reza Sayyah Jahromi, D. Clyde, and N. Donaldson, “Can Broad-band Image Restoration Rival Speckle Restoration?”, International Symposium on Optical Science and Technology, Seattle (Proc. SPIE, Vol. 4792, ISBN 0-8194-4559-2), pp. 185–192, 8–9 July 2002.
[8] A. Lambert, D. Fraser, M. Reza Sayyah Jahromi, and B.R. Hunt, “Super-resolution in Image Restoration of Wide-Area Images Viewed Through Atmospheric Turbulence”, International Symposium on Optical Science and Technology, Seattle (Proc. SPIE, Vol. 4792, ISBN 0-8194-4559-2), pp. 35–43, 8–9 July 2002.
[9] Carmen J. Carrano, “Speckle Imaging over Horizontal Paths”, International Symposium on Optical Science and Technology, Seattle (Proc. SPIE, Vol. 4825), pp. 109–120, 8–9 July 2002.
[10] A. Lambert, D. Fraser, G. Thorpe, M. Reza Sayyah Jahromi, David Clyde, J. Webb, and M. Tahtali, “Experiments in Horizontal Imaging”, Proceedings 2001 Signal Recovery and Synthesis Symposium, Optical Society of America, Albuquerque, New Mexico, pp. 13–15, 4–8 November 2001.
[11] D. Clyde, D. Fraser, A. Lambert, and I. Scott-Fleming, “Gradient Techniques For The Restoration Of Non-Uniformly Warped Images”, Proceedings 2001 Signal Recovery and Synthesis Symposium, Optical Society of America, Albuquerque, New Mexico, pp. 62–64, 4–8 November 2001.
[12] D. Fraser, A. Lambert, G. Thorpe, and M. Reza Sayyah Jahromi, “Image Restoration and Turbulence Visualization in Wide Field of View Imaging”, Invited Presentation, Imaging through Volume Turbulence Symposium, Optical Society of America Annual General Meeting, Long Beach, California, 14–19 October, 2001.
[13] D. Fraser, G. Thorpe, and A. Lambert, “Atmospheric Turbulence Visualization with Wide-Area Motion-Blur Restoration”, Journal of the Optical Society of America, A 16, pp. 1751–1758, 1999.
[14] D. Fraser, G. Thorpe, and A. Lambert, “Visualization of Turbulence and Motion-Blur Removal in Wide-Area Imaging Through the Atmosphere”, Invited Paper, Proceedings of the Optical Society of America, Summer Topical Meeting on Signal Recovery and Synthesis, Kailua-Kona, Hawaii, (OSA 1998 Technical Digest Series, OSA, Washington, D.C., 1998, ISBN 1–55752–521–8), pp. 16–19, 9–11June 1998.
[15] A. Lambert and D. Fraser, “A Linear Systems Approach to Simulation Of Optical Diffraction”, Applied Optics, Vol. 37, pp. 7933–7939, December 1998.
[16] M.A. van Dam, and R.G. Lane, “Tip/tilt Estimation From Defocused Images”, Journal of the Optical Society of America. A 19, pp. 745–752, 2002.
[17] M. Roggemann and B. Welsh, Imaging Through Turbulence, CRC Press, Boca Raton, Florida, 1996.
[18] T.S. McKechnie, “Light Propagation Through the Atmosphere and the Properties of Images Formed by Large Ground-Based Telescopes”, Journal of the Optical Society of America, A 8, pp. 346–65, 1991.
[19] N.F. Law, and R.G. Lane, “Blind Deconvolution Using Least Squares Minimisation”, Optics Communications, 128, pp. 341–352, 1996.
[20] B.L.K. Davey, R.G. Lane, and R.H.T. Bates, “Blind Deconvolution of Noisy Complex-valued Image”, Optics Communications, 89, pp. 353–6, 1989.
[21] G.R. Ayers, and J.C. Dainty, “Iterative Blind Deconvolution Method and its Applications”, Optics Letters, 13, pp. 547–549, 1988.
[22] R.H.T. Bates, and M.J. McDonnell, Image Restoration and Reconstruction, Clarendon Press, Oxford, 1986.
[23] A. Labeyrie, “Attainment of Diffraction-limited Resolution in Large Telescopes Fourier Analysing Speckle Patterns in Star Images”, Astronomy and Astrophysics, 6, pp. 85–87, 1970.
[23] D.L. Fried, “Optical Resolution Through a Randomly Inhomogenous Medium for Very Long and Very Short Exposures”, Journal of the Optical Society of America, A 56, pp. 1372–1379, 1966.
[24] J. Stone, et al, “Anisoplanatic Effects in Finite-aperture Optical Systems”, Journal of the Optical Society of America, A 11, pp. 347–57, 1994.
[25] E. Trucco and A. Verri, Introductory Techniques for 3-D Computer Vision, Chapter 8, Prentice Hall, 1998.