Restoring Old Films

This page explains the film restoration work in slightly more details than in-code comments.

1. Scene Cut Detection

Scene cuts are detected by comparing each frame with one preceeding it. If the difference bewteen the two is large enough, this transition is marked as a scene cut. The difference is obtained by combining the mean of squared differences between the original images and the mean of absolute differences between the images with equalised histograms.

The mean os squared differences between the original images alone is not enough due to frequent intensity variations, which would produce a lot of false positives. On the other hand, histogram equalisation is likely to create quite different images from two very similar ones and is also not enough on its own.

Combining the two yeilds good results, with most cut scenes detected correctly and no false positives found in testing. Each of the two classifiers (MSD and mean of absolute differences between histogram-equalized images) can 'vote', but both of them have to agree before a frame can be marked as a cut.

2. Intensity Flickering

Intensity flickering in this solutions is assumed to be global. For each frame a mean value of all the pixels is found. If the mean of two consecutive frames is found to be different, the second one is adjusted by substracting this difference from each pixel.

3. Camera Shake Correction

To find camera shake SIFT features are found in all frames of a given scene. Then, for each frame, the features are macthed against the features of the first frame in the scene and their shifts are computed. Next, the histogram of shifts is produced. The assumption here is made that there is a static reference object (like background buildings) throughout the scene and that most features will be detected on this reference object. Consequently the histogram bin with most feature shifts in it is taken as the shift due to camera shake.

Click on the image to download a sample video

When the shift in X and Y directions is known, a translation is applied to the frame to align it with the first one. This might sometimes produce black stripes at the edges of the image, where we do not have any information. In these cases, to make the effect less noticable, the regions are replaced with corresponding regions from a previous frame.

4. Blotch Detection and Removal

Blotch detection in old films is generally made substantially easier by the fact that the vast majority of blotches only appear for one frame. Here, however, the movies have been digitised and are subject to temporal interpolation. This means that most blotches will fade-in and fade-out over few consecutive frames (typically 4-5), which makes them much harder to distinguish from pixel changes due to, for instance, a moving object in the scene.

Blotch detection (greeen) and side-effect artifacts (red).

Nevertheless, for simplicity it is assumed in this work that blotches appear for one frame only. The result is that some blotches are not detected, while already some artefacts are introduced near moving objects.

Notes

The SIFT detection and matching algorithms used were modified versions of David Lowe's original implementation.