Fun with speckle patterns

I recently came across a really fascinating video – https://www.youtube.com/watch?v=tYFLze9VwB0 where they make use of a micro laser projector to generate a speckle pattern on a surface and then photograph it with a DSLR. They then simply touch a surface with their hand/finger and then take a photo of the speckle pattern again after the hand has been removed from the surface and compare the two speckle patterns to see the areas that have changed.

See their paper here – https://ieeexplore.ieee.org/document/6247655. They note the “approach takes advantage of the fact that tiny surface deformations cause phase changes in reflected coherent light which alter the speckle pattern visible under laser illumination”.

For more information on speckle patterns, see – https://en.wikipedia.org/wiki/Speckle_pattern

I was curious if I could replicate their experiment using a laser pointer and a microscope objective (25x) as a beam expander, to generate a larger beam diameter.

In order to capture images from my DSLR I made use of gphoto2 and tethered the camera to my computer, so that I could avoid having to manually press the shutter button, which I found would cause vibrations and alter the captured speckle pattern.

gphoto2 --capture-image-and-download

The following image shows the laser pointer pushed slightly inside a 25x microscope objective:

This image depicts a speckle pattern on a wall, simply using a laser pointer and pointing it through a 25x microscope objective:

I then touched the surface with my hand and removed it, and took another photograph of the speckle pattern.

You can hopefully make out my hand in the following image, created by finding the differences between two speckle patterns:

The following simple code was used to extract the differences between the two speckle patterns, I picked the red channel from the image and found out about this way to ‘difference’ two images from – https://www.pyimagesearch.com/2017/06/19/image-difference-with-opencv-and-python/

import cv2
import numpy as np
from skimage.metrics import structural_similarity as compare_ssim

a = cv2.imread('a.jpg')[:,:,2]
b = cv2.imread('b.jpg')[:,:,2]
(score, diff) = compare_ssim(a, b, full=True)
diff = (diff * 255).astype("uint8")
thresh = cv2.threshold(diff, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1]

cv2.imwrite("out.jpg", thresh)

Alternatively if you image the ‘diff’ data directly, without a threshold, the image looks like:

I’m just thinking about ways I could remove some of the noise, I’m currently trying different sizes of a kernel along with different numbers of iterations.

kernel = np.ones((3,3), np.uint8)
thresh = cv2.erode(thresh, kernel, iterations=3)

Gives an image like below:

Extracting keypresses from a calculator

The following image shows the calculator attached to a vice, to maintain stability, which is very important.

This image shows the calculator in the same position, with the keys 7856 pressed.

Using the same code snippet as above, you should be able to see keys 7856 where pressed from the difference of the two speckle patterns (without looking at the display ;).

Alternatively if you image the ‘diff’ data directly, without a threshold, the image looks like:

Speckle pattern difference changing over time

I took a photo of an initial speckle pattern then touched my hand on the wall and continued taking photos every 10s, creating a difference between the new photo and the initial speckle pattern each time.

I think temperature seems to play some part in the decay potentially. I noticed in a warm room I was finding it hard to pick up an image of my hand, when I opened the door and let it cool a bit, it seemed to give me a better image. This needs more experimentation!

Without threshold:

Code

https://github.com/anfractuosity/specklepattern – the code for using gphoto to take photos and also generating the above video

To Do

Initial experiments were conducted with a Tamron 17-50mm lens on a Canon 550D, it would be fun to try a more telephoto lens such as a 70-200mm lens I’ve got.

Also I would like to try a 100x microscope objective and a greater difference of the laser to surface. This might require a higher power laser though.

Related work

Also I found there’s interesting research using thermal cameras to obtain keypresses:

Heat of the Moment: Characterizing the Efficacy of Thermal Camera-Based Attacks – https://www.usenix.org/legacy/events/woot11/tech/final_files/Mowery.pdf