Bahamas

William had told me about the Bahamas many times. In particular a place called Elbow Cay (a “cay” is a small island like a sand bank near a coral reef). I finally found an opportunity to spend a week there in the summer of 2018. Unfortunately Hurricane Dorian, with winds at 200mph, the worst ever to hit the island, devastated the community just a year later, in September 2019.

I loved this Caribbean feeling, the turquoise ocean, the white sand beach, the palm trees, the wooden houses, the boat trips for snorkeling and the welcoming people.

OBD vs OBS

In 1995 Nico and I wrote our thesis memo on the comparison of OBD and OBS for the pruning of neural networks, under the supervision of Jan Larssen in the department of maths, statistics and computer sciences at DTU.

Optimal Brain Surgeon (OBS) by Babak Hassibi, David Stork and Gregory Wolff of Ricoh California Research Center, published in 1993.

Optimal Brain Damage (OBD) by Yann Le Cun, John Denker and Sara Solla of AT&T Bell Laboratories, published in 1989.

The idea of OBD was to adapt or optimize the size of a neural network by removing “unimportant” weights. Fewer weights means better generalization and fewer training examples needed, which leads to improved speed of learning and classification.
Make a tradeoff between network complexity and training set error
“saliency” is a measure of the effect on the training error.

The OBS team argued that its method was “significantly better” than OBD which “removed the wrong weights”. We wanted to test this assumption.

While Babak Hassibi completed his PhD in Electrical Engineering at Stanford University and is now a Professor at Caltech, Yann Le Cun is a Professor at NYU and Chief AI Scientist at Facebook.

Yann Le Cun published a book recently (2019) about the progress of AI which reminded me of OBD. Quand la machine apprend. TBC