I had a cheap optical mouse sitting around where one of the wires had broken, and wanted to do something with it, so I tapped into the sensor!
Optical mice basically have a low-resolution but high-speed greyscale camera in them. The sensor chip contains this camera, along with a processor. This processor acquires images and determines in what direction the micro-scale features beneath the mouse are moving. It then periodically communicates the movement signals back to the computer it is plugged in to.
Many of the sensor chips in cheap optical mice at the time were based on designs from Avago. These apparently are mostly no longer available, though there are a few Pixart Imaging devices that may or may not have all of the info needed to interface to them publicly available. Buying a few cheap mice and seeing what chips they have in them may still be the best option. The chip in my particular mouse was labeled A1610. Looking at the datasheets on their site, I found that the ADNS-2610 sensor had a similar pinout and features to my mystery device. It turns out that the sensor is an 18×18 pixel device, with 6-bit (64-level) greyscale output. Each pixel scales to 1/200" and images are acquired at 1500 frames per second. A variety of commands are available over a serial interface. In addition to the delta X and Y movement distances, the actual surface photo can also be read out.
To tap in to the sensor chip, I cut through the data traces connecting the sensor to the interface processor. This allowed me to solder wires directly on to the pins of the sensor device. I then connected the sensor to an Atmel AVR microcontroller. I then wrote AVR code which talked to the sensor directly, extracting the raw sensor pixel and movement data and sending it back to a computer via serial port.
This particular sensor is a very low-end model. The frame rate back to the PC after the serial interface is on the order of 5 frames per second. Combining the image data with offsets returned from the actual mouse functions of the device produces a crude handheld scanner, but the lag and blur make it very difficult to use. With a higher-end sensor, much better performance was easily obtainable.