Build Your Own Laser 3D Scanner
Downloads
Download the new course software source files 3DP2018-A4.zip. Remember to copy your homework1.hpp, homework1.cpp, homework2.hpp, homework2.cpp, homework3.hpp and homework3.cpp files into the src/homework directory.
Hardware
The picture above shows one of four similar low cost 3D scanners that we have fabricated for this assignment. This 3D scanner is controlled by the course software running in your computer. The scanner is connected to the computer through two USB ports. One of these ports is used to connect the built-in camera, which your computer showld recognize as a regular USB camera. The second USB port is used to connect the scanner controller to the computer. The course software communicates with the scanner controller through a serial port. This scanner comprises: 1) a ball-bearing turntable, belt-driven by a stepper motor; 2) a USB camera; 3) two red laser line generator moduless; 4) a control board; and 5) a number of simple 3D printed parts. The control printed circuit board contains an Arduino Nano microcontroller, a stepper motor driver board, and a few electronic components necessary to be power the laser modules and to able to turn them on and off under computer control. The USB port in the Arduino Nano is used to connect the controller board to the computer. In general, the Arduino Nano, the laser modules, and the stepper motor can be powered by the computer through the USB. The printed circuit board also has a connector for an external 5V power supply.
Arduino Software
The Arduino Nano in each of the four 3D scanners runs the following sketch Scanner-3DP-2018.ino.zip, which is already installed in the Arduinos.
The Arduino Nano board, as well as other Arduino models, contains a USB/Serial chip, since the microcontroller itself does not have native support for USB. When you connect the Arduino to your computer through a USB port, your computer may or may not recognize the USB/Serial device. It it does not recognize the device you will have to install the proper driver in your computer before you can continue. We have used a low cost Chinese Arduino Nano clone in these scanners, which uses the CH340G USB/Serial chip. Sources for the driver are easy to find in the Internet, such as in this web page . Once your computer recognizes the device, the course software should be able to establish a communication link with the Arduino, and it should be able to control the turntable.
3D Scanner Software
A new Turntable Scanning panel has been added to the course software to implement this assignment. The GUI is composed of three windows. The left window shows a 3D scene, the center window shows images, and the right window is the control panel.
Camera
The first step is to be able to capture images within the course software using the 3D scanner camera. You should connect the 3D scanner camera to your computer before you launch the course software, because the software makes a list of available cameras only at start-up. Then you should select the 3D scanner camera from the list of available cameras. We have used a "Logitech C920" camera in the construction of these scanners, on one hand because they are very small and light, and on the other hand because we had several available from a previous project. These cameras have a native resolution of 1920x1080 pixels, and they are mounted so that the long axis is vertical, as shown in the previous picture. To make the software display the images in the proper orientation, you should select 270 as the Rotation value. If you press the "Preview" button, the software should display live video from the camera.
These are cameras with auto-focus functionality. A number of other parameters are also automatically set, and most camera drivers let the camera adjust those parameters dynamically. You will need to locate camera driver software which should allow you to disable all the automatic settings and to set camera parameters. We cannot allow the parameters to change, since changing the parameters affects all our calibration routines, as well as the image processing algorithms. For example, I am using an app named "Logitech Camera Settings" which serves this purpose for my machine running OSX. There are similar applications for Windows and for other operating systems. The application should run in parallel with the course software so that you can adjust the parameters while the course software is displaying the camera video in preview mode. You should turn Auto-Focus off, and focus the camera manually using the slider for maximum sharpness at the intended distance. In general, there is no need to make changes to the Advanced Settings.
Turntable Control
The next step is to establish communication with the turntable controller, and to verify that the software is able to control the turntable. Connect the Arduino to the computer through a serial port. Verify that the computer recognizes the USB device. Again, you should do this before starting the course application. Then, in the Turntable panel press the "Port" button.
The following dialog window should pop-up. In this dialog you should be able to select the proper serial port and the serial port communication parameters: Baud Rate = 115200, Data Bits = 8, Parity = None, Stop Bits = 1, Flow Control = None. After you make your selections press the "Apply" button.
Back into the "Turntable" group, press the "Open" button to try to establish the communications link with the turntable controller. If successful, the "Open" button should be disabled, "Close" button should be enabled, and the communication parameters should be visible next to the "Close" button
The turntable controller software stores a number of parameteres which can modified. These parameters are explained in the Arduino sketch. To modify any of these parameters edit the sketch in the Arduino IDE and reload it.
Even though the number of steps per revolution of the motor is well known from the motor specifications, the fact that the motor has a built-in reduction gearbox, that the motor is connected to the turntable through a belt, and that the motor has a belt pulley but the turntable ball-bearing does not have teeth, it is quite difficult to determine the number of steps per revolution as a result of a computation. We have designed the turntable with a built-in magnetic sensor to implement a closed feedback loop. As a result we can direct the turntable to rotate to a specific angle. The zero angle is reset every time the controller is powered up, but then it keeps track of absolute position at very high resolution.
In this assignment we will refer to a "Frame" as a group of images captured with the turntable at a fixed position. A frame could be composed of 1 to 3 images, depending on whether the lasers are turned on, or not. If the lasers are kept off at all times, which we will need to calibrate the turntable, every frame would comprise a single image. The "degreesPerFrame" parameter in the sketch measures the number of degrees per Frame. A value of 5 degrees or higher should be used initially during code development and debugging. Later on, once your scanning software starts to work properly, you should reduce this value to 1 degree or less to reconstruct denser point clouds.
The software communicates with the Arduino through the serial port using a simple text-based protocol defined in the Arduino sketch. If you know the protocol, you can type a command in the "Command" line and press the "Send" button to send it to the Arduino. The Arduino will then execute the comand and send another string as the response. The response is ususally just "OK", but some commands result in longer responses. Closing and reopening the serial port should not change these values. The values will revert to those stored in the Arduino sketch as soon as the Arduino is powered off. To make the changes permanent the values should be updated in the Arduino sketch, and the sketch should be reloaded into the Arduino within Arduino IDE.
For example, the "degreesPerFrame" value can be set dynamically by sending a command to the arduino controller. To do so press the "Command" button in the "Arduino Turntable" group. The "Serial Port Command Line" Dialog will pop up. Type "DPF 0.5 " in the "Command" line, press the "Send" button, and wait until "OK" is shown in the "Response" line.
The Arduino controller will remeber this value while it is powered on through the USB cable, but will forget it as soon as it is powered off. To make it persists, the value should be updated in the Arduino sketch, and the sketch should be reloaded into the Arduino within Arduino IDE.
It is not necessary to describe the protocol here, since you will not have to use most of them directly. Again, the protocol is defined in the Arduino sketch, and it comprises a small number of commands. The only commands that might be useful to know are those which control the lasers, as well as those which make the turntable move by one frame or one revolution: "+L1" turns laser 1 on, and "-L1" turns it off; "+L2" turns laser 2 on, and "-L2" turns it off. "GF" followed by a frame number moves the turntable to that specific frame. A second parameter can be used to specify which laser, if any, should be turned on or off (0: both lasers off, 1: laser 1 on and laser 2 off, 2: laser 1 off and leser 2 on). For example, the command "GF 5 0" will move the turntable to the position corresponding to frame number 5 (25 degrees if "degreesPerFrame" is equal to 5 degrees), and turn the two laser off. A subsequent command "GF 5 1" will only turn laser 1 on, since the turntable is already at the location corresponding to frame 5. The software uses these comands to automate the image capture process.
Image Capture
As in previous assignments, before you start capturing images, you need to create a directory structure to save the images to be captured, as well as other files to be created, in an organized fashion.
In fact, you will need to create several different directories to store the images that you will use to calibrate the camera, the turntable, and the laser planes, and then one additional directory to store the images to be captured while scanning each object. For example, I created a directory named "/Users/taubin/Teaching/3DP/2018/data/laser-scans" as the top directory for this 3D scanner. Then I created a subdirectory of "laser-scans" named "calib", and three subdirectories of "calib" named "camera", "turntable", "laser1" and "laser2". Later on, for each scanning session I will create a subdirectory of "laser-scans" appropriately named. You may decide to use a different directory structure, but you need to remember to switch to the proper directory for each step of the calibration process, and later for each scanning session. If you do not switch directories, then the images that may exist in the work directory may be overwritten, and you may have to redo previous steps.
The software provide means to automate the process of image capture and turntable control. In the Capture group select the number of frames to be captured using the "Count" parameter. When the software is started, the frame counter is initialized to 0. The frame counter is increased after each image is captured. Note that the software does not check whether or not the file name to be used to save each image corresponds to an existing file. If the file exists, the software will silently overwrite it.
Each frame is composed of between one and three images, depending on which of the three checkboxes labeled "Blank", "Laser 1", and "Laser 2" are checked. At least one of these checkboxes has to be checked. For example, if only the "Blank" checkbox is checked, then each frame is composed of only one image with both lasers off. If "Blank" and "Laser 1" are checked, then each frame is composed of two images, a first images with both lesers off, and a second image with the first laser on and the second laser off. If the three checkboxes are checked, then each frame is composed of three images, a first image with both lasers off, a second image with the first laser on and the second laser off, and a third image with the first laser off and the second laser on. The software takes care of sending the proper commands to the turntable and to capture the images at the proper times to automate the process.
During each capture sequence, the frame number and the image number within the frame is displayed, while the corresponding image is captured, and the progress bar located next to the "Stop" button shows in analog form the progress towards the end of the capture sequence.
Since the communication through the serial port between the computer and the Arduino may not be very reliable, the software has to apply certain delay. For example, when the software sends a command to the Arduino requesting the turntable to advance to a certain frame number, the Arduino may send a response back to the computer before the mechanical action has been completed. If the computer sends another command at that time, the Arduino may just ignore it. You can specify this parameter in the "Delay" field. The default value for this parameter, 2000 ms, works well with the default turntable parameters stored in the Arduino sketch. If you reduce the number of frames per revolution, increase this value inversely proportionately and test the motion.
If you start a long sequence, and you need to stop it, perhaps because you started the sequence with the wrong parameters, you can always press the "STOP" button, change the parameters and start again. Keep in mind, though, that the frame counter will be reset to 0, and files existing in the work directory with the new frame names will be overwritten.
When you start a capture sequence the software clears the list of images shown in the "Images" group, and adds the name of the last file created as a line to the list after the corresponding image is created and the file is stored. The file names are created with the template "F_nF_nI.png" where nF is the frame number as specified by the frame counter, and nI is the image number within this frame.
After the capture sequence completes, or the "STOP" button is pressed, only the names of files created during the last capture session are still displayed in the list. If you now press the "Update" button, the list is updated with the names of all the image files existing in the work directory. If you press the "Delete All" button, all the image files in the work directory will be deleted without warning. If you click the mouse on an element of the list, the corresponding image will be loaded from the file and displayed on the leftmost window. You can use the keyboard "Up" and "Down" keys to navigate the list. Pressing the "Previous" and the "Next" buttons in the "Images" group will produce the same effect.
Calibration
The system must be calibrated before scanning is allowed. The calibration process comprises the estimation of the camera intrinsic parameters, the estimation of the turntable axis of rotation and turntable surface plane, and the estimation of the two laser planes.
Camera Calibration
The first step in the calibration process is camera calibration. This is a critical step, and needs to be performed as accurately as possible. You will use the software to automate the image capture process, and you will develop methods to implement camera calibration based on OpenCV within the software.
You will need a checkerboard panel of the proper size to perform this step, and one is provided. This checkerboard has 5 columns, and 9 rows, cell width of 14.8mm, and cell height of 14.8mm. It is a good idea to verify the cell dimensions. A second, smaller, checkerboard is also provided to perform the turntable calibration later on. This second checkerboard has 7 columns, and 9 rows, cell width of 8.85mm, and cell height of 8.85mm. The checkerboard parameters must be entered into GUI before the corresponding calibration step is performed. If the wrong parameters are entered the checkerboard detection step will fail, and as a result the calibration will also fail.
If you want to fabricate your own checkerboard, download the file named checkerboard-7x9.pdf and print it scaled down so that the resulting print fits within the field of view of the camera within the bounds of the turntable. Keep in mind that the turntable diameter is approximately 160mm, and the maximum height of the object to be scanned should not exceed 160mm. If the object is taller it would not feet in the field of view of the camera. Attach the scaled down checkerboard to a flat hard surface such as a piece of acrylic, aluminum, MDF, plywood, or foam board. I built my camera calibration checkerboard using 1/4 inch MDF board, which is a homogeneous engineered material which is quite flat, and does not deform much over time, or as function of weather conditions. Plywood or foam board may deform over time as function of humidity. Acrylic can be a good choice as well. Use a glue stick or some other liquid adhesive to attach the print to the board, and make sure that there are no bubbles. Using scotch tape to attach the print to the board is not a good idea. The resulting checkerboard board should be as flat as possible. Finally, measure the dimensions of the squares. The two dimensions may be slightly different, as a function of the printer design. Do not measure a single scuare. Measure a number of squares in a row, from end to end, and divide by the number of squares in the row. Repeat for a column. If you measure with a ruler, this process will result in more precise measurements.
Select the camera calibration work directory, where you will save the camera calibration images, and delete all the contents, if the directory is not already empty. In my case this directory would be "/Users/taubin/Teaching/3DP/2018/data/laser-scans/calib/camera". In the "Capture" group, specify the two laser to be off. However, read the Laser Plane Calibration section before you start performing this task, since capturing these images either with one laser in "ON" mode you could use the same images to perform camera calibration and laser plane calibration.
You will use the camera capture automation functionality described above to capture the images, even though to perform camera Set the "Count" parameter to the desired number of images. and press the "START" button. To perform camera calibration you do not necessarily have to use the the turntable. The turntable will turn during the capture sequence, but you can hold the checkerboard in your hand so that it is fully visible within the field of view of the camera while the capture automation process continues. If instead you want to capture one image at a time, press the "PREVIEW" button in the "Camera" group. The label of the "START" button in the "Capture" group will change to "ONE". Every time you press this button a file selection dialog will pop-up to allow you to set the name of the file.
You can also design and build a checkerboard stand so that the checkerboard can be placed on the turntable at a steeper angle with respect to the camera. I build another camera calibration checkerboard attached to a small camera ball mount, with the camera ball mount mounted on top of a heavy base. In this way I can change the orientation of the checkerboard, as well as its position on the turntable. Results will be more accurate if the checkerboard is not hold in your hand, but you have to make sure that you have enough images and orientations to cover all the desired work volume. Otherwise large error will result in areas not covered by calibration images.
Once you finish capturing the camera calibration images, perform the camera calibration precedure by first pressing the "Detect" button, and then the "Calibrate" button.
Turntable Calibration
You will use the results of assignment 3 to calibrate the turntable, i.e. to estimate the turntable plane, the turntable axis of rotation, and the object coordinate system attached to the turntable so that the origin of the coordinate system is located at the intersection of the axis of rotation and the turntable plane, the first two coordinate axes span the turntable plane, and the third coordinate axis spans the axis of rotation. Note that if you had not performed the camera calibration before attempting to perform this step, turntable calibration will fail.
Select the turntable calibration work directory, where you will save the turntable calibration images, and delete all the contents, if the directory is not already empty. In my case this directory would be "/Users/taubin/Teaching/3DP/2018/data/laser-scans/calib/turntable".
You also need a checkerboard to perform this step. Use the smaller checkerboard supplied with the scanner. Yo can put it flat on the turntable, or tilted at a certain angle by inserting a small object underneath a border of the checkerboard. Make sure that the checkerboard is contained with the circular boundary of the turntable, so that it does not not hit the stepper motor when it rotates. If the checkerboard moves within the turntable during the capture process, the turntable calibration will fail. You can also use double sided adhesive tape to secure the small checkerboard to the turntable to preven motion.
Check the "Turntable" checkbox in the "Calibration" group. Otherwise the turntable calibration methods will not be execued on the captured images. Enter the numbers of rows and columns, and the dimensions of the checkerboard cells in the "Cols", "Rows", "Cell Width" and "Cell Height" fields. Once these values are defined, and the checkerboard detection algorithm has been run successfully on all the images by pressing the "Detect" button, the "Calibrate" button should be enabled. Once it is enabled, you can press it to estimate the turntable plane, the turntable axis of rotation, and the object coordinate system attached to the turntable so that the origin of the coordinate system is located at the intersection of the axis of rotation and the turntable plane, the first two coordinate axes span the turntable plane, and the third coordinate axis spans the axis of rotation. Once the turntable is calibrated, the 3D View camera model will be updated to display a view of the 3D scen from a point of view similar to the camera's.
Laser Plane Calibration
You will use the results of assignment 3 to perform laser plane calibration. To calibrate the laser planes you need to capture images of checkerboards illuminated by the lasers. As explained in the Camera Calibration section, with one laser on the same images can be used both to perform camera calibration, and laser plane calibration. This is so because, in addition to the camera intrinsic parameters, the OpenCV camera calibration routine also estimate extrinsic parameters for each of the images used in the calibration process.
Select the laser plane calibration work directory, and make sure that it is empty. If you decide to use the same images for camera and laser plane calibration, first select the "Camera" checkbox and run camera calibration on these images, then select "Laser 1" or "Laser 2" and run the corresponding laser calibration method on the same images.
If camera calibration has not been perform, either on a different image set, or on the same image set, laser calibration will fail. The laser calibration procedure must be performed twice, once for each laser.
3D Scanning
Finally, you are ready to start 3D scanning objects. Once the camera is calibrated, the turntable is calibrated, and the laser planes are calibrated, the "SCAN" button in the "3D Scanning" group should be enabled. To scan an object, first select a new work directory, and make sure that it is empty. Place the object to be scanned on the turntable. In the "Capture" group, set the capture parameters. Check the "Blank" checkbox and at least one of the lasers checkboxes. If the two laser checkboxes are checked, the scanning process will result in more 3D points.
Once you press the "SCAN" button the software will run as when you press an image capture sequence, but after each frame is completed it will call your laser detection and triangulation methods to generate additional points. If only one laser is checked, the methods will be called once per frame. If both lasers are checked, then the methods will be called twice per frame. The laser detection method will be called with two images as arguments; one image where both lasers are off, and a second image where one laser is on. You should implement a laser detection algorithm more robust that the one that you have implemented before using the two images.
Submission Instructions
You must upload a Zip file of the complete application, including your implementation of the required functions, in canvas. If you need to communicate with the TA some implementation details, issues, exceptional results, extra work done, or anything else that you want to be considered while grading, do it so by writing a comment in the 'homework4.cpp' file, or add a document in the root folder of the software.
You must also include a folder called 'results' with screen shots of the application showing the reconstructed points, as well as the reconstructed point clouds resulting from pressing the "Save" button ine the "3D Scene" group.