**E27: Computer Vision** **ENGR 027/CPSC 072** **Spring 2021** **[Matt Zucker](../index.html)** | Lecture: | Tue/Thu 2-3:15PM ET on [Zoom](https://swarthmore.zoom.us/j/91558537154?pwd=cSs4c3F2MmtidGtvdXZ4dW1ERnoxZz09) | |-----------------|--------------------------------------------| | Office Hours: | Mon 2:30-3:30PM, Wed 3-4PM ET on [Zoom](https://swarthmore.zoom.us/j/91845438869), or by appointment | | Discussion forum: | | !!! WARNING ***Please note:*** This is a living document, check back frequently for updates! # Overview This class is about applying mathematical theory to endow computers with the ability to understand and interact with the real world through physical imaging sensors. Although there will be plenty of programming, coding is not the main focus. **If you are looking for a class about programming particular APIs (e.g. OpenCV, TensorFlow), you may be disappointed!** The course is divided into three broad areas of investigation: * *Appearance based methods* including filtering, morphological operators, convolutions, frequency domain methods, edge and feature detection, correlation, and template tracking. * *Probabilistic and learning based* methods such as classification, object recognition, and clustering. * *3D geometry based methods* including multiple view geometry, stereo and structured light, and structure from motion. # Resources * Textbook: [Richard Szeliski, *Computer Vision: Algorithms and Applications,* Springer 2010-21.](http://szeliski.org/Book/) * [OpenCV Reference manual](https://docs.opencv.org/master/) * [Numpy Reference manual](https://numpy.org/doc/stable/) * [Linear algebra refresher](../linalg-reintroduction.pdf) * [Python/numpy tutorial from Stanford's CS 231n](https://cs231n.github.io/python-numpy-tutorial/) # Schedule Please note that topics and dates are subject to change. Check this page frequently for updates, assignments, readings, etc. Feb 11, 2021: Introdution Topics: * Introduction * Linear algebra review * Image formation * Image representations Reading/resources: * Chapter 1 * Sections 2.1, 2.3 * [Linear algebra basics](../linalg-reintroduction.pdf) * [Installing OpenCV](install_opencv.html) * [Python notebook: OpenCV Hello World](https://edstem.org/us/courses/4562/workspaces/pVYrZK5WXn9dcd5dYPtQvq7tdHbY8gNs) Assignments: * [Homework 1](homework1.html) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=5d7fab5f-b8bf-44f0-ad71-accc0157e652) * [Whiteboard](notes/e27-2021-02-11.pdf) Feb 16, 2021: Background subtraction Topics: * Thresholding & color segmentation * Morphological operators Reading/resources: * Sections 3.1, 3.2, 3.3 Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=6feae1eb-c7a4-4126-9db2-acd1015b4fe4) * [Whiteboard (reconstructed after original lost)](notes/e27-2021-02-16.pdf) Feb 18, 2021: Filtering Topics: * Morphological operators cont'd * Adaptive thresholding * Convolution Assignments: * [Homework 2](homework2.html) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=ce85f785-e5bf-47a5-8b7d-acd30156511a) * [Whiteboard](notes/e27-2021-02-18.pdf) Reading/resources: * Section 3.2 (Feb 22, 2021): P1: Background subtraction Feb 23, 2021: Filtering & Edge detection Topics: * Convolution vs. cross-correlation * Image gradients Reading/resources: * Sections 3.2, 7.2 Assignments: * [Project 1](project1.html) * [Starter code](https://github.com/swatbotics/e27_s2021_project1_starter) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=41584421-9db4-4af5-b74a-acd8015a4f49) * [Whiteboard](notes/e27-2021-02-23.pdf) Feb 25, 2021: Edge detection Topics: * Canny edge detector Reading/resources: * Section 7.2 * [Python notebook: Filtering demo part 1](https://edstem.org/us/courses/4562/workspaces/pqdR9o0Aycg9pszZq4PReHHtW9UOfYoq) * [Python notebook: Filtering demo part 2](https://edstem.org/us/courses/4562/workspaces/pIQtbrkdAbM1Rlle85za6lT8b9x6vFao) Assignments: * [Homework 3](homework3.html) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=16fb4a7c-38e0-446e-a55f-acda0158cc9f) * [Whiteboard](notes/e27-2021-02-25.pdf) Mar 2, 2021: Frequency domain Topics: * Fourier transform Reading/resources: * Section 3.4, 3.5 * [Python demo](fourier.zip) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=6a537afb-9369-48cf-893d-acdf01579c30) * [Whiteboard](notes/e27-2021-03-02.pdf) Mar 4, 2021: Template matching & template tracking Topics: * Squared distances * Normalized cross-correlation * Review: ordinary least squares * Kanade-Lucas-Tomasi tracker Reading/resources: * Section 9.1 Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=0d1beaf5-3709-422c-9e80-ace10156d9e6) * [Whiteboard](notes/e27-2021-03-04.pdf) Assignments: * [Homework 4](homework4.html) (Mar 8, 2021): P2: Pyramid blending & hybrid images Mar 9, 2021: Machine Learning basics Topics: * Binary classification * Linear classifiers/single-layer perceptron Reading/resources: * Section 5.1 Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=0b54b3d2-fcf4-4638-9fc1-ace601592aed) * [Whiteboard](notes/e27-2021-03-09.pdf) Assignments: * [Project 2](project2.html) Mar 11, 2021: Neural networks Topics: * Multi-layer perceptron * Backpropagation of error Reading/resources: * 5.3 * [Neural networks handout](neural-networks.pdf) * [Python MNIST demo](nnet_demo.zip) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=442ee691-8ead-4dd0-a6a8-ace8015694b2) * [Whiteboard](notes/e27-2021-03-11.pdf) Assignments: * [Homework 5](homework5.html) Mar 16, 2021: Neural networks cont'd Topics: * Dataset practicalities * ML software packages: Keras Reading/resources: * [Keras code demos](keras_demos.zip) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=8bd9d1b2-aea5-4c67-833f-aced0149634a) * [Whiteboard](notes/e27-2021-03-16.pdf) Mar 18, 2021: Unsupervised and non-parametric learning Topics: * Nearest neighbor classification * PCA Reading/resources: * Section 5.2 * [MNIST kNN + PCA demo code](https://github.com/swatbotics/mnist_pca_knn/blob/main/mnist_pca_knn.py) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=8f9ce67b-504d-429b-915a-acef0146a4e8) * [Whiteboard](notes/e27-2021-03-18.pdf) Assignments: * [Homework 6](homework6.html) Mar 23, 2021: Clustering & $k$-Means Topics: * Clustering & $k$-means * Visual bag of words Resources: * [Python demo](vbow.zip) * [Homework problem from 2019 about demo](homework7_s2019.pdf) * [Varma & Zisserman paper](../papers/varma05textons.pdf) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=f9e432d0-065c-4740-b6d2-acf401403464) * [Whiteboard](notes/e27-2021-03-23.pdf) (Mar 25, 2021): Spring break Mar 30, 2021: Deep Learning Topics: * Modern activation functions * Convolutional networks & weight tying * Dropout Reading/resources: * 5.3, 5.4 * [Example code from class](keras_convnet.zip) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=fd3f8d9c-a1a8-4d85-8c0c-acfb01447339) * [Whiteboard](notes/e27-2021-03-30.pdf) Apr 1, 2021: Application: Resnet & GANs Topics: * Resnet * GANs Reading/resources: * 5.5 * Datasets: [CIFAR](https://www.cs.toronto.edu/~kriz/cifar.html), [ILSVRC2012](http://image-net.org/challenges/LSVRC/2012/analysis/#bbox_div), [ImageNet](http://www.image-net.org/challenges/LSVRC/index) * [VGG in Tensorflow](http://www.cs.toronto.edu/~frossard/post/vgg16/) * [*Deep Residual Learning for Image Recognition*, He et al. 2015](../papers/he2015resnet.pdf) * GANs: [Goodfellow et al. 2014](https://papers.nips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf), [2016](https://arxiv.org/pdf/1606.03498.pdf), [Isola et al. 2017](https://arxiv.org/pdf/1611.07004.pdf) * [pix2pix online examples](https://affinelayer.com/pixsrv/) Assignments: * [Homework 8](homework8.html) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=55d77214-8b3a-4396-988b-acfd01489cab) * [Whiteboard](notes/e27-2021-04-01.pdf) (Apr 5, 2021): P3: Image classification Apr 6, 2021: Transfer learning & manifold learning Topics: * ResNet application - fruit fly classification * Transfer learning * FaceNet * Feature visualization Resources: * [Transfer learning in Keras](https://keras.io/guides/transfer_learning/) * [Schroff et al. 2015 - FaceNet](https://arxiv.org/abs/1503.03832) * [Deep dream video](https://vimeo.com/132700334) * [Olah et al. 2017 - Feature Visualization](https://distill.pub/2017/feature-visualization/) Assignments: * [Project 3](project3.html) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=3e97b423-2aeb-496f-b33e-ad0201420a19) * [Whiteboard](notes/e27-2021-04-06.pdf) Apr 8, 2021: Face recognition / ML ethics Recording/slides: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=aa9441fe-6ab4-4efd-ae37-ad040144fafd) * [Slides](https://docs.google.com/presentation/d/1-24TFYCiidV2dnno6QFXrXdK0iBN6s5ncu53CSFSaIg/edit?usp=sharing) Assignments: * [Homework 9](homework9.html) Apr 13, 2021: Projective algebra fundamentals Topics: * Points in 2D * Lines in 2D * Homogeneous coordinates Reading/resources: * Sections 2.1, 3.6.1 * [Application: unprojecting text](https://mzucker.github.io/2016/10/11/unprojecting-text-with-ellipses.html) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=4be2a299-7c82-4be6-8145-ad0901480fc9) * [Whiteboard](notes/e27-2021-04-13.pdf) Apr 15, 2021: Homographies and homogeneous least squares Topics: * Homographies * Homogeneous least squares Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=a863755e-fc18-4f84-a698-ad0b0147dfdb) * [Whiteboard](notes/e27-2021-04-15.pdf) Assignments: * [Homework 10](homework10.html) (Apr 19, 2021): P4: Image stitching with homographies Apr 20, 2021: 3D geometric fundamentals Topics: * Homography recap * Rigid transformations Resources: * [In-class exercise on homographies](homography_exercise.html) Reading/resources: * Section 11.1 Assignments: * [Project 4](project4.html) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=2103d8b5-a321-4652-bb22-ad1001441ec2) * [Whiteboard](notes/e27-2021-04-20.pdf) Apr 22, 2021: Geometric image formation Topics: * Image formation redux * Intrinsic & extrinsic parameters Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=8c27944d-bf50-4faa-abaa-ad1201409e5b) * [Whiteboard](notes/e27-2021-04-22.pdf) Assignments: * [Homework 11](homework11.html) Apr 27, 2021: Camera calibration Topics: * Triangulation * Camera calibration * Algebraic vs geometric error Reading/resources: * Section 11.1 Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=b7e914dc-cade-474a-bf43-ad170143cc45) * [Whiteboard](notes/e27-2021-04-27.pdf) Apr 29, 2021: 3D imaging Topics: * Stereo vision * Structured light Reading/resources: * Sections 11.2, 12.3 * [Technical description of Kinect calibration](http://wiki.ros.org/kinect_calibration/technical) * [Lens distortion](https://www.edmundoptics.com/knowledge-center/application-notes/imaging/distortion/) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=924223bd-af4c-44f0-a945-ad19013ecc63) * [Whiteboard](notes/e27-2021-04-29.pdf) Assignments: * [Homework 12](homework12.html) * [Final project](finalproject.html) (Apr 30, 2021): Final project May 4, 2021: Feature detection & description Topics: * Feature detection * Feature matching * Random sample consensus (RANSAC) Reading/resources: * Section 7.1 * [FAST: Rosten et al. 2006](../papers/rosten2006fast.pdf) * [BRIEF: Calonder et al. 2010](../papers/calonder2010brief.pdf) * [ORB: Rublee et al. 2011](../papers/rublee2011orb.pdf) Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=9351be87-3713-45fc-b3cf-ad1e0142b385) * [Whiteboard](notes/e27-2021-05-04.pdf) May 6, 2021: Structure from Motion Topics: * Structure from Motion Reading/resources: * Chapter 11 Recording/whiteboard: * [Recording](https://swarthmore.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=1c8552d3-a2e5-4992-bcf3-ad2001454a71) * [Whiteboard](notes/e27-2021-05-06.pdf) May 20, 2021: Last day to submit graded work # Requirements **Prerequisites:** Either [ENGR 019](http://catalog.swarthmore.edu/preview_course.php?catoid=7&coid=8092) or [ENGR 056](https://catalog.swarthmore.edu/preview_course.php?catoid=7&coid=63600), or permission of the instructor. [MATH 027](https://catalog.swarthmore.edu/preview_course.php?catoid=7&coid=8463) or [MATH 028](https://catalog.swarthmore.edu/preview_course.php?catoid=7&coid=8464) is recommended. **Skills:** In practice, I expect you to be conversant in elementary programming concepts in Python, including basic loops, functions, and array processing. You should also be comfortable with the process of converting a set of mathematical equations into a working Python program. I also expect you to be comfortable with [linear algebra concepts](../linalg-reintroduction.pdf) such as solving linear systems, matrix inverses, rank and eigenvalues/vectors. We will also be using related geometric concepts such as the dot product and vector norms as well as rotations and translations. **Time:** I expect students to spend approximately 8 hours per week on this class (4 classes × 8 hours per class + [8 hours for paid student work](https://www.swarthmore.edu/student-employment/employment-faqs) = 40 hours). Although this figure will vary from individual to individual and week to week, you should plan to commit several hours outside of class to homework, reading, and projects each week. **Discussion forum**: We will use [this Ed forum](https://edstem.org/us/courses/4562/discussion/) throughout the semester to communicate course announcements and answer questions. Please use Ed (instead of just emailing me) for all course-related communications -- this allows students to see common problems and to engage in discussions about course material. **Wizards**: The course Wizards are Leia Rich and Megan Strachan. The class will have a weekly Wizard session on Tuesdays at 7PM Eastern. # Assignments Homework consisting of math, short answer questions, and small programming exercises will be assigned roughly weekly. Typically, homework will be assigned on Thursday, and be due online before the start of class the following Thursday. We will also have regular quizzes to be completed each week. Each quiz will generally be based on the homework due the previous week, and potentially course material prior to that as well. There will be several regularly scheduled projects as well as an open-ended final project. Ideally, projects will be completed by pairs of students, although I am willing to allow singletons if difficulties arise due to time zones or technological limitations. Grading will follow approximately the divisions shown below: * Homeworks: 35% * Weekly quizzes: 20% * Regular projects: 35% * Final project: 10% The majority of your homework grade will be based upon completion, not correctness. Each student may miss one homework or one quiz with no penalty; if a student completes all homeworks or quizzes, I will drop the lowest grade in that category. Although we will not have a final exam, I may administer one final quiz during the exam period. We will use our scheduled final exam time for short final project presentations. ## Collaboration and attribution * Feel free to collaborate with your classmates on homework; however, you must submit your own work. Duplicating others’ assignments verbatim (especially code!) is prohibited. * If you do discuss homework with your classmates, I expect you to disclose any such collaboration clearly in your submitted work. Err on the side of caution – it’s the best way to avoid awkward conversations about suspicious similarities between assignments. * Cite any external sources used, including the textbook, internet, discussions with other professors, etc. * Collaboration or communication with other students about quizzes is prohibited. * Aside from raising technical and procedural questions on the course discussion forum, do not collaborate on projects with others besides your partner. * Do not post homework, quiz, or project solutions online. Questions or answers that discuss solutions too closely will be deleted. Aside from the course-specific policies above, you are expected to understand and abide by the college's [policy on academic misconduct](https://www.swarthmore.edu/student-handbook/academic-policies#academic_misconduct). ## Late policy, extensions, etc. I expect students to do their best to complete all graded work on time, but I also understand we are in a global pandemic. If you need an extension or some other type of support due to extraordinary circumstances during this remote semester, please let me know and I'll do my best to figure out a plan with you. In the absence of any communication about extensions, grade penalties for unexcused late work will be assessed as follows: |Work type | Credit up to one week late | Credit after one week late | |----------|----------------------------|----------------------------| |Homework/quiz* | 50% | 0% | |Projects | 75% | 50% | *Students are allowed one free late homework submission and one free late quiz sumission (up to one week each) with no grade penalty. My philosophy in providing no-penalty contingencies for both missing and late homeworks and quizzes is to recognize that unforseen events happen and to provide some slack for when they do. Accordingly, I expect students to take advantage of these built-in opportunitites before I grant extensions or other extraordinary support. Swarthmore has a hard deadline for professors to submit grades after the end of the semester, which means that without special permission, all student work must be submitted to me by the end of finals on May 20 to count for a grade. And remember, the more notice and communication I have from you about your academic needs, the more help I can provide! # Software You will need an up-to-date installation of Python 3 along with OpenCV 4. It's available through multiple distribution channels. See [this document](install_opencv.html) for detailed installation instructions. OpenCV is also installed on the CS cluster machines. Make sure to run `python3` from the command line when you want to use it. # Accommodations If you believe you need accommodations for a disability or a chronic medical condition, please email Student Disability Services at studentdisabilityservices@swarthmore.edu to arrange an appointment to discuss your needs. As appropriate, the office will issue students with documented disabilities or medical conditions a formal Accommodations Letter. Since accommodations require early planning and are not retroactive, please contact Student Disability Services as soon as possible. For details about the accommodations process, visit the Student Disability Services website. You are also welcome to contact me privately to discuss your academic needs. However, all disability-related accommodations must be arranged, in advance, through Student Disability Services.