CDT 450: Computer Visions: Generative Media

Spring 2018


Music we generated in class


Machine learning transformed our reality. On a daily basis, computers are making vital and consequential decisions that affect millions of human lives, in split seconds, at immense scale, and almost without human supervision. The transformative power machine learning can unleash is just being discovered, but in our class we will look at the byproducts of utilitarian machine learning. In class and home assignments we will see how computers can learn to "paint" and "sketch", "write" a poem or advertisement copy, transfer visual artistic style (Van Gogh-ize a picture), hallucinate, "compose" music, and even - come up with recipes for surprising dishes (that we will cook and bring to class - mmm data-driven food... yum!) The class will lightly cover topics in applied mathematics for machine learning (algebras, calculus, probability theory, algorithms, etc.) but will focus on hands-on practical programmatic methods to implement computer visions in Python. Students will learn how machine learning (and specifically deep learning) models are constructed and can be used to generate information in multiple media (text, sound, picture, food :) and by the end of the course will be able to demonstrate these capabilities. 

Location, Time, Office Hours, Piazza

New Computer Science building, room 115, TuTh 1:00p-2:20p

My office hours will be held: TuTh 2:30-4:00p in room 145. Make an appointment using the Google scheduler.

Find our Piazza page at:
On Piazza we will be posting assignments, slides, readings, notebooks and discussions.

Timeline / Syllabus

Tentative timeline / syllabus below. Expect significant changes as the course rolls along.

Week Class Topics Assignment
1 1 Introduction: Course intro. Math primer.  
  2 Introduction: Applied machine learning concepts. Practicalities: Programming. HW0: Hello Machine-Learning World!
2 1 Introduction: Features, Working with Data.  
  2 Introduction: Linear models. Neural networks. Generative models. HW1: My first Neural Network
3 1 Images: Intro: Pixels, cameras. Computer vision.  
  2 smiley HW2: Fooling CNNs
4 1 Images: Fooling CNNs with Gradient Descent  
  2 Images: Style transfer. HW3: Fooling CNNs + style transfer
5 1 Images: Inception net. Visual hallucinations / DeepDream  
  2 Images: Auto-encoders and Variational auto-encoders (VAE). HW4: Visual hallucination
6 1 Images: Creative generative adversarial networks (GAN).  
  2 Images: Pix2Pix, CycleGAN HW5: VAEs + GANs
7 1 Music: Intro: Notes as characters, accents, duration.  
  2 Music: Simple predictive, generative, Markovian models for solo melodies. HW6: ETL
8   Spring break smiley
9 1 Music: Embedding, Representation, Recurrent Neural Networks (RNNs).  
  2 smiley HW7:
10 1 Music: Recurrent architectures for Ragtime music generation.  
  2 Music: Google Magenta tools. HW6+7+8: Ragtime, MelodyRNN
11 1 Text: Intro: Natural Language Processing  
  2 Text: Recurrent models for text. Char RNNs. HW9:
12 1 Guest lecture: Noah Weber - SBU NLP lab  
  2 Text: LSTMs, GRUs, Bidirectional layers and 1D convolutions. HW10:
13 1 Text: Variational auto-encoders for smooth text interpolation.  
  2 Text: Sequence-to-sequence, text style-transfer HW11:
14 1 Machine Learning for Art  
  2 Class Review HW9+10: CharRNNs, Text-VAEs
15 1 smiley  
  2 Class Review: Final exam prep  


  • Programming level: 9000   :)
  • Students are expected to have a working knowledge of scripting with Python (or another equivalent scripting environment).
    • We will be using: TensorFlow, Keras or PyTorch, Numpy, Pandas, Scikit-Learn, Jupyter notebooks.
    • Working in Processing is considered an equivalent scripting language. If you have a firm grasp of Processing, you should be fine "graduating" to Python, which I strongly recommend doing regardless of this class.
  • Students are expected to have a basic knowledge of network/internet (HTTP, URLs, etc.) and command-line scripting.
  • No formal college-level education in applied maths is required - but highly encouraged.
  • No formal education in the arts or music is required - but highly encouraged.

This will be a hands-on, practicum-style class with weekly assignments that will require coding. It'll be an exhilarating ride through the wonders of high-end state-of-the-art machine learning software, hardware and constructs, with the goal of harnessing this power towards making irregular media renderings. Expect to do a considerable amount of learning-by-doing and self-learning outside of the classroom.

Recommended Literature


Our currency is the S-BitCoin (Stony Brook-it Coin). Your goal in this semester is to earn 1 S-BitCoin. The algorithm is based on proof-of-work, and the work is: assignments, which earn you 33 mili-S-BitCoin, final exam, which earns you 500 mili-S-BitCoin, and 1 class attendance, which earns you 3.8 mili-S-BitCoin. S-BitCoin is expected to be very valuable one day...

  • Assignments:
    • Assignments will be 40% of the final grade.
    • Students are expected to complete all assignments. More than 4 missing assignments will award a failing grade in the course.
    • Late Assignments. Assignments will be accepted up to 48 hours late. A 10% penalty will be assessed if it is less than 24 hours late, while a 25% penalty will be assessed if it is between 24 and 48 hours late. Any assignments submitted after 48 hours from the deadline will earn a 0.
    • Submission will be via Blackboard upload.
    • Assignment submission is done individually: One Student One Submission [Satoshi Nakamoto]. However, you are encouraged to collaborate (and then make individual submissions).
  • This class will not have a mid-term exam.
  • Final:
    • The exam will be 50% of the final grade.
    • Students are expected to complete and pass the final exam. Missing or failing the final will award a failing grade in the course.
    • The final exam will be held on: Monday May 14, 5:30p - 8:00p.
    • The exam will be given electronically via Blackboard.
  • Attendance: will be kept and will make up for 10% of the final grade. However, there will not a cap on the allowed number of missed classes.
  • Grade scale (assignment and exams): A+ [100, 97); A [96, 94); A- [93, 90); B+ [89, 87); B [86, 84); B- [83, 80); C+ [79, 77); C [76, 74); D [73, 63). Grading is not curved.

Class Policies

My Policy

  • Be in class on time. Doors will close at 1:10p. Latecomers will be frowned upon, but if tardiness persists - action will be taken.
  • Respect your fellow students: Don't be loud, let others have their say. Absolutely no bullying! cyber- or any other way. There will be zero tolerance for bullyish behavior that will result in reporting and failing class. Piazza will not allow anonymous submissions.
  • No mobile phones or food, please. Our classroom doesn't allow food in it anyway.
  • No cheating. If you're going to cheat - don't get caught. If you get caught - you fail the [assignment/exam] and then reported.
  • Grading and re-grading policy: I grade very generously, and re-grade very stringently. If you're unhappy with your [assignment/exam] grade, consider if you want me to regrade your submission, since I'll be much more strict and re-grade the entire thing from scratch.
  • Coding: Write readable code please. Add # comments, use sufficient spacing, give variables a meaningfulName, use camelCase or snake_case, code DRY not WET, exfoliate dead code (remove your commented-out code, cull dead branches).

University Policies


Disability Support Services (DSS):

If you have a physical, psychological, medical or learning disability that may impact your course work, please contact Disability Support Services, ECC (Educational Communications Center) Building, room 128, (631) 632-6748. They will determine with you what accommodations, if any, are necessary and appropriate. All information and documentation is confidential.

Students who require assistance during emergency evacuation are encouraged to discuss their needs with their professors and Disability Support Services. For procedures and information go to the following website:

Academic Integrity:

Each student must pursue his or her academic goals honestly and be personally accountable for all submitted work. Representing another person's work as your own is always wrong. Faculty are required to report any suspected instances of academic dishonesty to the Academic Judiciary. For more comprehensive information on academic integrity, including categories of academic dishonesty, please refer to the academic judiciary website at

Incident Management:

Stony Brook University expects students to maintain standards of personal integrity that are in harmony with the educational goals of the institution; to observe national, state, and local laws and University regulations; and to respect the rights, privileges, and property of other people. Faculty are required to report to the Office of Judicial Affairs any disruptive behavior that interrupts their ability to teach, compromises the safety of the learning environment, or inhibits students' ability to learn.