Educator notes

It’s advisable that before starting this project, your students and yourself are all ready familiar with the Art Programming Basics.

Key concepts for this project

Using external libraries
Importing video files or the webcam-stream to Processing
Going through the pixels of the video with two for-loops
Converting the colors of the pixels.
Editing the colors and shapes of the pixels to create interesting results.

Learning Objectives

The student...

...can implement previous knowledge acquired from the Basics-module.
...can import video-files or webcam-stream to Processing.
...understands the basics of for-loops
...can go through each pixel and check its' color
...can change the color to grayscale.
...can play with different color conversions creatively.
...can draw different shapes in the places of the pixels.

Introduction

Transcript

Hey! I'm Alex from Mehackit. in this project you learn to add cool effects to your own videos. Something like this: Okay but how to do these things? When you add effects or filters to a video, what does actually happen? Well, videos consist of a series of images and an image consists of really small dots called pixels. Every pixel has its' own place and color. One dot doesn't look like much but together all the dots form an image and the images form the video. To make all sorts of cool effects to an image or video you need to tinker with the colors and sizes of these pixels and it's easy to do all of this in Processing. In this project you'll learn to access the pixels and do whatever you want with them. You can use your own video and make several versions of it or it's also possible to use the stream of a webcam if you like. After you've learned to access the pixels it's possible to use these skills to detect motion, objects, humans or other things. That's something we won't do during these videos but there's some hints for you to look into at the end of the project. So let's begin!

Loading the Video

Transcript

As the first step, I'm going to add a video to the canvas. To load, display and access the pixels of a video in Processing you need to use a library. A library is a bunch of code that helps you to do things without first having to write all the code yourself. In this case, the library tells Processing how to deal with video files, how to read the pixels, how to display them on the canvas and more. The library you will use is called Video and it's made by the Processing foundation. To use the library go to the Sketch-tab and hover over "Import library". In this case the library is already on my computer. If it's not installed yet, select add library. Type "video" in the search bar and find the right library made by the Processing foundation. If you click "Install", it will install the library to your computer. Now that the library is on my computer, I can add it to the project. I go back to sketch and select "Import library" and Video-library. This creates a line at the top of my program that says "import processing dot video". Next, I will create a movie object. This object is part of the video library and will handle all the things related to the video I'm going to use. Then I want to import the actual video-file to the project. There's an example video that you can download on this page. I'm going to use it now. You can also record your own video or even use the webcam-stream if you like. There's more instructions about utilizing the webcam stream in the next video. Adding the video is similar to adding images. In the Sketch-menu, press "Add file". This gives you a dialog where you can select the right video file. This creates a data folder in the same folder as the project and copies the video file there. To also load the video into our project, you need to load it into the movie-object. Let's do this in setup. When creating movie objects you have to add the word this in the parentheses. It's a reference to the current program you are making. After that there's the name of the video file. After loading the video file I want to set the size of the canvas to the size of the video. Iin Windows, you can see the size by clicking the video with the right mouse button and selecting Properties and "Details". This gives you the video width and video height. On a Mac, the size of the video is displayed as dimensions when right-clicking the video in Finder and selecting "More Info". I'm going to set the video to loop with the loop-command. This will start playing the video and keep looping it as long as the program is running. The first thing I need to do in the draw function is to check whether the video is ready and Processing can use it. Without this there might be all kinds of crashes, bugs and other issues. I have to update the video every frame using the read-command. This command will keep track of what frame to show. Now it's time to draw the video to the canvas. Video is just the series of images shown very fast after each other and in Processing, it's possible to draw the frames of the video with the image -command. It works! Now there's an unmodified video playing on the canvas So, the read-command always gets the next frame of the video and the image-command draws it. Now that the video is there, I can start playing around with the pixels!

Do this

Install the video-library as instructed in the video.

Film a video yourself and send it to your computer or use the example-video used in these tutorials. You can download the example video here.

Add your video to the canvas. There's example code below.

import processing.video.*;

Movie myMovie;

void setup() {
  size(640, 480);
  myMovie = new Movie(this, "example.mp4");
  myMovie.loop();
}

void draw() {
  if (myMovie.available()) {
    myMovie.read();
    myMovie.loadPixels(); 
  }
  // Original video in the top left corner
  image(myMovie, 0, 0);
}

What is a library in the context of coding?

A place where you can borrow programming literature.

It's a brand new programming language.

An extension to some programming language.

Using the Webcam

Transcript

Besides importing a video file to Processing it's possible to use the livestream from your webcam. I'll briefly show you how. You can copy the code below the video. This code here is similar to the one where you loaded the video file but instead of using the Movie-object, there's the Capture-object. This part here makes a list of available cameras. If the computer doesn't find any cameras it prints the text "no cameras found" to the console. Otherwise it will use the first camera from the list. In the draw function, it does just the same as with the video file. If the camera stream is available, it reads the next frame and draws it to the canvas. The width and height here are used to make sure that the video is displayed at the same size as the canvas. I'll run this. Hahah, there I am! If you have a camera and get it working you can use it in this project to make funny images of yourself!

Do this

If you want, you can continue with this project using your webcam-stream instead of using a video-file.

The code in the video is provided below.

import processing.video.*;
Capture cam;

void setup() {
  size(640, 480);
  String[] cameras = Capture.list();
  if (cameras.length == 0) {
    println("No cameras found.");
    exit();
  } 
  else {
    cam = new Capture(this, cameras[0]);
    cam.start();
  }
}

void draw() {
  if (cam.available()) {
    cam.read(); 
    image(cam, 0, 0, width, height);
  }
  
}

Access the Pixels

Transcript

Hi! When you have the video running on your canvas, you can start altering the colors. Let's continue with the code that draws the video to the screen. I want to draw the new version next to the original one so there will be two videos on the canvas. I'll set the size of the canvas to twice the width of the video. Do you remember what pixels are? Those tiny colored dots that form the image and the video. To change the colors you need to go through all of the pixels of the video, check the color values and change them. So how to do this? Let's start with the command that gives access to the individual pixels of the video. Then, I'm going to create two for-loops. If you don't remember what for loops are, don't worry. You'll understand more through this example and there's extra information below this video. The first for-loop goes through the y-axis of the current image. Now, pay attention! I'm going to put the second for-loop inside the first one and it goes through the x-axis. These two for-loops go through all the pixels of the of the im age. I'll show you how! The first loop going through the y-axis we'll start at zero. Then the loop going through the x-axis will start and keep going until x hits the width of the video. When the x-axis loop has finished the y-axis loop will increment by one and the x-axis loop will run again. This will continue until the loops have gone through all of the pixels together. The variables x and y always keep track of the coordinates of the current pixel. Next, I want to check the color of that pixel and change it. But how do I do this? There's a command to get the color value of a pixel and it's called "get". I save this color value to a variable. There's a datatype called "color" for this. Next, I want to change this pixelColor. The pixelColor has three values: the red, green and blue components. I can get for example the red component with an intuitive command. I only have to clarify to Processing that I want to use the red value as a color instead of just a number. Just like I can use the get-command to get the color from a pixel I can use the set command to set the color of a pixel. Let me run this. I can't see the changes I've made because I'm drawing the original video on top of the changes I've made. To draw the two-colored video next to the colored one I have to add the width of the video to the x-position. Now the red component of the original color is used as a grayscale color directly. The more red the original color is, the whiter it appears in a new version. The red is the most dominant color in the original video and that's why the new version is mostly white. As the canvas will be full of videos eventually it's good to write some notes here to clarify which commands are creating which video. I'll write a comment to remind myself that these commands create the video in the top right corner. And I'll do the same for the original video. Now it's your turn to make your first color conversion. There are some tips below this video.

Do this

Create the two for-loops that go through the pixels of the video.

There’s more info about for-loops here and in the topic Array Storm in the fourth chapter of the basics-module.

Change the colors of the pixels like instructed in the video. Create a new version of the video with altered colors and draw it next to the original video. There’s some ideas for color conversions below.

// This is the example conversion in the video. 
// Amount of red in the original color is used as the new color directly.
// Try using green and blue -commands instead of the red-command. 
color newColor = red(pixelColor);

// Amount of blue in the original color will be the amount of red in the new color. 
// Similarly, green will become red and blue will become green.
color newColor = color(blue(pixelColor), red(pixelColor), green(pixelColor));

// Try out different mathematical operations for the components of the color!
color newColor = color(red(pixelColor)/2, green(pixelColor) * 2, blue(pixelColor) + 40);

color newColor = red(pixelColor)

What does the line of code above mean?

The amount of red in the pixelColor will be saved to variable newColor and used as a grayscale-value.

The amount of red in the pixelColor will be saved to variable newColor and used as a the amount of red in the new color.

The amount of red in the pixelColor will be saved to variable newColor and used as the opacity of the color.

color newColor = color(green(pixelColor), blue(pixelColor), red(pixelColor))

Try the color conversion above. What happens?

The code doesn't change anything.

The code converts the color to a grayscale-color.

The code flips the rgb-values. Red will become green, green will become blue and blue will become red.

Tinker the Colors

Transcript

Welcome back! Now you get to do some more experiments with the colors. Let's turn the video to a more realistic black and white version. Creating black and white images from colored images is a technique often used in object recognition, motion detection and other camera vision projects. The command red gives you the red component of the color and there's the same commands to get the blue and the green components. To convert the color to a grayscale value I'll add up all the three values and divide the result by three. There it is! Now it calculates an average color from these three components. Tinkering with the divider gives some interesting results. Another way to do the same thing is multiply all the individual colors by 0.33 If I run this the result should be exactly the same. However, instead of dividing all the colors by three this also gives me control over the individual colors. So I can for example decide to amplify the green of the image and make blue and red weaker. Besides making this grayscale-conver sion, it's possible to just flip the colors using the green component as the amountof red, the blue component as the amount of green and so on. As you can see, I'm not adding all the color values and using one grayscale value but I've added commas to separate different R, G and B-values of the new color. I'm sure you already have some more ideas on how to convert the colors of your video. Time to try some of those out!

Do this

Try out some more color conversions! There’s examples below.

// The grayscale-conversion showed in the video.
// Each component of the color is multiplied by 0.33 to get an average grayscale-color of the pixel.
color newColor = color(red(pixelColor) * 0.33 + green(pixelColor) * 0.33 + blue(pixelColor) * 0.33);

// The example conversion showed in the video.
// Components of the rgb-color change places.
color newColor = color(blue(pixelColor), red(pixelColor), green(pixelColor));

// If the color is red enough, it is displayed normally. Otherwise, it is displayed as black.
// Try altering the threshold value and checking the green and blue components instead of checking the red-component.
color newColor;
if(red(pixelColor) > 200){
  newColor = pixelColor;
}else {
  newColor = color(0);
}

color newColor = color(red(pixelColor) * 1.2, green(pixelColor) * 0.8, blue(pixelColor) * 0.8);

What does the color conversion above do?

It amplifies the red component and makes the green and blue compoenents weaker.

It amplifies the green component and makes the red and blue compoenents weaker.

It amplifies the blue component and makes the green and blue compoenents weaker.

Rectangular Pixels

Transcript

Previously you changed the colors of the video and now you g et to change the sizes and shapes of the pixels as well! You are about to add another version of the video. So let's create an even bigger canvas to make more space. The for-loops now going through every pixel of the video: the x and y always increase by one. I'm going to take bigger steps in this example. Above the setup I will declare a new integer that will determine the size of these steps. I'll use this variable in the for loops. So now the computer will take jumps of 10 pixels while going through the video. I don't want to change the colors this time so I just choose the color of the pixel as a filling color and draw ellipses in the place of it. I'll use stepSize as the size of the ellipse. I want to add the video to the bottom left corner of the canvas so I always add the height of the video to the y-coordinate. Cool! What happens now is that the computer takes steps of 10 when going through the pixels. Then it checks the color of that pixel and draws an ellipse in the place of it. To fill up the blank space between the pixels, the size of the ellipse is 10 as well, the same as the stepSize. But what happened to the previous video? There are some tiny dots but a lot of white space between them. Do you know why? The computer now takes steps of 10 pixels but this set-command just sets the color of one pixel on the canvas. I replace the set-command with a shape. Let's use rectangles this time. And I use the newColor instead of the same pixelColor in the fill command to give the rectangles a different look. The ellipses and rectangles crete quite different atmospheres. If you look closely you'll notice that the shapes have a thin black outline though. I'll remove the outlines of these shapes with the noStroke-command to make the video look more uniform. I'll put it in setup because I only need to set the noStroke- command once. Now the outlines of the shapes are gone. The pixels are now bigger than in the start. But how to enlargen them even more? Now the computer takes even longer steps and the pixels become bigger. In addition to increasing the stepSize we can also make the shapes bigger to make them overlap. Let's try this with the circles. You can now play around with all the different sizes, stepSizes, shape sizes and play some more with the different colors to see different effects.

Do this

Follow the instructions in the video and create two new versions of the original video.

Try out different colors and shapes for the pixels.
Try out different values for the variable stepSize. What is the highest possible stepSize you can use while still understanding what’s happening in the video?
In the video, ellipses and rectangles are used in the place of the pixels. Try triangles instead!

See the code-example in the video. Which of the following statements is true?

The greater the variable stepSize is, the more pixels the computer checks.

The greater the stepSize is, the less pixels the computer checks.

The value of stepSize doesn't make a difference. Computer checks all of the pixels anyway.

Extra: Flowing Pixels

Transcript

You have drawn the video using shapes like ellipses and rectangles and changed the colors too. Next, you'll learn how to make the sizes of these shapes vary over time. There's now three versions of the video and I will make a fourth one. If your canvas is full of different versions of the video already, you can replace an older version or make the canvas bigger of course. The size of the pixels will vary over time. To notice the changes, I'll need to refresh the background after each frame. I'll draw a background rectangle behind all of the videos. I'll make it white and draw it after checking if the movie is available. Then, I'll make the size of the ellipses random. I create a variable and store a random value there. The maximum value is the stepSize because otherwise the shapes start to overlap with each other. I use this variable as the size of the ellipse. And I want to draw the ellipses to the bottom right corner so I have to add the width of the movie to the x- coordinates and the height of the movie to the y-coordinates. Okay, there it is. But the effect is quite chaotic. With the random function the size of the circles is jumping around very wild and can jump from ten to zero and back in two frames. The values have no relation to each other. I want to find a way to get random numbers that do behave like they fit better with the numbers before and after. To achieve something like this processing has a command called "noise". Noise is a bit like random but instead of completely random value it uses a mathematical method to guide the numbers a bit. With noise the circles will grow or shrink only a bit every frame. I'll comment this random- part out for a while and start with the noise-command. The noise-command creates a random number based on the input-data If the input changes the result changes as well. So there has to be a value that changes over time. Let's make a variable for that. I'll create it in the top of the program. I'm going to increase the timeCounter-variable a tiny bit every frame. I'll throw this variable inside the noise-command. Besides this number you have to enter some unique data for every pixel. Luckily we have the x- and the y-coordinates of every pixel we go through available. Let's first see what kind of values we get out of this noise-command. It seems that these are slowly changing decimal numbers between zero and one. So what to do with this number? It's hard to use numbers this small as the size of the ellipse directly. So let's multiply the noise with the number and use that as the size. I'll use the number 30. I'll uncomment this previous code and replace the random-command with the noise-command There's an effect but it's a bit chaotic still. Right now the noise-command has three input values: timeCounter, x and y. The more these input values change, the more the number generated by the noise function changes too. To make the changes smoother, I'll multiply x and y by a small decibel number. Cool! It's like there would be some kind of waves going through the picture. This was just the first example of playing with noise. The noise-effect isn't the easiest thing to understand. If you are eager to know more about how it actually does the calculation, there's more info in the Learn More -section of this project. And you'll learn more in the next video where I'll show you how to make the colors flow in a similar way!

Do this

Follow the instructions in the video and create a new version of the original video.

First, make the size of the pixels random.
Then, make the size of the pixels change more smoothly with the noise-command
Make some experiments with the noise-command. Change the decimal-multipliers of the parameters. There's some examples below.

EXTRA

If you're eager to know more about noise, check this video made by Daniel Shiffman!

// This is the example in the video.
float circleSize = noise(0.02 * x, 0.02 * y, timeCounter) * 30;

// Change the decimal multipliers to get different results.
// The more the parameters of the noise-function vary, the more noise there will be.
// Lower multipliers make the change smoother
float circleSize = noise(0.01 * x, 0.04 * y, timeCounter) * 30;

// You can use noise with two parameters too. 
float circleSize = noise(0.02 * x, timeCounter) * 30;

// You can use noise with only one parameter too!
// If you use x or y as the only parameter, the result is static.
// That is because x and y go always go through the same values.
float circleSize = noise(timeCounter) * 30;

Extra: Flowing Colors

Transcript

Using noise gives your creations a more natural feel than total randomness. Now, let's also change the colors of the pixels with the same idea. Our canvas is full already. So I'll replace the third version of the video in the bottom left corner with this new one. I want to take the three components of the color, change them a bit and finally draw the pixel with the new color. I'll start with the red component. I'll use the current red component and the timeCounter as inputs of the noise-command. To make the changes smoother, I multiply the red component with a small decimal number. Remember that the noise-command always gives us a number between 0 and 1. To get the new red component of the color, I multiply the maximum value of the color, 255, with the result of the noise-command. And I save that number to a variable. I'll do the same for the green and the blue components. Then I create a new color from these components. I'll save it to a variable and use it as a filling color. I'll draw the new version to the bottom left corner. I'll draw it with rectangles this time. The colors are flowing! And by changing these decimal values you can make the effect smoother or more chaotic.

Do this

Follow the instructions in the video and create a version with flowing colors. Make some experiments by affecting the components of the colors with the noise-command.. There’s some examples below.

EXTRA

If you’re eager to know more about noise, check this video made by Daniel Shiffman!

// This is the example in the video. noise is used with two parameters.
float noiseRed = noise(red(pixelColor) * 0.02, timeCounter) * 255;
float noiseGreen = noise(green(pixelColor) * 0.02, timeCounter) * 255;
float noiseBlue = noise(blue(pixelColor) * 0.02, timeCounter) * 255;

// Use the brightness, hue and saturation -commands to get interesting results.
// Now, saturation of the color affects the amount of red, hue affects the amount of green and brightness affects the amount of blue.
float noiseRed = noise(saturation(pixelColor) * 0.02, timeCounter) * 255;
float noiseGreen = noise(hue(pixelColor) * 0.02, timeCounter) * 255;
float noiseBlue = noise(brightness(pixelColor) * 0.02, timeCounter) * 255;

Summary

Transcript

You started with the plain video and have now probably made at least three different versions of it. You learned how to go through the pixels of a video with two for loops, save the color of a pixel to a variable and change it. You also drew different kinds of shapes in the places of the pixels. Now, let's see how you could still develop your project further. If you didn't try your webcam yet, now is a good time for that. You can find the code for using the webcam-stream in one of the previous videos. Wouldn't it be cool to have this kind of project in the hallway of the school where people could see these different versions of themselves as they are walking past? in this project you have only focused on the videos but the video doesn't have to be the only thing on the canvas. You can supplement the videos with your previous Processing-projects to make it more exciting. You learned to go through all the pixels of the video but it's possible to just process parts of them by altering the conditions of the for-loops. This gives you a chance to create different kinds of mirroring effects. Once you have learned to process pixels there's tons of things you can do. The Processing IDE includes a lot of interesting examples worth checking out. Remember that the video just consists of a series of images so whatever effects you are able to add to a video you can also use in still images. Now, spend some time experimenting with your project. There are some more links and tips below this video. And remember to save your project! If you feel like you want to continue with it afterwards you don't have to start from scratch.

Learn More

Transparency of Pixels

A RGB-color has three components, and you can use a fourth one to determine the opacity of the color. Try making some pixels partly transparent. You can e.g. use the amount of red as the opacity.

Saturation, Hue and Brightness Saturation, hue and brigtness. It's possible to check the hue, brightness and saturation of the color in Processing. Like all other color-values, these are numbers between 0 and 255 so you can use them as color-values directly.

color newColor = color(brightness(pixelColor), hue(pixelColor), saturation(pixelColor));

Motion detection?

Check this tutorial by Daniel Shiffman: https://www.youtube.com/watch?v=QLHMtE5XsMs

How noise works?

Check these examples in Processing.org about 1D, 2D and 3D noise. https://processing.org/examples/noise1d.html https://processing.org/examples/noise2d.html https://processing.org/examples/noise3d.html

What is a pixel-array?

In this project you used get-command to get the color of a pixel. get-command needs two parameters, the x- and the y-coordinate of the pixel.

color pixelColor = myMovie.get[x, y];

Although we see the pixels as a nice rectangle with the x- and y-axis, to the computer pixels form a long one-dimensional array. The computer creates this array automatically when you start to process the pixels. If you continue with pixel processing in the future, you probably encounter projects that use the pixel-array rather than something like the get-command. So understanding the array may be useful!

Array is just a long one-dimensional list of things so you can’t add the x- and y-coordinates inside the brackets, like you did with the get-command. Instead, you have to convert the x and y coordinates to a position in the array. Luckily, the calculation isn’t that hard.

color pixelColor = myMovie.pixels[(y*myMovie.width) + x];

For every time we’ve gone through all the pixels on the x-axis, we’ve increased y by one. This means that if we multiply y by the width of the video, we’ve got the number of pixels of all the full rows we’ve gone through already. Then we add the current x position in the loop to get to the position we are at right now.

Check this tutorial about pixels and arrays at Processing.org

If you want to know more about arrays, check the fourth chapter of the Art Programming Basics -module!