Source
r/processing•
hyperkinetic

Using Processing to filter (simplify) a video file.

Hi all. I'm trying to make a LED matrix display using the Arduino, which ideally reads the display data off an SD card. I've looked into a bunch of different file formats, and the closest thing to what I need appears to be Quicktime uncompressed 32-bit argb. It's raw RGB frame data, is readily supported by the authoring software I'm using, and allows me to output in the resolution of my display.

I looked into reading the .mov file directly, but it is a twisted mess of headers, structures, and metadata that is pretty useless to me. Processing has no problem opening and playing the video files I have. What I'm hoping to do next is extract the raw frame data and write it to a binary file to include on the SD card, but I can't seem to figure out what function would allow me to do that. Any of the 'save*' functions appear to open the file, write what you provide, and close again, leaving me with only one frame.

How can I use processing to take the stream of frames and write the raw RGB data to a file?

Posted Dec 17, 2014, 11:15 PM UTC

39
push_pop•

Maybe try the movieEvent() callback?

Edit: Maybe not. Sorry been a while since I've touched Processing... Do you ever use Cinder or OF? Would be pretty easy do do it with one of those. Then you could format the output file however you wanted.

OK I was bored. Here's how to get individual pixel values from a frame of a video. idk how to do file I/O in processing but that shouldn't be too hard. If you can't figure it out. I'll check it out:

Edit: THIS DOESN'T EVEN COMPILE MY BAD SORRY OOPS see below for (probably) working code

import processing.video.*;

PImage frame;
Movie myMovie;

void setup()
{
    myMovie = new Movie(this, "movie.mov");
    myMovie.loop();

    size(800,600);
}

void draw()
{
    frame = myMovie;
  //Note: frame.Size is psuedocode. Assume you know the dims or know how to get them.
   for(int i = 0; i < frame.Size; i++)
  {
     color pixel = frame.pixels[i];
     float r = pixel.red();
     float g = pixel.green();
     float b = pixel.blue();
  }
  image(frame, 0,0);
}

void movieEvent(Movie m) {
     m.read();
}
1
hyperkineticOP•

frame.Size is psuedocode. Assume you know the dims or know how to get them.

It's not in the documentation, but I found in a code example that: frame.width and frame.height return the dimensions of the frame.

Do the variables r,g, and b have to be float? They're byte internally.

1
push_pop•

Ok scratch all that above. There's a ton wrong. Try this: https://www.processing.org/reference/colorMode_.html

IT at least runs but I don't have any way to test the values;

import processing.video.*;

PImage frame;
Movie myMovie;

void setup()
{
    myMovie = new Movie(this, "movie.mov");
    colorMode(RGB, 255);
    myMovie.play();
    size(800,600);
}

void draw()
{

  while(myMovie.available())
  {
    myMovie.read();
    frame = myMovie;
    for(int i = 0; i < frame.width*frame.height; i++)
    {
          color pixel = frame.pixels[i];
          byte red  =  (byte)(pixel >> 16 & 0xFF);
          byte green = (byte)(pixel >> 8 & 0xFF);
          byte blue = (byte)(pixel & 0xFF);
    }
    image(frame,0,0);
  }
 
}
1
hyperkineticOP•

This looks promising! I'll give it a go and report back in a few... Thanks!

[Edit] no go. For whatever reason the applet crashes hard. I tried upgrading from 1.5.1 to the newest 2.x, but 2.x doesn't even open my source movie file in the sketches I had working. If I find a way to either keep a file open or append to a file I've written, I'll surely integrate the bits of code that comb over the movie and extract the values of interest. Thanks for all the help.

2
Freedom_Grenade•

Would this work for what you want to do for output?

1
hyperkineticOP•

Don't know. I'll give it a try!

[edit] Nope.

all files loaded and saved by the Processing API use UTF-8 encoding.

I really hate when functions make assumptions about how they're to be used. I can't believe that there isn't a way to stuff a raw array of bytes into a file in Processing.

1
push_pop•

Try this: https://www.processing.org/reference/binary_.html

writer.Print(binary(red));

EDIT: nope don't think that will work.

2
Freedom_Grenade•

Ok, so There is also saveBytes()

    ArrayList<Byte> bytelist = new ArrayList<Byte>(); 
    PImage img;
    String filename;
    
    public void setup() {
      img = loadImage("test.jpg");
      filename = "test.b";
      noLoop();
      bytelist.add((byte)(img.width >> 8));                  // just adding the width and height to the ArrayList
      bytelist.add((byte)(img.width));
      bytelist.add((byte)(img.height >> 8));
      bytelist.add((byte)(img.height));
    }
    
    public void draw() {
       
      addFrame(img, bytelist);                               
    
      saveBytes(filename, convertArrayList(bytelist) );
    }
    
    public void addFrame(PImage p, ArrayList<Byte> b) {
      p.loadPixels();
      for (int i = 0; i < p.pixels.length; i++) {
         b.add((byte)(p.pixels[i] >> 16));
         b.add((byte)(p.pixels[i] >> 8));
         b.add((byte)(p.pixels[i]));
      }
    }
    
    public byte[] convertArrayList(ArrayList<Byte> b) {
      byte[] bytes = new byte[b.size()];
      
      for (int i = 0; i < bytes.length; i++) {          
        bytes[i] = b.get(i).byteValue();
      }
      
      return bytes;
    }

Then try this in a different sketch (make sure to copy the 'test.b' to it)

    String filename;
    byte bytes[];
    PImage img;
    public void setup() {
      size(800,800);
    
      filename = "test.b";
      bytes = loadBytes(filename);
      
      int w = ((bytes[0] & 0xFF) << 8) | (bytes[1] & 0xFF);
      int h = ((bytes[2] & 0xFF) << 8) | (bytes[3] & 0xFF);  
      
      img = createImage(w,h,RGB);
      img.loadPixels();
      
      for(int i = 4; i < bytes.length - 3; i+=3) {
        img.pixels[(i-4)/3] = color(bytes[i]&0xFF,bytes[i+1]&0xFF,bytes[i+2]&0xFF);  
      }
    }
    
    public void draw() {
      image(img,0,0);
    }

This shows the output in a hex editor .. Sorry about the fact that all my top pixels were white, but it has the width and height in there..

You don't need to deal with the ArrayList if you know exactly how much data you are going to be using. The only problem I see with this is, if you use it on a movie your bytes[] array is going to be very large and I didn't find a way for saveBytes() to append instead of overwriting a file.
So you might want to take a look at something like FileOutputStream and set whether it appends or not.

1
hyperkineticOP•

Wow! Thanks so much for putting all that code together. I should be able to test it all tonight. I was a bit concerned about the array sizes too. My current source file is 16+MB, and I expect to author larger. My machine is up to the task. I've got 16GB, and I only have to process each one once.

I didn't find a way for saveBytes() to append instead of overwriting a file.

I found this to be the single most annoying limitation with Processing. I'm used to C where you open a file handle, deal with the file in any and every way you want, then close it when done. I can't believe the designers assumed that you'd only want to atomically write a single blob of data and be done, without any way of adding to it. It's extremely short sided.

So you might want to take a look at something like FileOutputStream

I'll surely take a look, but Java is not my native language. I find many of the things it does really odd and non-intuitive.

1