Sunday, August 4, 2013

Using CoreImage filters in Openframeworks

Macintosh OSX CoreImage has some nice filter implementations, so it can be attractive to use some of these within Openframeworks. The Gaussian Blur filter, for example, is flexible and fast. So the trick is to get CoreImage to work with an Openframeworks OpenGL textures.

I implemented the following solution in Openframeworks v0.7.4, working on Mac OS version 10.7.5, Xcode 4.6.3. It involves drawing the image to a framebuffer and then passing the texture ID for the framebuffer to CoreImage.

Note that the filename of "testApp.cpp" needs to be changed to "testApp.mm" and "main.cpp" to "main.mm". This changes these files to an Objective-C++ type, which can handle both C++ and Objective-C calls.

Note also that (apparently) the QuartzCore framework needs to be manually added to the Xcode project frameworks list.

N.B.: I have found that on occasional runs of this and similar code that occasionally (1 out of every 4-8 runs), the code generates a CoreImage: ROI is not tilable error, and the CI Filter doesn't run, even if the project is configured at a humble screen size like 800x600. I haven't been able to find a solution online, so if you know the solution, I'm all ears.

The "testApp.h" file:

#pragma once
//  *****************************************************************
//  Demonstration of using Max OSX Core Image filters in an
//  Open Frameworks 0.7.4 project
//
//  Bret Battey / BatHatMedia.com
//  August 4, 2013
//  *****************************************************************

#include "ofMain.h"
#include "QuartzCore/QuartzCore.h"  // One way to get the CI classes


class testApp : public ofBaseApp{

 public:
  void setup();
  void update();
  void draw();

  void keyPressed  (int key);
  void keyReleased(int key);
  void mouseMoved(int x, int y );
  void mouseDragged(int x, int y, int button);
  void mousePressed(int x, int y, int button);
  void mouseReleased(int x, int y, int button);
  void windowResized(int w, int h);
  void dragEvent(ofDragInfo dragInfo);
  void gotMessage(ofMessage msg);

    // Basics
    int     outWidth, outHeight;
    ofFbo   sourceFbo;

    
    // Core Image
    CGLContextObj   CGLContext;
    NSOpenGLPixelFormatAttribute*   attr;
    NSOpenGLPixelFormat*    pf;
    CGColorSpaceRef genericRGB;
    CIContext*  glCIcontext;
    CIImage*    inputCIImage;
    CIFilter*   blurFilter;
    CIImage*    blurredCIImage;
    CGSize      texSize;
    GLint       tex;
    CGRect      outRect;
    CGRect      inRect;

    
};


The "testApp.mm" file, up through the draw() call:

//  *****************************************************************
//  Demonstration of using Max OSX Core Image filters in an
//  Open Frameworks 0.7.4 project
//
//  Note that this has to be a .mm file (Objective-C++), since it
//  combines Objective-C and C++. >>> ALSO main.cpp needs to be changed
//  to main.mm
//
//  On Xcode, you must manually add the QuartzCore framework to the
//  Project / Target / Summary list of frameworks for the build to
//  succeed.
// 
//  Bret Battey / BatHatMedia.com
//  August 4, 2013
//  *****************************************************************


#include "testApp.h"

//--------------------------------------------------------------
void testApp::setup(){

    ofSetFrameRate(30);
    outWidth  = ofGetViewportWidth();
    outHeight = ofGetViewportHeight();
    ofEnableSmoothing();
    ofBackground(0);
    ofNoFill();
    ofSetLineWidth(4);

    // Setup a framebuffer for the drawing. Perhaps there is some way to do this
    // without a framebuffer, but this is the only way I could figure out how to
    // enable grabbing an OpenGL texture to pass to the CoreImage filter
    sourceFbo.allocate(outWidth, outHeight, GL_RGBA32F_ARB); //32-bit framebuffer for smoothness
    
    // SETUP THE CORE IMAGE CONTEXT, FILTERS, ETC
    // The appended .autorelease methods should auto cleanup the memory at exit.
    // Use a generic RGB color space:
    genericRGB = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);
    // Create the pixel format attributes... The Core Image Guide for processing images says:
    // "It’s important that the pixel format for the context includes the NSOpenGLPFANoRecovery constant as an
    // attribute. Otherwise Core Image may not be able to create another context that shares textures with this one."
    NSOpenGLPixelFormatAttribute attr[] = {
        NSOpenGLPFAAccelerated,
        NSOpenGLPFANoRecovery, 
        NSOpenGLPFAColorSize, 32,
        0
    };
    CGColorSpaceRelease(genericRGB);
    // Setup the pixel format object:
    pf=[[NSOpenGLPixelFormat alloc] initWithAttributes:attr].autorelease;
    // Setup the core image context, tied to the OF Open GL context:
    glCIcontext = [CIContext contextWithCGLContext: CGLGetCurrentContext()
                                       pixelFormat: CGLPixelFormatObj(pf)
                                        colorSpace: genericRGB
                                           options: nil].autorelease;
    // Setup a Gaussian Blur filter:
    blurFilter = [CIFilter filterWithName:@"CIGaussianBlur"].autorelease;
    // Supporting stuff
    texSize = CGSizeMake(outWidth, outHeight);
    inRect = CGRectMake(0,0,outWidth,outHeight);
    outRect = CGRectMake(0,0,outWidth,outHeight);


}

//--------------------------------------------------------------
void testApp::update(){
    
    sourceFbo.begin();
    
    // For feedback fun, let's not clear the Fbo after the first frame
    if(ofGetFrameNum()==1) {
        ofClear(0);
    }
    // Draw circle
    ofSetColor(20, 130, 250);
    ofCircle(outWidth/2,outHeight/2,10+(ofGetFrameNum()%40)*6);
    // Get the texture ID of the fbo:
    tex = sourceFbo.getTextureReference().texData.textureID; 
    // set the CI Image to link with the Fbo texture
    inputCIImage = [CIImage imageWithTexture:tex
                                        size:texSize
                                     flipped:NO
                                  colorSpace:genericRGB];
    // Blur filter
    [blurFilter setValue:inputCIImage forKey:@"inputImage"];
    [blurFilter setValue:[NSNumber numberWithFloat: 8] forKey:@"inputRadius"];
    blurredCIImage = [blurFilter valueForKey:@"outputImage"];
    // Draw it
    ofSetColor(255);
    [glCIcontext drawImage:blurredCIImage
                    inRect:outRect
                  fromRect:inRect];
    
    sourceFbo.end();
}

//--------------------------------------------------------------
void testApp::draw(){

    sourceFbo.draw(0,0);

}

6 comments:

  1. Thanks for this!

    I am having some trouble getting everything to link correctly in OF 0.8.0 and havent tried in 0.74

    I have copied your header and everything almost exactly - #include "QuartzCore/QuartzCore.h"
    and quartzcore is included properly - but it doesn't seem to be finding stuff properly - " NSOpenGLPixelFormatAttribute" and "NSOpenGLPixelFormat" are unknown type formats - and "CIContext" is also unknown

    If you have a sample file that has all of the necessary includes, that would be incredibly useful - I am on 10.8.5 with OF 0.8.0

    ReplyDelete
  2. I haven't worked in OS 10.8.5 or OF 0.8.0 yet, so if that is the source of the problem, I won't be able to help. Did you try adding the QuartzCore framework manually to the Xcode project frameworks list, as per the note above?

    ReplyDelete
  3. Thanks! I'm fairly sure I followed your instructions to the t and added quartzcore in everything so I haven't seen any issues there yet - it is in the "link file with binary" step and in my header (and i tried putting it in the "copy files" step at some point as well)

    If you have just a simple example file that I can check the build phases/settings that would probably be most helpful - I feel like there is some small flag or something that I'm not setting up correctly

    It's strange because appkit seems to not be linked properly too if the NSOpenGL objects arent working since those are in appkit, not quartzcore

    Here is a link to my current OF project file: https://www.dropbox.com/s/8mwj6mz9xbahogg/coreImage_sketch.zip

    ReplyDelete
  4. Ah ha! Got it almost working in 0.8.0 - the problem was that with the OF project generator my .mm and .h files had defaulted to C as their file type so it wasn't compiling properly. Once I changed those to Obj-C++ Source it worked much better.

    Now it's just crashing in the newer GLFW window, so I'm going to try it with a GLUT window and see if it builds properly.

    Thanks for your help...I'll post if I find anything interesting!

    ReplyDelete
  5. Just wanted to share with you....just based of using your example I made a complete addon that leverages about 65 filters and doesn't require the .mm changeup in main.cpp https://github.com/laserpilot/ofxCoreImage

    Works in 0.8.1 and OSX 10.8/10.9 - only works in the GLUT window, not in a GLFW window that is now the standard - still investigating that

    ReplyDelete
    Replies
    1. Great stuff! It is wonderful to see my post picked up and expanded in such a useful way.

      Delete