问题
I have been working with Processing and Cinder to modify Kinect input on the fly. However, I would also like to record the full stream (depth+color+accelerometer values, and whatever else is in there). I'm recording so I can try out different effects/treatments on the same material.
Because I am still just learning Cinder and Processing is quite slow/laggy, I have had trouble finding advice on a strategy for capturing the stream - anything (preferably in Cinder, oF, or Processing) would be really helpful.
回答1:
I've tried both Processing and OpenFrameworks. Processing is slower when displaying both images (depth and colour). OpenFrameworks slows a bit while writing the data to disk, but here's the basic approach:
- Setup Openframeworks (open and compile any sample to make sure you're up and running)
- Download the ofxKinect addon and copy the example project as described on github.
- Once you've got OF and the ofxKinect example running, it's just a matter of adding a few variable to save your data:
In this basic setup, I've created a couple of ofImage instances and a boolean to toggle saving. In the example the depth and RGB buffers are saved into ofxCvGrayscaleImage instances, but I haven't used OF and OpenCV enough to know how to do something as simple as saving an image to disk, which is why I've used two ofImage instances.
I don't know how comfortable you are with Processing, OF, Cinder, so, for arguments' sake I'll assume you know you're way around Processing, but you're still tackling C++.
OF is pretty similar to Processing, but there are a few differences:
- In Processing you have variables declaration and they're usage in the same file. In OF you've got a .h file where you declare you're variables and the .cpp file where you initialize and use your variables.
- In Processing you have the setup()(initialize variables) and draw()(update variables and draw to screen) methods, while in OF you have setup() (same as in Processing), update() (update variables only, nothing visual) and draw() (draw to screen using updated values)
- When working with images, you since you're coding in C++, you need to allocate memory first, as opposed to Processing/Java where you have memory management.
There's more differences that I won'te detail here. Do check out OF for Processing Users on the wiki
Back to the exampleKinect example, here my basic setup:
.h file:
#pragma once
#include "ofMain.h"
#include "ofxOpenCv.h"
#include "ofxKinect.h"
class testApp : public ofBaseApp {
public:
void setup();
void update();
void draw();
void exit();
void drawPointCloud();
void keyPressed (int key);
void mouseMoved(int x, int y );
void mouseDragged(int x, int y, int button);
void mousePressed(int x, int y, int button);
void mouseReleased(int x, int y, int button);
void windowResized(int w, int h);
ofxKinect kinect;
ofxCvColorImage colorImg;
ofxCvGrayscaleImage grayImage;
ofxCvGrayscaleImage grayThresh;
ofxCvGrayscaleImage grayThreshFar;
ofxCvContourFinder contourFinder;
ofImage colorData;//to save RGB data to disk
ofImage grayData;//to save depth data to disk
bool bThreshWithOpenCV;
bool drawPC;
bool saveData;//save to disk toggle
int nearThreshold;
int farThreshold;
int angle;
int pointCloudRotationY;
int saveCount;//counter used for naming 'frames'
};
and the .cpp file:
#include "testApp.h"
//--------------------------------------------------------------
void testApp::setup() {
//kinect.init(true); //shows infrared image
kinect.init();
kinect.setVerbose(true);
kinect.open();
colorImg.allocate(kinect.width, kinect.height);
grayImage.allocate(kinect.width, kinect.height);
grayThresh.allocate(kinect.width, kinect.height);
grayThreshFar.allocate(kinect.width, kinect.height);
//allocate memory for these ofImages which will be saved to disk
colorData.allocate(kinect.width, kinect.height, OF_IMAGE_COLOR);
grayData.allocate(kinect.width, kinect.height, OF_IMAGE_GRAYSCALE);
nearThreshold = 230;
farThreshold = 70;
bThreshWithOpenCV = true;
ofSetFrameRate(60);
// zero the tilt on startup
angle = 0;
kinect.setCameraTiltAngle(angle);
// start from the front
pointCloudRotationY = 180;
drawPC = false;
saveCount = 0;//init frame counter
}
//--------------------------------------------------------------
void testApp::update() {
ofBackground(100, 100, 100);
kinect.update();
if(kinect.isFrameNew()) // there is a new frame and we are connected
{
grayImage.setFromPixels(kinect.getDepthPixels(), kinect.width, kinect.height);
if(saveData){
//if toggled, set depth and rgb pixels to respective ofImage, save to disk and update the 'frame' counter
grayData.setFromPixels(kinect.getDepthPixels(), kinect.width, kinect.height,true);
colorData.setFromPixels(kinect.getCalibratedRGBPixels(), kinect.width, kinect.height,true);
grayData.saveImage("depth"+ofToString(saveCount)+".png");
colorData.saveImage("color"+ofToString(saveCount)+".png");
saveCount++;
}
//we do two thresholds - one for the far plane and one for the near plane
//we then do a cvAnd to get the pixels which are a union of the two thresholds.
if( bThreshWithOpenCV ){
grayThreshFar = grayImage;
grayThresh = grayImage;
grayThresh.threshold(nearThreshold, true);
grayThreshFar.threshold(farThreshold);
cvAnd(grayThresh.getCvImage(), grayThreshFar.getCvImage(), grayImage.getCvImage(), NULL);
}else{
//or we do it ourselves - show people how they can work with the pixels
unsigned char * pix = grayImage.getPixels();
int numPixels = grayImage.getWidth() * grayImage.getHeight();
for(int i = 0; i < numPixels; i++){
if( pix[i] < nearThreshold && pix[i] > farThreshold ){
pix[i] = 255;
}else{
pix[i] = 0;
}
}
}
//update the cv image
grayImage.flagImageChanged();
// find contours which are between the size of 20 pixels and 1/3 the w*h pixels.
// also, find holes is set to true so we will get interior contours as well....
contourFinder.findContours(grayImage, 10, (kinect.width*kinect.height)/2, 20, false);
}
}
//--------------------------------------------------------------
void testApp::draw() {
ofSetColor(255, 255, 255);
if(drawPC){
ofPushMatrix();
ofTranslate(420, 320);
// we need a proper camera class
drawPointCloud();
ofPopMatrix();
}else{
kinect.drawDepth(10, 10, 400, 300);
kinect.draw(420, 10, 400, 300);
grayImage.draw(10, 320, 400, 300);
contourFinder.draw(10, 320, 400, 300);
}
ofSetColor(255, 255, 255);
stringstream reportStream;
reportStream << "accel is: " << ofToString(kinect.getMksAccel().x, 2) << " / "
<< ofToString(kinect.getMksAccel().y, 2) << " / "
<< ofToString(kinect.getMksAccel().z, 2) << endl
<< "press p to switch between images and point cloud, rotate the point cloud with the mouse" << endl
<< "using opencv threshold = " << bThreshWithOpenCV <<" (press spacebar)" << endl
<< "set near threshold " << nearThreshold << " (press: + -)" << endl
<< "set far threshold " << farThreshold << " (press: < >) num blobs found " << contourFinder.nBlobs
<< ", fps: " << ofGetFrameRate() << endl
<< "press c to close the connection and o to open it again, connection is: " << kinect.isConnected() << endl
<< "press s to toggle saving depth and color data. currently saving: " << saveData << endl
<< "press UP and DOWN to change the tilt angle: " << angle << " degrees";
ofDrawBitmapString(reportStream.str(),20,656);
}
void testApp::drawPointCloud() {
ofScale(400, 400, 400);
int w = 640;
int h = 480;
ofRotateY(pointCloudRotationY);
float* distancePixels = kinect.getDistancePixels();
glBegin(GL_POINTS);
int step = 2;
for(int y = 0; y < h; y += step) {
for(int x = 0; x < w; x += step) {
ofPoint cur = kinect.getWorldCoordinateFor(x, y);
ofColor color = kinect.getCalibratedColorAt(x,y);
glColor3ub((unsigned char)color.r,(unsigned char)color.g,(unsigned char)color.b);
glVertex3f(cur.x, cur.y, cur.z);
}
}
glEnd();
}
//--------------------------------------------------------------
void testApp::exit() {
kinect.setCameraTiltAngle(0); // zero the tilt on exit
kinect.close();
}
//--------------------------------------------------------------
void testApp::keyPressed (int key) {
switch (key) {
case ' ':
bThreshWithOpenCV = !bThreshWithOpenCV;
break;
case'p':
drawPC = !drawPC;
break;
case '>':
case '.':
farThreshold ++;
if (farThreshold > 255) farThreshold = 255;
break;
case '<':
case ',':
farThreshold --;
if (farThreshold < 0) farThreshold = 0;
break;
case '+':
case '=':
nearThreshold ++;
if (nearThreshold > 255) nearThreshold = 255;
break;
case '-':
nearThreshold --;
if (nearThreshold < 0) nearThreshold = 0;
break;
case 'w':
kinect.enableDepthNearValueWhite(!kinect.isDepthNearValueWhite());
break;
case 'o':
kinect.setCameraTiltAngle(angle); // go back to prev tilt
kinect.open();
break;
case 'c':
kinect.setCameraTiltAngle(0); // zero the tilt
kinect.close();
break;
case 's'://s to toggle saving data
saveData = !saveData;
break;
case OF_KEY_UP:
angle++;
if(angle>30) angle=30;
kinect.setCameraTiltAngle(angle);
break;
case OF_KEY_DOWN:
angle--;
if(angle<-30) angle=-30;
kinect.setCameraTiltAngle(angle);
break;
}
}
//--------------------------------------------------------------
void testApp::mouseMoved(int x, int y) {
pointCloudRotationY = x;
}
//--------------------------------------------------------------
void testApp::mouseDragged(int x, int y, int button)
{}
//--------------------------------------------------------------
void testApp::mousePressed(int x, int y, int button)
{}
//--------------------------------------------------------------
void testApp::mouseReleased(int x, int y, int button)
{}
//--------------------------------------------------------------
void testApp::windowResized(int w, int h)
{}
This is a very basic setup. Feel free to modify (add tilt angle to the saved data, etc.) I'm pretty sure there are ways to improve this speedwise (e.g. don't update ofxCvGrayscaleImage instances and don't draw images to screen while saving, or stack a few frames and write them at interval as opposed to on every frame, etc.)
Goodluck
来源：https://stackoverflow.com/questions/6434972/what-is-the-most-reliable-way-to-record-a-kinect-stream-for-later-playback