connectOnFrameAvailable() provides TangoImageBuffer with curious format infos

问题

Also trying to get access to color data bytes from color cam of Tango, I was stuck on java API by being able to connect tango Cam to a surface for display (but just OK for display in fact, no easy access to raw data, nor time stamp)... so finally I switch using C API on native code (latest FERMAT lib and header) and follow recommendation I found on stack Overflow by registering a derivated sample code to connectOnFrameAvailable()... (I start using PointCloudActivity sample for that test).

First problem I found is somewhat a side effect of registering to that callback, that works usually fine (callbacks gets fire regularly), but then another callback that I also registered, to get xyz clouds, start to fail to fire. Like in sample code I mentioned, clouds are get through a onXYZijAvailable() callback, that the app registers using TangoService_connectOnXYZijAvailable(onXYZijAvailable).

So failing to get xyz callback fired is not happening always, but usually half of the time, during tests, with a awful workaround that is by taking the app in background then foreground again ... this is curious, is this "recover" related to On-pause/On-resume low level stuff??). If someone has clues .... By the way in Java API, same side effect was observed, once connecting cam texture for display (through Tango adequate API ...)

But here is my second "problem", back to acquiring YV12 color data from camera : through registering to TangoService_connectOnFrameAvailable( TangoCameraId::TANGO_CAMERA_COLOR, nullptr, onFrameAvailable) and providing static funtion onFrameAvailable defined like this :

static void onFrameAvailable(void* ctx, TangoCameraId id, const TangoImageBuffer* buffer)
{
   ...
   LOGI("OnFrameAvailable(): Cam frame data received");
   // Check if data format of expected type : YV12 , i.e.
   // TangoImageFormatType::TANGO_HAL_PIXEL_FORMAT_YV12 
   //  i.e.  = 0x32315659  // YCrCb 4:2:0 Planar
   //LOGI("OnFrameAvailable(): Frame data format (%x)", buffer->format);
   .... 
}

the problem is that width, height, stride information of received TangoImageBuffer structure seems valid (1280x720, ...), BUT the format returned is changing every-time, and not the expected magic number (here 0x32315659) ... I am doing something wrong there ? (but other info are OK ...)

Also, there is apparently only one data format defined (YV12 ) here, but seeing Fish Eye images from demo app, it seems grey level image, is it using same (color) format as low level capture than the RGB cam ???

回答1:

1) Regarding the image from the camera, I came to the same conclusion you did - only availability of image data is through the C API

2) Regarding the image - I haven't had any issues with YUV, and my last encounter with this stuff was when I wrote JPEG stuff - the format is naked, i.e. it's an organizational structure and has no header information save the undefined metadata in the first line of pixels mentioned here - Here's a link to some code that may help you decode the image in a response to another message here

3) Regarding point cloud returns - Please note this information is anecdotal, and to some degree the product of superstition - what works for me only does that sometimes, and may not work at all for you

Tango does seem to have a remarkable knack to simply stop producing point clouds. I think a lot of it has to do with very sensitive timing internally (I wonder if anyone mentioned that Linux ain't an RTOS when this was first crafted)
Almost all issues I encounter can be attributed to screwing up the timing where
A. Debugging the C level can may point clouds stop coming
B. Bugs in the native or java code that cause hiccups in the threads that are handling the callbacks can cause point clouds to stop coming
C. Excessive load can cause the system to loose sync, at which point the point clouds will stop coming - this is detectable, you will start to see a silvery grid pattern appear in rectangular areas of the image, and point clouds will cease. Rarely, the system will recover if load decreases, the silvery pattern goes away, and point clouds come back - more commonly the silvery pattern (I think its the 3d spatializing grid) grows to cover more of the image - at least a restart of the app is required for me, and a full tablet reboot every 3rd time or so

Summarizing, that's my suspicions and countermeasures, but it's based completely on personal experience -

来源：https://stackoverflow.com/questions/28165769/connectonframeavailable-provides-tangoimagebuffer-with-curious-format-infos

标签

android

colors

google-project-tango