iOS AVFoundation: Setting Orientation of Video

Question

I\'ve been struggling with several dimensions to the problem of controlling video orientation during and after capture on an iOS device. Thanks to previous answers and docum

Answer 1

Since iOS 5 you can request rotated CVPixelBuffers using AVCaptureVideoDataOutput documented here. This gives you the correct orientation without having to reprocess the video again with AVAssetExportSession.

Answer 2

In case anyone else is looking for this answer as well, this is the method that I cooked up (modified a bit to simplify):

- (void)encodeVideoOrientation:(NSURL *)anOutputFileURL
{
CGAffineTransform rotationTransform;
CGAffineTransform rotateTranslate;
CGSize renderSize;

switch (self.recordingOrientation)
{
    // set these 3 values based on orientation

}


AVURLAsset * videoAsset = [[AVURLAsset alloc]initWithURL:anOutputFileURL options:nil];

AVAssetTrack *sourceVideoTrack = [[videoAsset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
AVAssetTrack *sourceAudioTrack = [[videoAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];

AVMutableComposition* composition = [AVMutableComposition composition];

AVMutableCompositionTrack *compositionVideoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
[compositionVideoTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration)
                               ofTrack:sourceVideoTrack
                                atTime:kCMTimeZero error:nil];
[compositionVideoTrack setPreferredTransform:sourceVideoTrack.preferredTransform];

AVMutableCompositionTrack *compositionAudioTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio
                                                                            preferredTrackID:kCMPersistentTrackID_Invalid];
[compositionAudioTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration)
                               ofTrack:sourceAudioTrack
                                atTime:kCMTimeZero error:nil];



AVMutableVideoCompositionInstruction *instruction = [AVMutableVideoCompositionInstruction videoCompositionInstruction];
AVMutableVideoCompositionLayerInstruction *layerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:compositionVideoTrack];
[layerInstruction setTransform:rotateTranslate atTime:kCMTimeZero];

AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];
videoComposition.frameDuration = CMTimeMake(1,30);
videoComposition.renderScale = 1.0;
videoComposition.renderSize = renderSize;
instruction.layerInstructions = [NSArray arrayWithObject: layerInstruction];
instruction.timeRange = CMTimeRangeMake(kCMTimeZero, videoAsset.duration);
videoComposition.instructions = [NSArray arrayWithObject: instruction];

AVAssetExportSession * assetExport = [[AVAssetExportSession alloc] initWithAsset:composition
                                                                      presetName:AVAssetExportPresetMediumQuality];

NSString* videoName = @"export.mov";
NSString *exportPath = [NSTemporaryDirectory() stringByAppendingPathComponent:videoName];

NSURL * exportUrl = [NSURL fileURLWithPath:exportPath];

if ([[NSFileManager defaultManager] fileExistsAtPath:exportPath])
{
    [[NSFileManager defaultManager] removeItemAtPath:exportPath error:nil];
}

assetExport.outputFileType = AVFileTypeMPEG4;
assetExport.outputURL = exportUrl;
assetExport.shouldOptimizeForNetworkUse = YES;
assetExport.videoComposition = videoComposition;

[assetExport exportAsynchronouslyWithCompletionHandler:
 ^(void ) {
     switch (assetExport.status)
     {
         case AVAssetExportSessionStatusCompleted:
             //                export complete
             NSLog(@"Export Complete");
             break;
         case AVAssetExportSessionStatusFailed:
             NSLog(@"Export Failed");
             NSLog(@"ExportSessionError: %@", [assetExport.error localizedDescription]);
             //                export error (see exportSession.error)
             break;
         case AVAssetExportSessionStatusCancelled:
             NSLog(@"Export Failed");
             NSLog(@"ExportSessionError: %@", [assetExport.error localizedDescription]);
             //                export cancelled
             break;
     }
 }];

}

This stuff is poorly documented, unfortunately, but by stringing examples together from other SO questions and reading the header files, I was able to get this working. Hope this helps anyone else!

Answer 3

In Apple's documentation here it states:

Clients may now receive physically rotated CVPixelBuffers in their AVCaptureVideoDataOutput -captureOutput:didOutputSampleBuffer:fromConnection: delegate callback. In previous iOS versions, the front-facing camera would always deliver buffers in AVCaptureVideoOrientationLandscapeLeft and the back-facing camera would always deliver buffers in AVCaptureVideoOrientationLandscapeRight. All 4 AVCaptureVideoOrientations are supported, and rotation is hardware accelerated. To request buffer rotation, a client calls -setVideoOrientation: on the AVCaptureVideoDataOutput's video AVCaptureConnection. Note that physically rotating buffers does come with a performance cost, so only request rotation if it's necessary. If, for instance, you want rotated video written to a QuickTime movie file using AVAssetWriter, it is preferable to set the -transform property on the AVAssetWriterInput rather than physically rotate the buffers in AVCaptureVideoDataOutput.

So the posted solution by Aaron Vegh that uses an AVAssetExportSession works, but is not needed. Like the Apple doc's say, if you'd like to have the orientation set correctly so that it plays in non-apple quicktime players like VLC or on the web using Chrome, you must set the video orientation on the AVCaptureConnection for the AVCaptureVideoDataOutput. If you try to set it for the AVAssetWriterInput you will get an incorrect orientation for players like VLC and Chrome.

Here is my code where I set it during setting up the capture session:

// DECLARED AS PROPERTIES ABOVE
@property (strong,nonatomic) AVCaptureDeviceInput *audioIn;
@property (strong,nonatomic) AVCaptureAudioDataOutput *audioOut;
@property (strong,nonatomic) AVCaptureDeviceInput *videoIn;
@property (strong,nonatomic) AVCaptureVideoDataOutput *videoOut;
@property (strong,nonatomic) AVCaptureConnection *audioConnection;
@property (strong,nonatomic) AVCaptureConnection *videoConnection;
------------------------------------------------------------------
------------------------------------------------------------------

-(void)setupCaptureSession{
// Setup Session
self.session = [[AVCaptureSession alloc]init];
[self.session setSessionPreset:AVCaptureSessionPreset640x480];

// Create Audio connection ----------------------------------------
self.audioIn = [[AVCaptureDeviceInput alloc]initWithDevice:[self getAudioDevice] error:nil];
if ([self.session canAddInput:self.audioIn]) {
    [self.session addInput:self.audioIn];
}

self.audioOut = [[AVCaptureAudioDataOutput alloc]init];
dispatch_queue_t audioCaptureQueue = dispatch_queue_create("Audio Capture Queue", DISPATCH_QUEUE_SERIAL);
[self.audioOut setSampleBufferDelegate:self queue:audioCaptureQueue];
if ([self.session canAddOutput:self.audioOut]) {
    [self.session addOutput:self.audioOut];
}
self.audioConnection = [self.audioOut connectionWithMediaType:AVMediaTypeAudio];

// Create Video connection ----------------------------------------
self.videoIn = [[AVCaptureDeviceInput alloc]initWithDevice:[self videoDeviceWithPosition:AVCaptureDevicePositionBack] error:nil];
if ([self.session canAddInput:self.videoIn]) {
    [self.session addInput:self.videoIn];
}

self.videoOut = [[AVCaptureVideoDataOutput alloc]init];
[self.videoOut setAlwaysDiscardsLateVideoFrames:NO];
[self.videoOut setVideoSettings:nil];
dispatch_queue_t videoCaptureQueue =  dispatch_queue_create("Video Capture Queue", DISPATCH_QUEUE_SERIAL);
[self.videoOut setSampleBufferDelegate:self queue:videoCaptureQueue];
if ([self.session canAddOutput:self.videoOut]) {
    [self.session addOutput:self.videoOut];
}

self.videoConnection = [self.videoOut connectionWithMediaType:AVMediaTypeVideo];
// SET THE ORIENTATION HERE -------------------------------------------------
[self.videoConnection setVideoOrientation:AVCaptureVideoOrientationPortrait];
// --------------------------------------------------------------------------

// Create Preview Layer -------------------------------------------
AVCaptureVideoPreviewLayer *previewLayer = [[AVCaptureVideoPreviewLayer alloc]initWithSession:self.session];
CGRect bounds = self.videoView.bounds;
previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;
previewLayer.bounds = bounds;
previewLayer.position=CGPointMake(CGRectGetMidX(bounds), CGRectGetMidY(bounds));
[self.videoView.layer addSublayer:previewLayer];

// Start session
[self.session startRunning];

}

Answer 4

Finally,based on the answers of @Aaron Vegh and @Prince, I figured out my resolution: //Converting video

+(void)convertMOVToMp4:(NSString *)movFilePath completion:(void (^)(NSString *mp4FilePath))block{


AVURLAsset * videoAsset = [[AVURLAsset alloc]initWithURL:[NSURL fileURLWithPath:movFilePath]  options:nil];

AVAssetTrack *sourceAudioTrack = [[videoAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0];

AVMutableComposition* composition = [AVMutableComposition composition];


AVMutableCompositionTrack *compositionAudioTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio
                                                                            preferredTrackID:kCMPersistentTrackID_Invalid];
[compositionAudioTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, videoAsset.duration)
                               ofTrack:sourceAudioTrack
                                atTime:kCMTimeZero error:nil];




AVAssetExportSession * assetExport = [[AVAssetExportSession alloc] initWithAsset:composition
                                                                      presetName:AVAssetExportPresetMediumQuality];


NSString *exportPath =  [movFilePath stringByReplacingOccurrencesOfString:@".MOV" withString:@".mp4"];


NSURL * exportUrl = [NSURL fileURLWithPath:exportPath];


assetExport.outputFileType = AVFileTypeMPEG4;
assetExport.outputURL = exportUrl;
assetExport.shouldOptimizeForNetworkUse = YES;
assetExport.videoComposition = [self getVideoComposition:videoAsset composition:composition];

[assetExport exportAsynchronouslyWithCompletionHandler:
 ^(void ) {
     switch (assetExport.status)
     {
         case AVAssetExportSessionStatusCompleted:
             //                export complete
                    if (block) {
                         block(exportPath);
                }
             break;
         case AVAssetExportSessionStatusFailed:
             block(nil);
             break;
         case AVAssetExportSessionStatusCancelled:
            block(nil);
             break;
     }
 }];
}

//get current orientation

  +(AVMutableVideoComposition *) getVideoComposition:(AVAsset *)asset composition:( AVMutableComposition*)composition{
    BOOL isPortrait_ = [self isVideoPortrait:asset];


    AVMutableCompositionTrack *compositionVideoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];


    AVAssetTrack *videoTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
    [compositionVideoTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, asset.duration) ofTrack:videoTrack atTime:kCMTimeZero error:nil];

    AVMutableVideoCompositionLayerInstruction *layerInst = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:compositionVideoTrack];

    CGAffineTransform transform = videoTrack.preferredTransform;
    [layerInst setTransform:transform atTime:kCMTimeZero];


    AVMutableVideoCompositionInstruction *inst = [AVMutableVideoCompositionInstruction videoCompositionInstruction];
    inst.timeRange = CMTimeRangeMake(kCMTimeZero, asset.duration);
    inst.layerInstructions = [NSArray arrayWithObject:layerInst];


    AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];
    videoComposition.instructions = [NSArray arrayWithObject:inst];

    CGSize videoSize = videoTrack.naturalSize;
    if(isPortrait_) {
        NSLog(@"video is portrait ");
        videoSize = CGSizeMake(videoSize.height, videoSize.width);
    }
    videoComposition.renderSize = videoSize;
    videoComposition.frameDuration = CMTimeMake(1,30);
    videoComposition.renderScale = 1.0;
    return videoComposition;
   }

//get video

+(BOOL) isVideoPortrait:(AVAsset *)asset{
BOOL isPortrait = FALSE;
NSArray *tracks = [asset tracksWithMediaType:AVMediaTypeVideo];
if([tracks    count] > 0) {
    AVAssetTrack *videoTrack = [tracks objectAtIndex:0];

    CGAffineTransform t = videoTrack.preferredTransform;
    // Portrait
    if(t.a == 0 && t.b == 1.0 && t.c == -1.0 && t.d == 0)
    {
        isPortrait = YES;
    }
    // PortraitUpsideDown
    if(t.a == 0 && t.b == -1.0 && t.c == 1.0 && t.d == 0)  {

        isPortrait = YES;
    }
    // LandscapeRight
    if(t.a == 1.0 && t.b == 0 && t.c == 0 && t.d == 1.0)
    {
        isPortrait = FALSE;
    }
    // LandscapeLeft
    if(t.a == -1.0 && t.b == 0 && t.c == 0 && t.d == -1.0)
    {
        isPortrait = FALSE;
    }
}
return isPortrait;

}

Answer 5

Use these below method to set correct orientation according to video asset orientation in AVMutableVideoComposition

-(AVMutableVideoComposition *) getVideoComposition:(AVAsset *)asset
{
  AVAssetTrack *videoTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
  AVMutableComposition *composition = [AVMutableComposition composition];
  AVMutableVideoComposition *videoComposition = [AVMutableVideoComposition videoComposition];
  CGSize videoSize = videoTrack.naturalSize;
  BOOL isPortrait_ = [self isVideoPortrait:asset];
  if(isPortrait_) {
      NSLog(@"video is portrait ");
      videoSize = CGSizeMake(videoSize.height, videoSize.width);
  }
  composition.naturalSize     = videoSize;
  videoComposition.renderSize = videoSize;
  // videoComposition.renderSize = videoTrack.naturalSize; //
  videoComposition.frameDuration = CMTimeMakeWithSeconds( 1 / videoTrack.nominalFrameRate, 600);

  AVMutableCompositionTrack *compositionVideoTrack;
  compositionVideoTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
  [compositionVideoTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, asset.duration) ofTrack:videoTrack atTime:kCMTimeZero error:nil];
  AVMutableVideoCompositionLayerInstruction *layerInst;
  layerInst = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:videoTrack];
  [layerInst setTransform:videoTrack.preferredTransform atTime:kCMTimeZero];
  AVMutableVideoCompositionInstruction *inst = [AVMutableVideoCompositionInstruction videoCompositionInstruction];
  inst.timeRange = CMTimeRangeMake(kCMTimeZero, asset.duration);
  inst.layerInstructions = [NSArray arrayWithObject:layerInst];
  videoComposition.instructions = [NSArray arrayWithObject:inst];
  return videoComposition;
}


-(BOOL) isVideoPortrait:(AVAsset *)asset
{
  BOOL isPortrait = FALSE;
  NSArray *tracks = [asset tracksWithMediaType:AVMediaTypeVideo];
  if([tracks    count] > 0) {
    AVAssetTrack *videoTrack = [tracks objectAtIndex:0];

    CGAffineTransform t = videoTrack.preferredTransform;
    // Portrait
    if(t.a == 0 && t.b == 1.0 && t.c == -1.0 && t.d == 0)
    {
        isPortrait = YES;
    }
    // PortraitUpsideDown
    if(t.a == 0 && t.b == -1.0 && t.c == 1.0 && t.d == 0)  {

        isPortrait = YES;
    }
    // LandscapeRight
    if(t.a == 1.0 && t.b == 0 && t.c == 0 && t.d == 1.0)
    {
        isPortrait = FALSE;
    }
    // LandscapeLeft
    if(t.a == -1.0 && t.b == 0 && t.c == 0 && t.d == -1.0)
    {
        isPortrait = FALSE;
    }
   }
  return isPortrait;
}