How can I access Amazon AWS S3 using GSOAP for C and C++?

Question

I have searched everywhere for this and I could not find a single decent code. How can I access Amazon AWS S3 service using GSOAP?

Answer 1

The code below is from the OP. Originally, the post contained both the question and the answer and I am turning it into a Q&A format.

The signature has to be of the format

base64encode((HMAC-SHA1(ActionName+"AmazonS3"+XMLTimestamp)))

The HMAC, SHA1and B64 utils are available in openssl.

The format for SOAP requests is given by the wsdl.

The REST interface is different.

After wsdl2h to generate the header and soapcpp2 to generate the GSOAP client code the following will be the code to access the service:

Requirements : OpenSSL, GSOAP.

Build with the compiler preprocessor directive WITH_OPENSSL. Link with libraries libeay32 and ssleay32.

#include "AmazonS3SoapBinding.nsmap" //generated from soapcpp2
#include "soapAmazonS3SoapBindingProxy.h" //generated from soapcpp2
#include <stdio.h>
#include <string.h>
#include <stdarg.h>
#include <openssl/sha.h>
#include <openssl/hmac.h>
#include <openssl/evp.h>

/* convert to base64 */
std::string base64_encodestring(char* text, int len) {
    EVP_ENCODE_CTX ectx;
    int size = len*2;
    size = size > 64 ? size : 64;
    unsigned char* out = (unsigned char*)malloc( size );
    int outlen = 0;
    int tlen = 0;

    EVP_EncodeInit(&ectx);
    EVP_EncodeUpdate(&ectx,
            out,
            &outlen,
            (const unsigned char*)text,
            len
            );
    tlen += outlen;
    EVP_EncodeFinal( &ectx, out+tlen, &outlen );
    tlen += outlen;

    std::string str((char*)out, tlen );
    free( out );
    return str;
}

/* return the utc date+time in xml format */
const char* xml_datetime() {

  /*"YYYY-mm-ddTHH:MM:SS.000Z\"*/
  const int MAX=25;
  static char output[MAX+1];
  time_t now = time(NULL);

  strftime( output, MAX+1, "%Y-%m-%dT%H:%M:%S.000Z", gmtime( &now ) );

  std::cout <<output<<std::endl;
  return output;
}

/* first argument is the signing key */
/* all subsequent argumets are concatenated */
/* must end list with NULL */
char* aws_signature(char* key, ...) {
  unsigned int i, len;
  char *data, **list = &key;

  static char hmac[EVP_MAX_MD_SIZE];

  for (i = 1, len = 0; *(list+i) != NULL; ++i) {
    len += strlen( *(list+i) );
  }

  data = (char*)malloc(sizeof(char) * (len+1));

  if (data) {
    for ( i = 1, len = 0 ; *(list+i) != NULL ; ++i ) {
      strncpy( data+len, *(list+i), strlen(*(list+i)) );
      len += strlen(*(list+i));
    }
    data[len]='\0';

    std::cout<<data<<std::endl;
    HMAC( EVP_sha1(),
          key,  strlen(key),
          (unsigned char*)data, strlen(data),
          (unsigned char*) hmac, &len
        );
    free(data);
  }

  std::string b64data=base64_encodestring(hmac, len);

  strcpy(hmac,b64data.c_str());

  return hmac;
};

int main(void) {
   AmazonS3SoapBindingProxy client;

   soap_ssl_client_context(&client,
           /* for encryption w/o authentication */
           SOAP_SSL_NO_AUTHENTICATION,
           /* SOAP_SSL_DEFAULT | SOAP_SSL_SKIP_HOST_CHECK, */
           /* if we don't want the host name checks since
            * these will change from machine to machine */
           /*SOAP_SSL_DEFAULT,*/
           /* use SOAP_SSL_DEFAULT in production code */
           NULL,  /* keyfile (cert+key): required only when
                     client must  authenticate to server
                     (see SSL docs to create this file) */
           NULL,  /* password to read the keyfile */
           NULL,  /* optional cacert file to store trusted
                     certificates, use cacerts.pem for all
                     public certificates issued by common CAs */
           NULL,  /* optional capath to directory with trusted
                     certificates */
           NULL   /* if randfile!=NULL: use a file with random
                     data to seed randomness */
    );

    /* use this if you are behind a proxy server.....
        client.proxy_host="proxyserver"; // proxy hostname
        client.proxy_port=4250;
        client.proxy_userid="username";  // user pass if proxy
        client.proxy_passwd="password";  // requires authentication
        client.proxy_http_version="1.1"; // http version
    */
    _ns1__ListAllMyBuckets buk_req;
    _ns1__ListAllMyBucketsResponse buk_resp;
    ns1__ListAllMyBucketsResult buk_res;
    buk_res.soap=&client;

    buk_req.AWSAccessKeyId=new std::string("ACCESSKEY");
    buk_req.soap=&client;

    /* ListAllMyBuckets is the method I want to call here.
     * change it for other S3 services that you wish to call.*/

    char *sig=aws_signature(
            "SECRETKEY",
            "AmazonS3",
            "ListAllMyBuckets",
            xml_datetime(),
            NULL
            );


    buk_req.Signature=new std::string(sig);
    buk_req.Timestamp=new time_t(time(NULL));

    buk_resp.soap=&client;
    buk_resp.ListAllMyBucketsResponse=&buk_res;

    client.ListAllMyBuckets(&buk_req,&buk_resp);

    client.soap_stream_fault(std::cout);

    std::vector<ns1__ListAllMyBucketsEntry * >::iterator itr;

    for(itr=buk_resp.ListAllMyBucketsResponse->Buckets->Bucket.begin();
            itr!=buk_resp.ListAllMyBucketsResponse->Buckets->Bucket.end();
            itr++
            ) {
        std::cout<<(*itr)->Name<<std::endl;
    }

}

Answer 2

How can I access Amazon AWS S3 using GSOAP for C and C++?

Step 1

Use gSOAP's wsd2lh tool to convert Amazon's S3 WSDL to an interface header file aws-s3.h:

wsdl2h -t typemap.dat -o aws-s3.h http://doc.s3.amazonaws.com/2006-03-01/AmazonS3.wsdl

Use option -c to generate C source code instead of the default C++ source code. The typemap.dat file is located in the gsoap directory of the gSOAP distribution.

Step 2

Use the soapcpp2 tool on the header file created from the wsdl2h tool.

soapcpp2 -C -j aws-s3.h

This generates client-side code (-C option) with C++ service proxies and objects (-j option) from the aws-s3.h header. Omit -j for C code.

Step 3

Use the auto-generated AmazonS3SoapBindingProxy proxy methods to access AWS S3 and create a base64-encoded, HMAC-SHA1 hashed signature for AWS S3. The signature is a string with the base64-encoded version of the HMAC-SHA1 hashed string "AmazonS3" + OPERATION_NAME + Timestamp:

/*
    createbucket.cpp
    Example AWS S3 CreateBucket service invocation
*/

#include "soapAmazonS3SoapBindingProxy.h"
#include "AmazonS3SoapBinding.nsmap"
#include <fstream>

// Make allocation of primitive values quick and easy:
template<class T>
T * soap_make(struct soap *soap, T val) {
    T *p = (T*)soap_malloc(soap, sizeof(T));
    *p = val;
    return p;
}

// Make base64-encoded, HMAC-SHA1 hashed signature for AWS S3
std::string soap_make_s3__signature(struct soap *soap, char const *operation, char const *key) {
    std::string signature = "AmazonS3";
    signature += operation;
    char UTCstamp[40]; //to hold ISO 8601 time format
    time_t now;
    time(&now);
    strftime(UTCstamp, sizeof UTCstamp, "%Y-%m-%dT%H:%M:%S.000Z", gmtime(&now));
    signature += UTCstamp;
    // Get the HMAC-SHA1 digest of the signature string
    unsigned char * digest;
    digest = HMAC(EVP_sha1(), key, strlen(key), 
    (unsigned char*)(signature.c_str()), 
    signature.length(), NULL, NULL);        
    char signatureBase64[20];
    // Convert the digest to base64
    soap_s2base64(soap, digest, signatureBase64, sizeof signatureBase64);
    return std::string(signatureBase64);
}

// Read access keys from file generated by AWS CLI
bool getAWSKeys(std::string path, std::string user, std::string &accessKey, std::string &secretKey) {
    std::ifstream credentialsFile(path.c_str());
    if (!credentialsFile.is_open())
        return false;
    
    std::string line;
    while (std::getline(credentialsFile, line)) {
        // Keep going until we get to the desired user
        if (line.find(user) == std::string::npos)
            continue;
        
        while (std::getline(credentialsFile, line)) {
            // Keep going until we get to the access key lines
            if (line.find("aws_access_key_id") == std::string::npos)
                continue;

            // Grab keys and trim whitespace
            size_t first, last;
            accessKey = line.substr(line.find_first_of('=')+1);
            first = accessKey.find_first_not_of(' ');
            if (first == std::string::npos)
                return false;
            last = accessKey.find_last_not_of(' ');
            accessKey.substr(first, last-first+1).swap(accessKey); 

            std::getline(credentialsFile, line);
            secretKey = line.substr(line.find_first_of('=')+1);
            first = secretKey.find_first_not_of(' ');
            if (first == std::string::npos)
                return false;
            last = secretKey.find_last_not_of(' ');
            secretKey.substr(first, last-first+1).swap(secretKey);
            
            return true;
        }

    }
    return false;
}

int main(int argc, char **argv) {    
    // Load AWS keys from file
    std::string accessKey, secretKey;
    // Use the path to your AWS credentials file
    std::string credentialsFile = (argc > 2 ? argv[2] : "path_to_aws_credentials_file"); 
    std::string user = "default";
    if (!getAWSKeys(credentialsFile, user, accessKey, secretKey)) {
        std::cout << "Couldn't read AWS keys for user " << user 
                  << " from file " << credentialsFile << '\n';
        return 0;
    }
    
    // Create a proxy to invoke AWS S3 services
    AmazonS3SoapBindingProxy aws(SOAP_XML_INDENT);

    // Create bucket
    
    // Set the arguments of the CreateBucket service operation
    _s3__CreateBucket createBucketReq;
    std::string bucketName = (argc > 1 ? argv[1] : "BucketName");
    createBucketReq.Bucket = bucketName;
    createBucketReq.AWSAccessKeyId  = soap_new_std__string(aws.soap);
    *createBucketReq.AWSAccessKeyId = accessKey;      
    createBucketReq.Timestamp       = soap_make(aws.soap, time(0));
    createBucketReq.Signature       = soap_new_std__string(aws.soap);
    *createBucketReq.Signature      = soap_make_s3__signature(aws.soap, 
                                                              "CreateBucket", 
                                                              secretKey.c_str());
                                                              
    // Store the result of the service
    _s3__CreateBucketResponse createBucketRes;

    // Create a bucket
    if (aws.CreateBucket(&createBucketReq, createBucketRes)) {
        aws.soap_stream_fault(std::cerr);
    }
    /*
        NOTE: you must add the line:
           _s3__CreateBucketResponse = $ s3__CreateBucketResult* CreateBucketResponse;
        to the typemap.dat file because Amazon's response doesn't match
        their promised schema. This adds the variable CreateBucketResponse
        to the _s3__CreateBucketResponse class so we can access the response.
    */
    else if (createBucketRes.CreateBucketResponse) {
        s3__CreateBucketResult &result = *createBucketRes.CreateBucketResponse;
        std::cout << "You are the owner of bucket '" << result.BucketName << "'." << std::endl;
    }

    // Delete all managed data
    aws.destroy();

    return 0;
}

The C code look similar, with the main difference being the use of function calls instead of method invocations, i.e. soap_call___s3__CreateBucket(&createBucketReq, &createBucketRes). All this is explained in the generated aws-s4.h file.

Compile the generated files with your source code:

c++ -DSOAP_MAXDIMESIZE=104857600 -DWITH_OPENSSL -o createbucket createbucket.cpp soapAmazonS3SoapBindingProxy.cpp soapC.cpp stdsoap2.cpp -lssl -lcrypto

The SOAP_MAXDIMESIZE=104857600 ensures that DIME attachment sizes can be large enough while preventing denial of service attacks using DIME. The DIME header has the attachment size, so an attacker could set that arbitrary large to deplete memory resources. Other posts failed to mention this.

Run createbucket, and a new bucket will be created.

In the final .cpp file, note that we check the command line arguments (argv) when setting credentialsFile and bucketName. This allows the program to be called with arguments:

./createbucket BucketName path_to_credentials_file

For more details on all of this, I suggest to read the excellent CodeProject article on How to use AWS S3 in C++ with gSOAP by Chris Moutsos from which parts of this explanation originates.