Getting an empty CSV when downloading from URL in Cpp

问题

My goal is to get a CSV or XLS from a specific URL using Cpp.

When opening the following link

http://www.centrodeinformacao.ren.pt/userControls/GetExcel.aspx?T=CRG&P=01-01-2007&variation=PT

, can see in Browser tools

a 302 redirect and the file being actually downloaded from the following URL

http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=02-01-2007&variation=PT

as shown in the next image (Request URL)

If I go to any of the two links manually, the a .xls file downloads just fine so we might as well use the one after redirection.

I've decided to move on an use libcurl with Visual Studio 2017 on my W10 machine. The recommended way to include libcurl in a Visual Studio 2017 project is to use vcpkg and that's what I used.

1. Install vcpkg

Open Git Bash, cd C:/Program Files/ and cloned this repo.

Open Command Prompt, cd C:/Program Files/vcpkg, run bootstrap-vcpkg.bat

and after run vcpkg integrate install

2. Install libcurl

Run vcpkg install curl

3. Create a new project

Simply create Visual C++ > Windows Desktop > Windows Console Application

to be able to use #include <curl/curl.h> right away

4. Current Result

Then, inspired in the following answers

https://stackoverflow.com/a/6951203/9415908
https://stackoverflow.com/a/1636415/9415908

And using the following code

#include "pch.h"
#define _CRT_SECURE_NO_WARNINGS
#include <iostream>
#include <iostream>
#include <stdio.h>
#include <curl/curl.h>
#include <string.h>

size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
    size_t written = fwrite(ptr, size, nmemb, stream);
    return written;
}

void downloadFile(const char* url, const char* fname) {
    CURL *curl;
    FILE *fp;
    CURLcode res;
    curl = curl_easy_init();
    if (curl) {
        fp = fopen(fname, "wb");
        curl_easy_setopt(curl, CURLOPT_URL, url);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
        res = curl_easy_perform(curl);
        curl_easy_cleanup(curl);
        fclose(fp);
    }
}

int main(void) {

    downloadFile("http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=01-01-2007&variation=PT", "C:\\Users\\molecoder\\Desktop\\test.csv");

}

I can see a test.csv in the desired folder but it's an empty file.

回答1:

Once going to that specific URL, a .xls file downloads. I don't mind getting a XLS instead of a CSV and so changing it to I was able to get the file as expected.

downloadFile("http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=01-01-2007&variation=PT", "C:\\Users\\molecoder\\Desktop\\test.xls");

This is the final code

#include "pch.h"
#define _CRT_SECURE_NO_WARNINGS
#include <iostream>
#include <iostream>
#include <stdio.h>
#include <curl/curl.h>
#include <string.h>

size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
    size_t written = fwrite(ptr, size, nmemb, stream);
    return written;
}

void downloadFile(const char* url, const char* fname) {
    CURL *curl;
    FILE *fp;
    CURLcode res;
    curl = curl_easy_init();
    if (curl) {
        fp = fopen(fname, "wb");
        curl_easy_setopt(curl, CURLOPT_URL, url);
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
        res = curl_easy_perform(curl);
        curl_easy_cleanup(curl);
        fclose(fp);
    }
}

int main(void) {

    downloadFile("http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=01-01-2007&variation=PT", "C:\\Users\\molecoder\\Desktop\\test.xls");

}

回答2:

You have to understand how your shell works.

Note: When you use system() it is the shell that executes the command.

Your command:

http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=01-01-2007&variation=PT

Your shell is treating the "&" character as the "and" operator. Execute the left and right side of the "and" as commands. So it is treating the above as three commands:

http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG
P=01-01-2007
variation=PT

It fails on the second command P=01-01-2007

To fix this you may want to quote the string.

"http://www.centrodeinformacao.ren.pt/_layouts/CI.GetExcel/SafeGetExcel.aspx?T=CRG&P=01-01-2007&variation=PT"

 op.insert(0, 1, '"');
 op.append(1, '"');
 system(op.c_str());

来源：https://stackoverflow.com/questions/62850908/getting-an-empty-csv-when-downloading-from-url-in-cpp

标签

c++

file

download

visual-studio-2017

libcurl