Using libclang to parse in C++ in Python

Question

After some research and a few questions, I ended up exploring libclang library in order to parse C++ source files in Python.

Given a C++ source

int fac(int n) {
    return (n>1) ? n∗fac(n−1) : 1;
}

for (int i = 0; i < linecount; i++) {
   sum += array[i];
}

double mean = sum/linecount;

I am trying to identify the tokens fac as a function name, n as variable name, i as a variable name, mean as variable name, along with each ones position. I interested in eventually tokenizing them.

I have read some very useful articles (eli's, Gaetan's) as well as some stack overflow questions 35113197, 13236500.

However, given I am new in Python and struggling to understand the basics of libclang, I would very much appreciate some example chunk of code which implements the above for me to pick up and understand from.

Answer 1

It's not immediately obvious from the libclang API what an appropriate approach to extracting token is. However, it's rare that you would ever need (or want) to drop down to this level - the cursor layer is typically much more useful.

However, if this is what you need - a minimal example might look something like:

import clang.cindex

s = '''
int fac(int n) {
    return (n>1) ? n*fac(n-1) : 1;
}
'''

idx = clang.cindex.Index.create()
tu = idx.parse('tmp.cpp', args=['-std=c++11'],  
                unsaved_files=[('tmp.cpp', s)],  options=0)
for t in tu.get_tokens(extent=tu.cursor.extent):
    print t.kind

Which (for my version of clang) produces

TokenKind.KEYWORD
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.KEYWORD
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION
TokenKind.KEYWORD
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.LITERAL
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.IDENTIFIER
TokenKind.PUNCTUATION
TokenKind.LITERAL
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION
TokenKind.LITERAL
TokenKind.PUNCTUATION
TokenKind.PUNCTUATION