How to scan BigQuery table with DLP looking for sensitive data?

问题

I would like to analyze my tables in BigQuery using DLP. It is possible ? How to do that ?

回答1:

It's possible. You need to define your storage_config to use BigQuery. If you want to save findings in another table, then add a save_findings action to the job config. Without actions you will only have access to the job's finding summary via the projects.dlpJobs.get method.

Follow the example in python to invoke DLP to scan BigQuery:

client_dlp = dlp_v2.DlpServiceClient.from_service_account_json(JSON_FILE_NAME)

inspect_job_data = {
    'storage_config': {
        'big_query_options': {
            'table_reference': {
                'project_id': GCP_PROJECT_ID,
                'dataset_id': DATASET_ID,
                'table_id': TABLE_ID
            },
            'rows_limit':10000,
            'sample_method':'RANDOM_START',
        },
    },
    'inspect_config': {
        'info_types': [
            {'name': 'ALL_BASIC'},
        ],
    },
    'actions': [
        {
            'save_findings': {
                'output_config':{
                    'table':{
                        'project_id': GCP_PROJECT_ID,
                        'dataset_id': DATASET_ID,
                        'table_id': '{}_DLP'.format(TABLE_ID)
                    }
                }

            },
        },
    ]
}
operation = client_dlp.create_dlp_job(parent=client_dlp.project_path(GCP_PROJECT_ID), inspect_job=inspect_job_data)

And a query to analyze the result:

client_bq = bigquery.Client.from_service_account_json(JSON_FILE_NAME)
# Perform a query.
QUERY = (
    'WITH result AS ('
    'SELECT'
    ' c1.info_type.name,'
    ' c1.likelihood,'
    ' content_locations.record_location.record_key.big_query_key.table_reference as bq,'
    ' content_locations.record_location.field_id as column '
    'FROM '
    ' `'+ GCP_PROJECT_ID +'.'+  DATASET_ID +'.'+  TABLE_ID  +'_DLP` as c1 '
    'CROSS JOIN UNNEST(c1.location.content_locations) AS content_locations '
    'WHERE c1.likelihood in (\'LIKELY\',\'VERY_LIKELY\'))'
    'SELECT r.name as info_type, r.likelihood, r.bq.project_id, r.bq.dataset_id,'
    ' r.bq.table_id, r.column.name, count(*) as count  FROM result r GROUP By 1,2,3,4,5,6 '
    'ORDER By COUNT DESC'
)
query_job = client_bq.query(QUERY)  # API request
rows = query_job.result() 
for row in rows:
    print('RULES: {} ({}) | COLUMN: {}.{}.{}:{} | count->{}'.format
          (row.info_type, row.likelihood, row.project_id,row.dataset_id,row.table_id,row.name, row.count)

You can find more details here

来源：https://stackoverflow.com/questions/54455062/how-to-scan-bigquery-table-with-dlp-looking-for-sensitive-data

标签

google-bigquery

google-cloud-dlp