bulkloader

问题 I'm using app engine's bulkloader to import a CSV file into my datastore. I've got a number of columns that I want to merge into one, for example they're all URLs, but not all of them are supplied and there is a superseding order, eg: url_main url_temp url_test I want to say: "Ok, if url_main exists, use that, otherwise user url_test and then use url_temp " Is it, therefore, possible to create a custom import transform that references columns and merges them into one based on conditions? 回答1:

问题 In short: how can I configure bulkloader to insert data into 2 models with references? I have a person and fruit class, with person linking to fruit: class Fruit(db.Model): name = db.StringProperty() class Person(db.Model): name = db.StringProperty() customer = db.ReferenceProperty(Fruit) And I want to upload this CSV data: Name,Fruit Bob,Banana Joe,Apple Tim,Banana I tried using create_foreign_key as in the docs: transformers: - kind: fruit connector: csv property_map: - property: fruit

问题 I'm currently using the --kind parameter to specify which kind to download and the --filename to specify the name of the csv file to produce. The --rps_limit , --bandwidth_limit and --batch_size are used to speed up the download. For example, to download my Game kind I'm using: appcfg.py download_data --config_file=bulkloader.yaml --kind=Game --filename=game.csv --application=MyAppId --url=http://MyAppId.appspot.com/remote_api --rps_limit=500 --bandwidth_limit=2500000 --batch_size=100 So is

问题 The docs say appcfg.py --dump --app_id=<app-id> \ --url=http://<appname>.appspot.com/remote_api \ --filename=<data-filename> The thing itself barfs: Usage: appcfg.py [options] <action> appcfg.py: error: no such option: --dump $ cat /usr/local/share/google_appengine/VERSION release: "1.3.4" Thoughts? 回答1: I found that I had to use download_data instead of --dump and --application instead of --app_id, for example: appcfg.py download_data --application=app_id --url=http://etc --filename=file 回答2

问题 I have a map reduce job that connects to HBASE and I can't figure out where I am running into this error: Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.eclipse.jdt.internal

问题 I've got a class that is using db.SelfReferenceProperty to create a tree-like structure. When trying to populate the database using appcfg.py upload_data -- config_file=bulkloader.yaml --kind=Group --filename=group.csv (...) , I'm getting an exception saying BadValueError: name must not be empty . (Full stack below) I tried ordering the data to make sure that a Groups that had a foreign key pointing at them were first. That didn't work. By commenting from the bulkloader.yaml the line making

问题 I am converting a script to use the new bulkloader. (What was wrong with the original bulkloader? - I prefer writing Python to editing configuration files...) Anyway, I want to prevent duplicates by assigning a combination of properties to the key. The docs say: If you want to use or calculate a key from the import data, specify a key using the same syntax as the property map; that is, external_name, import_template, and so on. All the examples apply a transform to the current property. How

问题 Has anyone ever encountered a situation where your assets (image png files) got deleted from your web path? Let me explain it little more clearly. I am loading some images located in my localhost (not in flex4 application path) from my flex4 application using the loader and also with BulkLoader This is the second time it happened that some of the images got deleted from the path which are in localhost. I am not sure what is causing this? is that the loader? or bulkloader or the webserver?

问题 I was looking at Where are the reference pages of the Google App Engine bulkloader transform? and figured out most of my bulkloader.yaml configuration with the exception of one case. One of my Kinds 'Product' has a property called site. If present this is a deep key for a Customer Kind and a Site kind. Now the problem I am having is with the non_if_empty. In the below case it will not ever create the deep key. It always comes back none. If I remove the transform.none_if_empty it will fail as

问题 I'm trying to upload some data to my App Engine datastore using the bulkuploader. For one of my entity types, I have one property that is calculated from another, so I'd really like to do some post-processing on each entity as it's imported to do this calculation. I keep seeing brief mentions of the post_import_function transform tag, but no real comprehensive documentation or examples. For now, I'm just trying to do a simple test just to get my post_import_function to work. My entity model: