Bulkupload to AppEngine

I found that the Bulkupload is very useful in AppEngine, please refer to here for documentation. The configuration is pretty much self explanatory:
transformers:
- kind: Unit
  connector: csv
  property_map:
    - property: id
      external_name: id
      import_transform: int
    - property: type
      external_name: type
    - property: unit_id
      external_name: unit_id
    - property: title
      external_name: title
    - property: release_date
      external_name: release_date
    - property: now_available
      external_name: now_available
      import_transform: 'lambda x: True if x == "True" else False'

I like the import_transform option a lot, which allows a lot of creativity.

Just tried a few and worked beautifully:


CGuo@CGUO-E8420 ~/Dropbox/docs/workspace4.2/coursebuilder/data
$  appcfg.py upload_data --url=http://localhost:8083/_ah/remote_api --config_file=../bulkloader.yaml --filename=unit.csv --kind=Unit
03:22 PM Uploading data records.
[INFO    ] Logging to bulkloader-log-20120920.152255
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Please enter login credentials for localhost:8083
Email: mycguo@gmail.com
Password for mycguo@gmail.com:
[INFO    ] Opening database: bulkloader-progress-20120920.152255.sql3
[INFO    ] Connecting to localhost:8083/_ah/remote_api
[INFO    ] Starting import; maximum 10 entities per post
..
[INFO    ] 11 entities total, 0 previously transferred
[INFO    ] 11 entities (3197 bytes) transferred in 18.7 seconds
[INFO    ] All entities successfully transferred


CGuo@CGUO-E8420 ~/Dropbox/docs/workspace4.2/coursebuilder/data
$  appcfg.py upload_data --url=http://localhost:8083/_ah/remote_api --config_file=../bulkloader.yaml --filename=lesson.csv --kind=Lesson
03:25 PM Uploading data records.
[INFO    ] Logging to bulkloader-log-20120920.152522
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Please enter login credentials for localhost:8083
Email: mycguo@gmail.com
Password for mycguo@gmail.com:
[INFO    ] Opening database: bulkloader-progress-20120920.152522.sql3
[INFO    ] Connecting to localhost:8083/_ah/remote_api
[INFO    ] Starting import; maximum 10 entities per post
...
[INFO    ] 29 entities total, 0 previously transferred
[INFO    ] 29 entities (14656 bytes) transferred in 14.4 seconds
[INFO    ] All entities successfully transferred

Comments

blog comments powered by Disqus