数据处理

csv处理

  • i-csv
  • import-csv
  • export-csv
  • json2csv
  • fastcsv

json2csv

json to csv converter

https://github.com/i5ting/json2csv

import-csv

https://github.com/i5ting/import-csv

默认使用gbk编码

  1. var import_csv = require('import-csv')
  3. import_csv('test.csv',function(err, data){
  4.   console.log(data);
  5. })

指定字符编码机

  1. var import_csv = require('import-csv')
  3. import_csv('test.csv',function(err, data){
  4.   console.log(data);
  5. }, 'gbk')

export-csv

https://github.com/i5ting/export-csv

默认使用gbk编码

  1. var export_csv = require('.')
  3. var data = [
  4.   {a:1,b:2},
  5.   {a:2,b:2},
  6.   {a:3,b:2}
  7. ]
  8. export_csv(data, 'test1.csv')

过滤item，重写数据

  1. var export_csv = require('.')
  3. var data = [
  4.   { a: 1, b: 2 },
  5.   { a: 2, b: 2 },
  6.   { a: 3, b: 2 }
  7. ]
  8. export_csv(data, 'test3.csv', function (item) {
  9.   for (var key in item) {
  10.     item[key] = 'yy +' + item[key];
  11.   }
  12.   return item;
  13. })

指定回调函数

  1. var export_csv = require('.')
  3. var data = [
  4.   {a:1,b:2},
  5.   {a:2,b:2},
  6.   {a:3,b:2}
  7. ]
  8. export_csv(data, 'test2.csv', function (item) {
  9.   return item;
  10. }, function () {
  11.   console.log('end...');
  12. })

指定回调函数，并生成header

  1. var export_csv = require('.')
  3. var data = [
  4.   {a:1,b:2},
  5.   {a:2,b:2},
  6.   {a:3,b:2}
  7. ]
  8. export_csv(data, 'test2.csv', function (item) {
  9.   return item;
  10. }, function () {
  11.   console.log('end...');
  12. }, true)

mongodb如何统计

  1. var json2csv = require('json2csv');
  2. require('../db.js');
  3. var fs = require('fs');
  4. var Order = require('../app/models/order');
  5. var Activity = require('../app/models/activity');
  6. var Wechat = require('../app/models/wechat');
  7. var Contact = require('../app/models/contact');
  9. function main(){
  10.   Order.model.aggregate([
  11.     {$match : {status2 : {$exists : true} }},
  12.     {$group : {_id : "$activity" , count : {$sum : "$product_count"}}}
  13.     ]).exec(function (err, data) {
  14.         var obj = [];
  15.         for (var i = data.length - 1; i >= 0; i--) {
  16.           var json = {
  17.             "url" : "http://shop.mengxiaoban.cn/iscroll.html?id=" + String(data[i]._id),
  18.             "count" : String(data[i].count)
  19.           }
  20.           obj.push(json);
  21.           console.log(json.url);
  22.         }
  23.         var opts = {
  24.           data: obj,
  25.           fields: ['url','count'],
  26.           fieldNames: ['url','count'],
  27.           quotes: ''
  28.         };
  29.         json2csv(opts, function(err, csv) {
  30.           if (err) console.log(err);
  31.           fs.writeFile('./1119todayurl.csv', csv, function(err) {
  32.             if (err) throw err;
  33.             console.log('file saved');
  34.           });
  35.         });
  36.       })
  37.   }
  38. }
  40. main();

AGGREGATION 关联

《SQL to Aggregation Mapping Chart》

http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/

优化查询速度

了解索引优化

  1.  ContactSchema = new Schema({
  2.     ...
  3.     owner: {
  4.       type: Schema.ObjectId,
  5.       required: true,
  6.       index: true
  7.     }
  8. });

也可以这样的

  1. ContactSchema.ensureIndexes(owner);

了解explain

  1.  db.usermodels.find({
  2.   '_id' :{ 
  3.     "$gt" :ObjectId("55940ae59c39572851075bfd")
  4.   } 
  5. }).explain()

关注点

  • stage：查询策略
  • nReturned：返回的文档行数
  • needTime：耗时（毫秒）
  • indexBounds：所用的索引

http://docs.mongodb.org/v3.0/reference/explain-results/

了解profile

profile级别有三种：

  • 0：不开启
  • 1：记录慢命令，默认为大于100ms
  • 2：记录所有命令
  • 3、查询profiling记录
    开启
  1. db.setProfilingLevel(2, 20)

默认记录在system.profile中

  1. db['system.profile'].find()

MONGODB的部署

  • replset
  • shard

我写的《 mongodb运维之副本集实践》

https://cnodejs.org/topic/5590adbbebf9c92d17e734de