ISE Technology Innovation 2008 | Web CouchDB - Time to relax!! | Yohei Sasaki [email protected] 2008/08/01 ISE Technology Innovation 2008 | ご注意 この資料に含まれる情報は可能な限り正確を期しておりますが、日本アイ・ビー・エム システムズ・エンジニアリング株式会 社の正式なレビューを受けておらず、当資料に記載された内容に関して日本アイ・ビー・エム システムズ・エンジニアリング 株式会社は何ら保証するものではありません。 従って、この情報の利用またはこれらの技法の実施はひとえに使用者の責任において為されるものであり、資料の内容によって 受けたいかなる被害に関しても一切の保証をするものではありません。 当資料をコピー等で複製することは、日本アイ・ビー・エム システムズ・エンジニアリング株式会社および執筆者の承諾なし ではできません。また、当資料に記載された製品名または会社名はそれぞれの各社の商標または登録商標です。 2 ISE Technology Innovation 2008 | IBM ( IBM IBM ) Apache Geronimo Cell Processor Ruby on Rails PHP on i5/OS ... Hadoop Apache CouchDB Web , . . 3 ISE Technology Innovation 2008 | Apache CouchDB Apache CouchDB 4 ISE Technology Innovation 2008 | Apache CouchDB 1. HTTP 3. MapReduce 2. ... ISE Technology Innovation 2008 | CouchDB Couch Cluster Of Unreliable Commodity Hardware http://incubator.apache.org/couchdb/ 6 ... ISE Technology Innovation 2008 | CouchDB 2008 6 0.8.0 svn trunk developerWorks Japan wiki CouchDB 1 2 3 CouchDB + (Rails – ActiveRecord) ... ”CouchDB developerWorks” 7 ISE Technology Innovation 2008 | Q. CouchDB 1. Document Oriented 2. REST 3. MapReduce 4. Erlang Impl 5. Replication 6. Other Q. Web 1. User Interface 3. Database 2. Programming Model 4. Other 皆様にいただいた回答を使って、 デモを進めていきます。 8 ISE Technology Innovation 2008 | Apache CouchDB 1. HTTP 3. MapReduce 2. Apache CouchDB ISE Technology Innovation 2008 | ( Web ... ) JSON HTTP REST style MapReduce JavaScript(or CRUD )= SQL 10 ISE Technology Innovation 2008 | “ ” ? http://en.wikipedia.org/wiki/Document-oriented_database ... “ ”=“ ” Web ... Wiki, Blog ...(etc) Web-CMS QA (search word, bookmark, ...) 11 JSON ISE Technology Innovation 2008 | CouchDB “ ” JSON JSON JSON アンケート用紙 アンケート用紙テンプレートドキュメント 用紙テンプレートドキュメント { "type" : “Survey", “created_by" : “[email protected]", "tags" : [“CouchDB", “rest", ... ], “title” : ... “content” : ... } 12 HTTP ISE Technology Innovation 2008 | CouchDB HTTP GET POST PUT DELETE : : : : HTTP CRUD (POST ) URI GET /survey/aabbcc CouchDB Server HTTP { "type" : “Survey", “created_by" : “[email protected]", "tags" : [“CouchDB", “rest", ... ], “title” : ... “content” : ... Client } 13 MapReduce ISE Technology Innovation 2008 | CouchDB MapReduce JavaScript SpiderMonkey 2 MapReduce ? => No. 14 MapReduce ISE Technology Innovation 2008 | 2004 Operating Systems Design and Implementation MapReduce: Simplified Data Processing on Large Clusters OSDI’04 / Jeff Dean, Sajay Ghemawat (Google) http://labs.google.com/papers/mapreduce.html Apache Hadoop CouchDB MapReduce 15 CouchDB ISE Technology Innovation 2008 | JavaScript MapReduce (Key, Value) (Map) key Optional URI Optional JavaScript Value (Reduce) reduce map K1 V1 K1 V2 K2 K3 V3 V4 K1 V’1 V3 K2 V’3 V4 K3 V’4 K1 V1 K2 K3 V2 16 MapReduce ISE Technology Innovation 2008 | { { ... "tags" : [“CouchDB", “REST", ... ], ... } count_by_tags : { map : function(doc){ for(var i in doc.tags) emit(doc.tags[i], 1) }, reduce : function(key, values){ return sum(values); } } } map reduce key value CouchDB 1 key values CouchDB 1 1 CouchDB 2 CouchDB 1 K1 K2 1 1 K1 1 K1 1 K2 1 K2 1 17 ISE Technology Innovation 2008 | CouchDB ” ” ” ” replication replication 18 CouchDB ISE Technology Innovation 2008 | MVCC (Multi Version Concurrency Control) Read / Write Read Write Erlang Write HTTP Apache Incubator Wiki ( Wiki Windows developerWorks / ) Linux Mac OS ? Ubuntu 8.04 Desktop + CouchDB 0.8.0 19 ( )CouchDB ( ) ISE Technology Innovation 2008 | 0.8.0-incubating erlang http => mochiweb Lucene => Web jQuery (Futon) http://incubator.apache.org/couchdb/ 20 0: Web ISE Technology Innovation 2008 | Futon Web http://localhost:5984/_utils/ JavaScript(+ jQuery) Undocument Firebug + Futon Futon !? 21 QA Time ISE Technology Innovation 2008 | ( )Apache CouchDB ) Web ( JSON HTTP REST style MapReduce JavaScript(or CRUD )= SQL 22 ISE Technology Innovation 2008 | Apache CouchDB 1. 2.HTTP 3. MapReduce DB ISE Technology Innovation 2008 | HTTP HTTP URI / : =1 http://{host}:{port}/{dbname}/{doc_id} HTTP verb GET POST PUT DELETE HTTP : : : : (POST ) JSON 24 ISE Technology Innovation 2008 | CouchDB 4 DB CRUD http://{host}:{port}/{dbname} DB CRUD http://{host}:{port}/{dbname}/{doc_id} / DB http://{host}:{port}/{dbname}/_design/ http://{host}:{port}/{dbname}/_view/ 25 HTTP : ISE Technology Innovation 2008 | POST http://localhost:5984/technight/ technight DELETE http://localhost:5984/technight/ technight 4XX HTTP 200 OK 5XX 26 ISE Technology Innovation 2008 | Q. CouchDB 1. Document Oriented 2. REST 3. MapReduce 4. Erlang Impl 5. Replication 6. Other Q. Web 1. User Interface 3. Database 2. Programming Model 4. Other 皆様にいただいた回答を使って、 デモを進めていきます。 27 1: ISE Technology Innovation 2008 | DB technight curl curl –v –X PUT http://localhost:5984/technight 28 HTTP : ISE Technology Innovation 2008 | (ID ) POST http://localhost:5984/technight/ technight ID id (ID rev JSON ) PUT http://localhost:5984/technight/document1 document1 ID GET http://localhost:5984/technight/document1 document1 ID 後述! 29 ISE Technology Innovation 2008 | Q. CouchDB 1. Document Oriented 2. REST 3. MapReduce 4. Erlang Impl 5. Replication 6. Other Q. Web 1. User Interface 3. Database 2. Programming Model 4. Other 皆様にいただいた回答を使って、 デモを進めていきます。 30 2: POST ISE Technology Innovation 2008 | != == 今回はこのフォーマットで入れていくことにします { “A1” : “REST”, “A2” : “ProgrammingModel” } (例えば)「あ、やっぱり複数選択可能にしよう」 { “A1” : [“REST”, “MapReduce] “A2” : [“ProgrammingModel”] } 31 2: ISE Technology Innovation 2008 | POST ID (CouchDB ) POST curl –v –X POST –d ¥ “{ ‘A1’ : ‘REST’, ‘A2’ : ‘Programming Model’ }” ¥ http://localhost:5984/technight 32 ISE Technology Innovation 2008 | HTTP Client HTTP = CouchDB + JSON JSON json.org JSON JSON Hash, 33 1: Ruby ISE Technology Innovation 2008 | HTTP net/http JSON ruby json gem install json http://json.rubyforge.org/ Rails / ActiveSupport::JSON Unicode decode developerWorks http://www.ibm.com/developerworks/jp/web/library/j_wa-couchdb02/ 34 2: ProjectZero ISE Technology Innovation 2008 | HTTP (1/2) Connection API JSON zero.json 35 2: ProjectZero (2/2) ISE Technology Innovation 2008 | def document = [ “A1” : “REST”, “A2” : “ProgrammingModel” ] def body = Json.encode(document) Connection.Response res = Connection.doPOST( “http://localhost:5984/zerodb”, [“Content-Type” : “application/json”], body) def status = res.getResponseStatus() if( status == 201 ){ ... } 36 CouchDB ISE Technology Innovation 2008 | PUT http://localhost:5984/technight/document1 document1 ID _rev ( ) 201 DELETE http://localhost:5984/technight/document1?rev=1234 document1 1234 ?? 37 ISE Technology Innovation 2008 | CouchDB JSON PUT /{dbname}/{doc_id} { “_id” : “{doc_id}”, “_rev” : “{lastest_revision}”, ... } DELETE /{dbname}/{doc_id}?rev={latest_revision} 38 – ISE Technology Innovation 2008 | PUT /db/doc1 { { “_rev” : “1234” “hoge” : “fuga” } “hoge” : “fuga” } { “id” : “doc1” “rev” : “1234” } PUTリクエストのレスポンスには(成功すれば)常に revプロパティが含まれる 39 – ISE Technology Innovation 2008 | PUT /db/doc1 { “_rev” : “1234” “hoge” : “foo” } {{ “_rev” “rev” :: “5678” “1234” “hoge” : “foo” “hoge” : “fuga” }} _rev に最新のリビジョン番号を指定してPUTする 追記し、新しいrevを割り振る 最新のrevと一致しない場合は 412 Precondition failed を返す 40 – ISE Technology Innovation 2008 | DELETE /db/doc1?rev=5678 {{ { “_rev” : :“1357” “rev” “5678” “rev” : “1234” “_deleted”: “true” “hoge” : “foo” “hoge” : “fuga” }} } rev に最新のリビジョン番号を指定してDELETEする 削除ドキュメント(_deletedがtrue)を追記する この状態で、GET /db/doc1 をリクエストすると404 Not Found になる 41 URI ISE Technology Innovation 2008 | PUT /db/doc1 { “hoge” : “renew” } PUT – {{ {{ “_rev” :: “1357” “2468” “_rev” “rev” : “5678” “rev” : “1234” “hoge” : “renew” “_deleted”: “true” “hoge” : “foo” }} “hoge” : “fuga” }} 新規作成時と同じ形(revは不要)でPUT可能 42 – URI ISE Technology Innovation 2008 | GET /db/doc1?revs=true {{ {{ “_rev” :: “1357” “2468” “_rev” “rev” : “5678” “rev” : “1234” “hoge” : “renew” “_deleted”: “true” “hoge” : “foo” }} “hoge” : “fuga” }} { “rev” : “2468”, “_revs”: [ “2468”, “1357”, “5678”, “1234” ] } すべてのリビジョン番号を配列で返す 43 – ISE Technology Innovation 2008 | GET /db/doc1?rev=1234 {{ {{ “_rev” :: “1357” “2468” “_rev” “rev” : “5678” “rev” : “1234” “hoge” : “renew” “_deleted”: “true” “hoge” : “foo” }} “hoge” : “fuga” }} { “_rev” : “1234” “hoge” : “fuga” } リビジョンを指定して取得することは可能 44 ISE Technology Innovation 2008 | JSON ” ” JSON PUT GET (_attachments) base64 base64 ( base64 ) 45 ( ISE Technology Innovation 2008 | ) _attachments “ ” : {“content_type” : ... , “data” : ...} PUT { “_attachments”: { “a.txt” : { “content_type” : “text¥/plain”, “data” : “base64 encodeされた値” }, “b.txt” : { ... } } } 46 ( ISE Technology Innovation 2008 | ) GET /{dbname}/{doc_id} (content_type length) { “_attachments”: { “a.txt” : { “content_type” : “text¥/plain”, “length” : “長さ” }, “b.txt” : { ... } } } GET /{dbname}/{doc_id}/{filename} base64 HTTP CouchDB Content-Type “content_type” 47 1 ISE Technology Innovation 2008 | [POST|PUT] http://{host}:{port}/{db}/_bulk_docs POST, PUT “docs” : [ { JSON id }, { }, ... ] { “docs” : [ { “A1”: “MapReduce”, ... }, { “A1”: “REST”, ... }, ... ] } 48 HTTP ISE Technology Innovation 2008 | CouchDB – HTTP ” ” CouchDB /{dbname}/{doc_id} DELETE URI JSON 1 49 break time ... MapReduce ISE Technology Innovation 2008 | 10 50 ISE Technology Innovation 2008 | Apache CouchDB 1. 2. HTTP 3.MapReduce ISE Technology Innovation 2008 | Q. CouchDB 1. Document Oriented 2. REST 3. MapReduce 4. Erlang Impl 5. Replication 6. Other Q. Web 1. User Interface 3. Database 2. Programming Model 4. Other 皆様にいただいた回答を使って、 デモを進めていきます。 52 ( ) 2: POST ISE Technology Innovation 2008 | 今回はこのフォーマットで入れてます { “A1” : “REST”, “A2” : “ProgrammingModel” } 53 2: POST – ISE Technology Innovation 2008 | { "docs" : [ { "A1" : "REST", "A2" : "Programming Model" }, { "A1" : "Document Oriented", "A2" : "User Interface" }, { "A1" : "REST", "A2" : "Programming Model" }, { "A1" : "Document Oriented", "A2" : "Database" }, { "A1" : "MapReduce", "A2" : "User Interface" }, { "A1" : "Document Oriented", "A2" : "User Interface" }, { "A1" : "MapReduce", "A2" : "Database" }, { "A1" : "MapReduce", "A2" : "Programming Model" }, { "A1" : "Document Oriented", "A2" : "Other" }, { "A1" : "REST", "A2" : "Programming Model" }, { "A1" : "REST", "A2" : "Programming Model" }, { "A1" : "Other", "A2" : "Programming Model" }, { "A1" : "Other", "A2" : "Programming Model" }, { "A1" : "Replication", "A2" : "Database" }, { "A1" : "Replication", "A2" : "User Interface" }, { "A1" : "MapReduce", "A2" : "Database" }, { "A1" : "MapReduce", "A2" : "Database" }, { "A1" : "REST", "A2" : "User Interface" }, { "A1" : "MapReduce", "A2" : "Database" }, { "A1" : "Document Oriented", "A2" : "User Interface" }, { "A1" : "Replication", "A2" : "Programming Model" }, { "A1" : "Document Oriented", "A2" : "Database" } ] } 54 CouchDB JSON URI 2. 3. (*) ISE Technology Innovation 2008 | 1. 55 ISE Technology Innovation 2008 | 1. JSON JSON { “language” : “javascript” “views” : “View名” : { “map” : “map関数の文字列”, “reduce” : “reduce関数の文字列” }, “View名” : { reduce はoptional ... } } } 56 2. 3. ISE Technology Innovation 2008 | 1. PUT JSON URI http://{host}:{port}/{dbname}/_design/{designname} dbname : designname : PUT _id, _rev DELETE GET _id, _rev PUT PUT URI http://{host}:{port}/{dbname}/_view/{designname}/{view} view : views View 57 ISE Technology Innovation 2008 | PUT http://localhost:5984/technight/_design/answers { “language” : “javascript” “views” : “all” : { : “function(doc){ “map” emit(null, doc); }”, }, } 見やすさのため改行 } 58 ISE Technology Innovation 2008 | View URI GET http://localhost:5984/technight/_view/answers JSON { total_rows : N, offset : M, rows : [ { id: map ID , key: emit() , value: emit() }, ... ]} {"total_rows":4,"offset":0,"rows":[ {"id":“....", "key":null,"value":{...}, {"id":“....", "key":null,"value”:{...}, {"id":“....", "key":null,"value" :{...}, {"id":“....”, "key":null,"value":{...} ]} 59 CouchDB - : Query Server function (){..} HTTP API ISE Technology Innovation 2008 | PUT /technight/_design/answers 60 CouchDB View URI - (1/2) GET /technight/_view/answers/all Query Server function (){..} HTTP API ISE Technology Innovation 2008 | PUT /technight/_design/answers 61 CouchDB View URI - (2/2) GET /technight/_view/answers/all View Query Server function (){..} HTTP API ISE Technology Innovation 2008 | PUT /technight/_design/answers 62 QueryServer ISE Technology Innovation 2008 | CouchDB / View ”language” CouchDB (couch.ini ) language: javascript /usr/local/bin/couchjs /usr/local/share/couchdb/server/main.js javascript 63 QueryServer true function(doc) { emit(null, doc) } デザインドキュメントに 記述したmap関数 QueryServer map Query Server ISE Technology Innovation 2008 | [“add_fun”, “function(doc){ ... }”] “funciton(...)...” eval (main.js) 64 QueryServer DB 1 1 [[[k,v],[k,v]...]...] function(doc) { emit(null, doc) } デザインドキュメントで 定義したmap関数 Query Server ISE Technology Innovation 2008 | [“map_doc”, {“_id” : ...}] CouchDB QueryServer JSON QueryServer CouchDB emit(key,value) key,value JSON CouchDB JSON map QueryServer (DBDIR)/.{dbname}_design/{desig nname}.view 65 CouchDB - ( ) GET /technight/_view/answers/all View Query Server function (){..} HTTP API ISE Technology Innovation 2008 | PUT /technight/_design/answers 66 CouchDB ISE Technology Innovation 2008 | MapReduce map map 0 Key, Value emit(key, value) map View ( View ( View ) View K1 V1 K1 V2 K2 K3 V3 V4 QS ) map 67 map ISE Technology Innovation 2008 | key map View URI View URI reduce key,value Query key JSON rows 68 key ( A1 - ) ISE Technology Innovation 2008 | ”all_by_answer1” PUT http://localhost:5984/technight/_design/answers { “language” : “javascript” “views” : “all” : {...}, “all_by_answer1” : { “map” : “function(doc){ emit(doc.A1, doc); }”, }, } 見やすさのため改行 } 69 key ( ISE Technology Innovation 2008 | A1 - ) View URI GET http://localhost:5984/technight/_view/answers/all_by_answe r1 {"total_rows":4,"offset":0,"rows":[ {"id":“...”, "key":”MapReduce”, “value”:{...}, {"id":“...”, "key":”MapReduce”, “value”:{...}, {"id":“...”, "key":”Programming...”, “value”:{...}, {"id":“...”, "key":”REST”, “value”:{...}, ]} Key にA1の回答が含まれ、 Key順でソートされる 70 key ( ISE Technology Innovation 2008 | A1 - ) View URI GET http://localhost:5984/technight/_view/answers/all_by_answe r1?key=“REST” {"total_rows":4,"offset":0,"rows":[ {"id":“...", “key":”REST”, {"id":“...", "key":”REST”, ]} "value”:{...}, "value”:{...}, key=“REST” なものだけを取得 KeyへのQueryパラメーターは、JSON形式で指定し、 実際にはURL エンコードして渡す %22REST%22 71 Query ISE Technology Innovation 2008 | map ” map ” Query ” ” map Query key key View ... etc key Query (key=...) (startkey=...[&endkey=...]) (descending=false) key ” ” “ JSON ” 72 CouchDB View URI - Query Parameter GET /technight/_view/ answers/all_by_answer1?key=“REST” View - Query Server function (){..} HTTP API ISE Technology Innovation 2008 | PUT /technight/_design/answers Query 73 CouchDB ISE Technology Innovation 2008 | MapReduce reduce reduce map array key,value reduce array / key reduce map K1 V1 K1 V2 K2 K3 V3 V4 K1 V’1 V3 K2 V’2 V3 K2 V’3 K1 V1 K2 K2 V2 74 reduce CouchDB ISE Technology Innovation 2008 | ” count_by_answer1” { “language” : “javascript” “views” : 見やすさのため改行 “all” : {...} , “all_by_answer1” : {...} , “count_by_answer1” : { “map”: “function(doc){ emit(doc.A1, 1); }”, “reduce” : “function(key, values, rr){ return sum(values); }” } } 75 reduce CouchDB ISE Technology Innovation 2008 | View URI (Key ) GET http://localhost:5984/technight/_view/answers/count_by_answer1?key=“RE ST” {“rows":[ {“key”:null, “value”: 18} ]} View URI (Key ) GET http://localhost:5984/technight/_view/answers/count_by_answer1 {“rows":[ {“key”:null, “value”: 44} ]} 76 CouchDB View URI - : reduce function (){..} ) Query Server GET /employee/_view/ employee/count_by_answer1?key=“REST” (= HTTP API ISE Technology Innovation 2008 | PUT /employee/_design/answers Query Reduce 77 reduce ISE Technology Innovation 2008 | map Query ( (doc.A2.Select, 1) key=“REST” reduce ) reduce 1 (key, [1,1,1,1,1, ...]) key=XXXX にマッチする Key, Valueペアのvalue部分がリストになり reduce関数に渡される。 map REST 1 null REST 1 .. .. 1 1 2 78 group ISE Technology Innovation 2008 | Wiki group_level undocumented {“rows":[ {“key”:”REST”, {“key”:”MapReduce”, ,,, ]} “value”: 18}, “value”: 10}, {“rows":[ {“key”:[”REST”, “Database”], {“key”:[”REST”, “Prog...”], {“key”:[”REST”, “User...”], ... ]} “value”: 3}, “value”: 9}, “value”: 6}, 79 reduce ISE Technology Innovation 2008 | ( group=true ) View http://localhost:5984/technight/_view/answers/count_by_ans wer1?group=true key reduce group reduce map REST 1 REST 1 .. .. 1 1 REST 2 1 .. 1 1 .. 1 REST 1 .. .. 1 80 Array ISE Technology Innovation 2008 | key Array { “language” : “javascript” “views” : “count_by_answers” : { “map”: “function(doc){ ([doc.A1, emit([doc.A1, doc.A2], doc.A2] 1); }”, “reduce” : “function(key, values, rr){ return sum(values); }” } } 81 reduce ISE Technology Innovation 2008 | ( group=true ) View http://localhost:5984/technight/_view/answers/count_by_ans wers?group=true key reduce group [“REST”, “Database”] REST,Database [“REST”, “Programming Model”] 1 REST, Database REST, Database 1 1 1 REST, Database 2 reduce REST, Progra... 1 REST, Progra.. 1 REST, Progra.. 1 reduce 82 reduce ISE Technology Innovation 2008 | ( group=true ) View http://localhost:5984/technight/_view/answers/count_by_ans wers?group=true&group_level=1 key reduce group [“REST”, “Database”] 1( 1 ) REST, Database 1 REST, Database 1 [“REST”, “Programming Model”] [”REST”] REST 1 1 1 REST REST, Progra.. 1 3 reduce 83 : ISE Technology Innovation 2008 | group_level group_level key Array key key=[”REST”] “ZZZZZ”] startkey=[“REST”]&endkey=[“REST”, [“REST”, *] adhoc startkey=[“REST”] REST, Database 1 REST, Database 1 REST 1 1 REST REST, Progra.. endkey=[“REST”, “ZZZZZ”] 1 1 3 reduce 84 : reduce ISE Technology Innovation 2008 | reduce 3 Map document id]) value true Map key ([key, key rereduce reduce 2 (undocumented) 85 ISE Technology Innovation 2008 | map, reduce log(obj) log(“hoge”); couchdb (/usr/local/var/log/couchdb/couchdb.log) [Tue, 29 Jul 2008 10:23:29 GMT] [info] [<0.336.0>] Query Server Log Message: ”hoge” Futon curl 86 MapReduce ISE Technology Innovation 2008 | CouchDB Map View – MapReduce (Key,Value) View Query (reduce reduce ) View 87 ISE Technology Innovation 2008 | 88 : ISE Technology Innovation 2008 | Apache CouchDB “ ” Apache CouchDB HTTP( ) MapReduce( ) SW CouchDB RDDB(Ruby ... ), SimpleDB(Amazon, SaaS) 89 ISE Technology Innovation 2008 | CouchDB http://incubator.apache.org/couchdb/ CouchDB wiki http://wiki.apache.org/couchdb/ CouchDB / http://wiki.apache.org/couchdb/InTheWild http://damienkatz.net/ http://jan.prima.de/plok/ http://www.cmlenz.net/ 90 ISE Technology Innovation 2008 | ご清聴ありがとうございました *** Next Innovation with Our Technical Vitality *** 91