slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
F# and NoSQL Introducing FoSQL PowerPoint Presentation
Download Presentation
F# and NoSQL Introducing FoSQL

Loading in 2 Seconds...

play fullscreen
1 / 51

F# and NoSQL Introducing FoSQL - PowerPoint PPT Presentation


  • 150 Views
  • Uploaded on

John C. Zablocki Development Manager, HealthcareSource Organizer, Beantown ALT.NET New England F# User Group 2011-09-12. F# and NoSQL Introducing FoSQL. Agenda. NoSQL Overview MongoDB Basic Concepts MongoDB Shell MongoDB C# Driver CouchDB Basic Concepts CouchDB and cURL

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'F# and NoSQL Introducing FoSQL' - werner


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1
John C. Zablocki

Development Manager, HealthcareSource

Organizer, Beantown ALT.NET

New England F# User Group

2011-09-12

F# and NoSQLIntroducing FoSQL

agenda
Agenda
  • NoSQL Overview
  • MongoDBBasic Concepts
  • MongoDB Shell
  • MongoDBC# Driver
  • CouchDB Basic Concepts
  • CouchDB and cURL
  • LoveSeat
  • NoSQLDesign Considerations
  • Questions?
nosql
NoSQL

Not Only SQL

what is nosql
What is NoSQL?
  • Coined in 1998 by Carlos Strozzi to describe a database that did not expose a SQL interface
  • In 2008, Eric Evans reintroduced the term to describe the growing non-RDBMS movement
  • Broadly refers to a set of data stores that do not use SQL or a relational data model
  • Popularized by large web sites such as Google, Facebook and Digg
nosql databases1
NoSQL Databases
  • NoSQL databases come in a variety of flavors
    • XML (myXMLDB, Tamino, Sedna)
    • Wide Column (Cassandra, Hbase, Big Table)
    • Key/Value (Redis, Memcached with BerkleyDB)
    • Object (db4o, JADE)
    • Graph (neo4j, InfoGrid)
    • Document store (CouchDB, MongoDB)
mongodb concepts
MongoDB - Concepts
  • Schema-less documents stored in collections
  • Documents are stored as BSON (Binary JSON)
  • JavaScript used to query and manipulate documents and collections
  • Each document in a collection has a unique BSON ObjectId field named _id
  • Collections belong to a database
installing mongodb on windows
Installing MongoDB on Windows
  • Download the binaries from mongodb.org
  • Extract to Program Files directory (or wherever)
  • Create a directory c:\data\db
  • Run mongod.exe from the command line with the --install switch
  • See http://bit.ly/aed1RW for some gotchas
  • To run the daemon without installing, simply run mongod.exe without arguments
  • Run mongo.exe to verify the daemon is running
mongodb shell
MongoDB - Shell
  • The MongoDB interactive JavaScript shell (mongo.exe) is a command line utility for working with MongoDB servers
  • Allows for CRUD operations on collections
  • May be used for basic administration
  • Creating indexes
  • Cloning databases
  • Also useful as a test-bed while building apps
mongodb shell1
MongoDB - Shell

/* This script file demonstrates the basics of MongoDB from the interactive shell.    It's not intended to be a best-practive example for anything!

*/

/*Connect to a server:port/database (defaults are localhost:27017/test ):*/mongo.exelocalhost:27017/AltNetGroup//Switch database:use CodeCamp//View collections in a database:showcollections//create an index on Name fielddb.Posts.ensureIndex({ Name : 1 });//copy one database to anotherdb.copyDatabase("CodeCamp", "AltNetGroup")

mongodb shell2
MongoDB - Shell

//create a documentvarpost = { Title: "On Installing MongoDB as a Service on Windows" }//insert a document, if the collection doesn't exist it's createddb.Posts.insert(post);//verify that the document was createddb.Posts.find();//write a query to find a post with a valid titlevarquery = { Title: { $ne: null} }//use that query to find the postvarpost = db.Posts.findOne(query);//this line will actually set the content after pressing enterpost.Content = "When installing MongoDB as a service on Windows..."

mongodb shell3
MongoDB - Shell

//update the content to include an author using collection update method

db.Posts.update( { Title : "On Installing MongoDB as a Service on Windows" }, { Author : "John Zablocki" } )

//check that the post was updated

db.Posts.findOne()

//where'd my document go? updates are in place, replacing entire docs!

//need to use the $set operator to update partial documents - start over

//first remove the new document. Notice remove takes a function argument.

//find and findOne also accept functions as arguments

db.Posts.remove(function (e) { returnthis.Author == "John Zablocki" })

//rerun the first statements up to but not including the db.Posts.update(...

db.Posts.update({ Title: "On Installing MongoDB as a Service on Windows" },

{ $set: { Author: "John Zablocki" } })

//verify that the update worked

db.Posts.findOne()

mongodb shell4
MongoDB - Shell

//add two more tags

db.Posts.update(

{ Title: "On Installing MongoDB as a Service on Windows" },

{ $pushAll: { Tags: ["windows", "nosql"] } })

//add another post

db.Posts.insert(

{ Author : "John Zablocki", Title : "On MapReduce in MongoDB",

Tags: ["mongodb", "nosql"]

})

//verify that last insert worked

db.Posts.findOne(function (e) { returnthis.Title.indexOf("MapReduce") != -1; })

mongodb shell5
MongoDB - Shell

//add a "like" counter to the post. The booleanarguments tell //updatenot to insert if the document doesn't exist and to //update all documents, not just one respectively

db.Posts.update({ Author: "John Zablocki" }, { $set: { Likes: 0} }, false, true)

//increment the likes counter for the mapreduce article

db.Posts.update({ Title: /mapreduce/i }, { $inc: { Likes: 1} })

//check that the counter was incremented

db.Posts.findOne({ Title: /mapreduce/i }).Likes

mongodb shell6
MongoDB - Shell

//use MapReduce to get counts of the tagscreate the map and reduce functionsvarmap = function() {if (!this.Tags) { return; } for (varindexinthis.Tags) { emit(this.Tags[index], 1);                 }            };//conceptually, reduce gets called like://reduce("mvc", [1, 1]);//reduce("norm", [1]varreduce = function(key, vals) {varcount = 0;for (varindexinvals) { count += vals[index];                 }returncount;             };/*

mongodb shell7
MongoDB - Shell

run the mapreduce command on the Posts collectionusing the map and reduce functions defined abovestore the results in a collection named Tags*/varresult = db.runCommand(    {mapreduce : "Posts",map : map,reduce : reduce,out : "Tags"   });db.Tags.find()

mongodb shell8
MongoDB - Shell

//first, insert some datadb["UserActions"].insert({ Username : "jzablocki", Action : "Login"})db["UserActions"].insert({ Username : "jzablocki", Action : "Login"})db["UserActions"].insert({ Username : "jzablocki", Action : "Login"})db["UserActions"].insert({ Username : "jzablocki", Action : "PasswordChange"})db["UserActions"].insert({ Username : "mfreedman", Action : "PasswordChange"})db["UserActions"].insert({ Username : "mfreedman", Action : "PasswordChange"})db["UserActions"].insert({ Username : "mfreedman", Action : "Login"})//now run the group bydb.UserActions.group(     { key :  { Username : true, Action : true },cond : null,reduce : function(doc, out) { out.count++; },initial: { count: 0 }});

mongodb c driver
MongoDB C# Driver
  • 10gen developed and supported
  • Consists of two primary components, a BSON serializer and the MongoDB driver
  • Support for typed and untyped collections, MapReduce, and all CRUD operations
  • Currently lacking a LINQ provider
  • Current version (as of 5/5/11) is 1.0.4098.x
mongodb c driver the basics
MongoDB C# Driver – The Basics

//MongoServer manages access to MongoDatabaseletmongoServer=MongoServer.Create("mongodb://localhost:27017")//MongoDatabase used to access MongoCollection instancesletmongoDatabase=mongoServer.GetDatabase("FSUG");//reference the collectionletmongoCollection=mongoDatabase.GetCollection<Artist>(COLLECTION)letdoSetup=//drop collection before running samples    ifmongoCollection.Exists()thenmongoCollection.Drop()

mongodb c driver crud
MongoDB C# Driver - CRUD

//Insert a document into a typed collection let artist ={ Name ="The Decembrists"; Genre ="Folk"; Id =ObjectId.GenerateNewId(); Albums =[]; Tags =[]}mongoCollection.Insert(artist)|> ignore//Updating (replacing) a document in a typed collectionletupdatedArtist={ artist with Name ="The Decemberists"}mongoCollection.Save(updatedArtist)|> ignore//Updating a nested collectionmongoDatabase.GetCollection<Artist>(COLLECTION).Update(Query.EQ("Name",BsonValue.Create("The Decemberists")),Update.PushAll("Albums",BsonArray.Create(["Picaresque";"Hazards of Love";"The Crane Wife"])))|> ignore

mongodb c driver crud1
MongoDB C# Driver - CRUD

//Find all documents in a typed collectionlet artists =mongoDatabase.GetCollection<Artist>(COLLECTION).FindAll()Console.WriteLine("Artist name: "+artists.FirstOrDefault().Name)//Query with a document speclet artist =mongoDatabase.GetCollection<Artist>(COLLECTION) .FindOne(Query.EQ("Name",BsonValue.Create("The Decemberists")))Console.WriteLine("Album count: {0}",artist.Albums.Count())//Count the documents in a collectionlet count =mongoDatabase.GetCollection<Artist>(COLLECTION).Count()Console.WriteLine("Document count: {0}", count)

mongodb c driver queries
MongoDB C# Driver - Queries

let artists =mongoDatabase.GetCollection<Artist>(COLLECTION)//Find items in typed collectionletartistsStartingWithFoo=artists.Find(Query.Matches("Name",BsonRegularExpression.Create(newRegex("foo",RegexOptions.IgnoreCase))))Console.WriteLine("First artist starting with Foo: {0}",artistsStartingWithFoo.First().Name);//Find artists without pulling back nested collectionsletartistsWithDecInTheName=artists.Find(Query.Matches("Name",BsonRegularExpression.Create("Dec"))).SetFields("Name");Console.WriteLine("First artist with dec in name: {0}",artistsWithDecInTheName.First().Name);//Find artists with a given tagletartistsWithIndieTag=artists.Find(Query.In("Tags",BsonArray.Create(["Indie"])))Console.WriteLine("First artist with indie tag: "+artistsWithIndieTag.First().Name);

mongodb c driver mapreduce
MongoDB C# Driver - MapReduce

//Add some tagsmongoDatabase.GetCollection<Artist>(COLLECTION).Update(Query.EQ("Name",BsonValue.Create("The Decemberists")),Update.PushAll("Tags",BsonArray.Create(["Folk rock";"Indie"])))|> ignorelet artist ={ Name ="Foo Fighters";                   Genre ="Hard rock";                   Albums =mutableListAddRange(["The Colour and the Shape";"Wasted Light"]);                   Tags =mutableListAddRange(["Hard Rock";"Grunge"]); Id =ObjectId.Empty}mongoDatabase.GetCollection<Artist>(COLLECTION).Save(artist)|> ignore

mongodb c driver mapreduce1
MongoDB C# Driver - MapReduce

//Create map and reduce functonslet map =@"function() {                    if (!this.Tags ) { return; }                    for (index in this.Tags) { emit(this.Tags[index], 1); }                    }";let reduce =@"function(previous, current) {var count = 0;                        for (index in current) { count += current[index]; }                        return count;                    }";let result =mongoDatabase.GetCollection<Artist>(COLLECTION).MapReduce(BsonJavaScript.Create(map),BsonJavaScript.Create(reduce),MapReduceOptions.SetKeepTemp(true).SetOutput(MapReduceOutput.op_Implicit("Tags")))let collection =mongoDatabase.GetCollection<Tag>(result.CollectionName);Console.WriteLine("Tag count: {0}",collection.Count())

mongodb c driver groupby
MongoDB C# Driver - GroupBy

//add one more artist for good measurelet artists =mongoDatabase.GetCollection<Artist>(COLLECTION)let artist ={ Name ="The Fratellis"; Genre ="Rock"; Id =ObjectId.GenerateNewId(); Albums =mutableListAddRange(["Costello Music"]); Tags =new List<string>()}artists.Insert(artist)|> ignorelet reduce =BsonJavaScript.Create("function(obj, out) { out.count += obj.Albums.length; }")letgroupBy=mongoDatabase.GetCollection<Artist>(COLLECTION).Group(Query.Null,GroupBy.Keys("Name"),newBsonDocument("count", BsonInt32.Create(0)), reduce,null)for item ingroupBydoConsole.WriteLine("{0}: {1} Album(s)",item.GetValue(0),item.GetValue(1));

about couchdb
About CouchDB
  • Open source, Apache supported project
  • Document-oriented database
  • Written in Erlang
  • RESTfulAPI (POST/PUT/GET/DELETE) for managing CouchDB:
    • Servers
    • Databases
    • Documents
    • Replication
couchdb concepts
CouchDB - Concepts
  • Schema-less documents stored as JSON
  • RESTful API for database and document operations (POST/PUT/GET/DELETE)
  • Each document has a unique field named “_id”
  • Each document has a revision field named “_rev” (used for change tracking)
  • Related documents may have common “type” field by convention – vaguely analogous to collections or tables
design documents
Design Documents
  • Design Documents represent application boundaries (users, blogs, posts, etc.)
  • Used to define views, shows, lists and validation functions, attachments, etc.
    • Views allow for efficient querying of documents
    • Show and List functions allow for efficient document and view transformations
    • Validation functions place constraints on document creation
sample design document
Sample Design Document

{ "_id": "_design/artist",

"validate_doc_update" : "function(newDoc, oldDoc) { if (!newDoc.name) { throw({ forbidden : 'Name is required'}); } }",

"shows" :

{

"csv" : "function(doc, req) { return doc._id + ',' + doc.name }"

},

"views":

{

"all" : {

"map" : "function(doc) { emit(null, doc) }"

},

"by_name" : {

"map" : "function(doc) { emit(doc.name, doc) }"

},

"by_name_starts_with" : {

"map" : "function(doc) { var match = doc.name.match(/^.{0,3}/i)[0]; if (match) { emit(match, doc) } }"

},

"by_tag" : {

"map" : "function(doc) { for(i in doc.tags) { emit(doc.tags[i], doc) } }"

},

},

"lists" :

{

"all_csv" : "function(head, row) { while(row = getRow()) { send(row.value._id + ',' + row.value.name + '\\r\\n'); } }"

}

}

installing couchdb on windows
Installing CouchDB on Windows
  • Download an installer from https://github.com/dch/couchdb/downloads
  • Download curl at http://curl.haxx.se/download/curl-7.19.5-win32-ssl-sspi.zip, unzip and set path
  • Run the following from the command linecurl.exe http://127.0.0.1:5984
    • If all is running, response should be {“couchdb” : “Welcome”, “version”, “1.1.0”}
  • Check out http://wiki.apache.org/couchdb/Quirks_on_Windows for some gotchas
curl basics
cURL Basics
  • cURL is an open source, command line utility for transferring data to and from a server
  • cURL supports all common Internet protocols, including SMTP, POP3, FTP, IMAP, GOPHER, HTTP and HTTPS
  • Examples:
    • curl http://www.bing.com
    • curl –F email=test@live.comhttp://www.live.com
    • curl –X GET http://www.bing.com?q=couchdb
curl and couchdb
cURL and CouchDB
  • Check server version
    • curl http://localhost:5984
  • Create database
    • curl –X PUT http://localhost:5984/albums
  • Delete database
    • curl –X Delete http://localhost:5984/cds
  • Get a UUID
    • curl http://localhost:5984/_uuids
  • Create document
    • curl –X POST http://localhost:5984/albums -d “{ \”artist\” : \”The Decembrists\” }” –H “Content-Type: application-json”
  • Get document by ID
    • curl http://localhost:5984/artists/a10a5006d96c9e174d28944994042946
couchdb futon1
CouchDB - Futon
  • Futon is a simple web admin for managing CouchDB instances and is accessible at http://127.0.0.1:5984/_utils/
  • Used for setting server configuration
  • Allows for database administration (create/delete, compact/cleanup, security)
  • Allows for CRUD operations on documents
  • Creating and testing views
  • Creating design documents
couchdb net client libraries
CouchDB .NET Client Libraries
  • SharpCouch – simple CouchDB wrapper and GUI client. Last commit 2008
  • Divan – Nearly API complete. Some LINQ support. Last commit 2010
  • Relax – Built with CQSR consideration. Complex library. Recent commit (May 2011)
loveseat
LoveSeat
  • Document, View, List and Show API complete.
  • Fluent HTTP API for non-implemented API features, such as creating design documents
  • Support for strongly typed documents, using generics and Type convention
  • Last commit August 2011 by jzablocki.
couchdb loveseat the basics
CouchDBLoveSeat – The Basics

let DESIGN_DOC ="artist"let DATABASE ="vtcodecamp"//database names cannot have uppercase charactersletcouchClient=newCouchClient("127.0.0.1",5984,null,null)letcouchDatabase=couchClient.GetDatabase(DATABASE)couchDatabase.SetDefaultDesignDoc(DESIGN_DOC)letdoSetup=ifcouchClient.HasDatabase(DATABASE)thencouchClient.DeleteDatabase(DATABASE)|> ignorecouchClient.CreateDatabase(DATABASE)|> ignore

couchdb loveseat design doc
CouchDBLoveSeat – Design Doc

//Create map and reduce functons for tag countsvar design = string.Format(@"{{ ""_id"": ""_design/artist"",                            ""all"" : {{                                    ""map"" : ""function(doc) {{ emit(null, doc) }}""                                }},                                ""by_name"" : {{                                    ""map"" : ""function(doc) {{ emit(doc.name, doc) }}"" }});varrequest=newCouchRequest("http://127.0.0.1:5984/music/_design/artist”);var response = request.Put().Form()

.ContentType("multipart/formdata")

.Data(JObject.Parse(design)).GetResponse();

couchdb loveseat crud
CouchDBLoveSeat - CRUD

//Create POCO instancelet artist =newArtist(Guid.NewGuid(),"The Decembrists",null,[],[],["Boston";"Boston";"Hartford";"Burlington"])//Inserting a document into a typed collection - GUID Id will be created prior insert in property, not by driverlet result =couchDatabase.CreateDocument(new Document<Artist>(artist))//Updating (replacing) a document in a typed collection//after creating document, document revision id is in result, but POCO not updatedletupdatedArtist=newArtist(artist.Id,"The Decemberists",result.Last.Last.ToString(),artist.Albums,artist.Tags,artist.TourStops)let foo =couchDatabase.SaveDocument(new Document<Artist>(updatedArtist))    DATABASE

design considerations
Design Considerations
  • Your object graph is your data model
  • Don't be afraid to store data redundantly
    • Your graph might be redundant!
  • Not everything has to fit in 1 document
  • Don't be afraid to store aggregate statistics with a document.
object graph as data model
Object Graph as Data Model
  • Generally speaking, most MongoDB drivers will serialize an object graph as a single document
    • The relationships of your classes creates an implied schema!
    • Migrating this schema is not trivial if you are trying to deserialize properties that did not or no longer exist
  • Consider use cases carefully to avoid inefficiently structured documents
  • Projection queries will be your friend
redundant data is ok
Redundant Data is OK
  • Optimize documents for quick reads and writes
  • Your application layer will have to maintain referential integrity!
  • If every time you access a Post document, you need some of an Author document's data, store that data with Post
  • Design simple classes for this redundant data for reusability (see AuthorInfo in Meringue)
don t stuff it all in one document
Don’t Stuff it All In One Document
  • Nothaving formal relationships does not mean throwing away relationships
  • Consider a user and his or her logged actions
    • The user would likely have a User class/doc with properties for name, email, etc.
    • User actions are generally write heavy and read out of band.
    • Don't clutter user documents - create a separate collection for user actions
don t be afraid of extra data
Don't Be Afraid of Extra Data
  • The schema-less nature of documents makes it easy to store meta data about that document – particularly aggregate data
  • Consider a blog post with a rating feature
    • Each rating would be stored as a nested document of the post
    • Rather than compute vote totals and averages real time, simply add these properties to the document and update on writes
final thoughts
Final Thoughts

Eat food. Not too much. Mostly Plants.

- Michael Pollan

final thoughts1
Final Thoughts

Write code. Not too much. Mostly C#.

- John Zablocki

links
Links
  • http://dllHell.net - my blog
  • http://www.CodeVoyeur.com- my code
  • http://www.linkedin.com/in/johnzablocki
  • http://twitter.com/codevoyeur
  • http://mongodb.org - Official MongoDBsite
  • http://couchdb.org- Official CouchDB site
  • http://bitbucket.org/johnzablocki/meringue
  • http://bitbucket.org/johnzablocki/codevoyeur-samples
  • http://about.me/johnzablocki