byPhil WhelanonJanuary 31, 2011
Quora has taken the tech and entrepreneurial world by storm, providing a system that works so fluidly that it is sometimes hard to see what the big fuss is all about. This slick tool is powered, not only by an intelligent crowd of askers and answerers, but by a well-crafted backend created by co-founders who honed their skills at Facebook.
It is not surprising that, with all the smart people using this smart tool, there are many pondering on how it works so well. The NoSQL boffins scratch their heads and ponder such questions as, “Why does Quora use MySQL as the data store rather than NoSQLs such as Cassandra, MongoDB, CouchDB, etc?“.
In this blog post I will delve into the snippets of information available on Quora and look at Quora from a technical perspective. What technical decisions have they made? What does their architecture look like? What languages and frameworks do they use? How do they make that search bar respond so quickly?
What’s Cooking Under That Hood?
Charlie Cheever Follows “14 Rules for Faster-Loading Web Sites”
Components Of Quora
The general components that make up Quora are…
You can askquestions
You cananswerquestions (anonymously if you desire)
You cancommenton answered questions
You canvote-up or vote-down answers to questions
Questions can be assigned totopics
You can write apost(a informative statement, rather like a orphaned answer or blog post)
You canfollowquestions, topics or other users
A super-fast auto-completesearch-box at the top, which doubles as the method for entering new questions
The last point, the super-fast auto-complete search-box, is one of the defining features of Quora. You can see immediately, as you begin to enter a question, whether somebody else has already asked that question or if there is a topic or post on the subject. Let’s start there…
What’s Cooking Under That Hood?
The Search-Box
Only the questions, topic labels, user names or post titles are indexed and served up to the search-box. There is no full-text search, so searching the content of questions and answers will not work. The text that is indexed is tokenized so that words in a different order will still be matched. Prefix matching enables best matches to be shown before the entire word is entered. For instance, typing “mi” might immediately show “Microsoft” in the results.
There is some simple stemming of words, since “nears” matches “near”, but “pony” does not match “ponies”. “Topic-aliases” allow for similar matches on topic names, such as “startup” and “start-up”. These topic-aliases have been manually entered by users. Otherwise these would not match.
If a duplicate question is redirected to another question (a feature of Quora), then that original question will still appear in the search results, since it increases the chances of a match. There is non-gramindexing, so slight mis-spellings will not match. For instance, “gooogle” (with an extra “o”) finds nothing.
Previously, they did use an open source search server, calledSphinx. It supports the features they are using above, but they have since moved from thisdue to real-time constraints. Their new solution is built in-house and allows them better prefix indexing and control over the matching algorithms. They built this in Python.
What libraries does Quora use for search?
Adam D’Angelo, Quora Founder (Nov 13, 2010)
Our search is custom-written. It doesn’t use any libraries aside from Thrift, and Python’s unicode library, which we use for unicode normalization.
Speedy Queries
Did I mention that the search-box is fast? I did some tests and found the responses to be impressive. Queries are sent over AJAX as a GET request. Responses come back as JSON with the rendered HTML embedded inside the JSON. Rendering of the results on the server-side, as opposed to rendering them in JavaScript, seems to be due to the need to highlight matching words in the text. This is sometimes too complex for JavaScript. For instance, typing “categories” might highlight the world “category” in the result text.
I was seeing responses of roughly 50 milliseconds per query from myLinodemachine. Quora does not short-change you when sending requests. From within the browser, I found typing “Microsoft” (9 characters) would result in nine requests to the Quora search server, no matter how fast you type. As you willsee later, the server is in control, so if it did become over-loaded, then it could update the results less frequently without changing the JavaScript.
Quora uses persistent connections. A HTTP connection is established with the server when you start typing the search query. This connection is kept open and further requests are made on this same open connection. The connection will terminate (times-out) if not used for 60 seconds. If a connection times-out then a new connection is established when typing begins.
To simulate the typing of a word into the search-box, I sent the following requests, character-by-character, across a persistent connection. For instance “butler” is six requests (“b”, “bu”, “but” … “butler”).
"butler" (6 chars) duration: 0.393 secs 0.065 secs per query
"butler monkeys" (14 chars) duration: 0.672 secs 0.048 secs per query
"fasdisajfosdffsa" (16 chars) duration: 0.746 secs 0.046 secs per query
That last query was used to test if there was a slow-down for a word that would obviously not be in a caching layer. I saw no slow-down. This means that they are not caching, caching is only used to take the load off the backend search engine or they are doing something smarter (e.g. if there is no match for “fasd” then there will be no match for “fasdi”, so abort).
Is Quora going to implement full-text search?
Adam D’Angelo, I made a lot of the early Quora … (Sep 1, 2010)
Yes, eventually. We haven’t implemented this yet because we’ve prioritized other things, but we will definitely do it in the future.
Webnode2 And LiveNode
Webnode2 and LiveNode are some of Quora’s internal systems, which were built for managing the content. Webnode2 generates HTML, CSS and JavaScript and is tightly coupled with LiveNode, which is responsible for managing the display of the content on the webpage. Charlie Cheever says that he were to start a similar project without LiveNode, then thefirst thing he would do is rebuild it.
They seem very pleased with the technology they have built andstruggled to find its weaknesses. One weakness is that it is tricky for LiveNode to keep track of what is happening within the browser as it pushes changes from the server. If users A and B are viewing the same question then ones interactions will affect the other. For instance, if user A up-votes an answer then that answer will be promoted and will visibly move up the page. This display change will be pushed over AJAX to user B’s browser. Any prior browser-side change that user B made, such as expanding a comments section, might be lost.
LiveNode is written inPython, C++, and JavaScript.jQueryandCythonis also used.
While theywould like to open-source LiveNodeand have tried to keep code separation, doing so right now would be too much work and would take time away from their main goal, which is making Quora better.
Charlie Cheever points out that webnode2 isunrelated to the “free and easy website builder” called Webnode at webnode.com.
Amazon Web Services
Amazon EC2 and S3 is used for their hosting. While this is not as cost-effective in the long-term as running your own servers, it is perfectly designed for fast growing companies like Quora.
Ubuntu Linux
Quora uses Ubuntu Linux as its OS of choice. No major surprises there. It is easy to deploy and manage on Amazon EC2. Adam D’Angelopoints outthat he used Debian Linux at high school and college and stuck with it because“it works and there hasn’t been a compelling reason to switch”.
Static Content
You only need to look at the source HTML of any Quora webpage to see that they are using Amazon’s distributed content delivery network,Cloudfront. URLs are in the form…
http://d2o7bfz2il9cb7.cloudfront.net/main-thumb-670336-25-7kmigSSkkdusoE6gHRkdQsXfjuTCaxQs.jpeg
CloudFront is used for all static images, CSS and JavaScript (except for Google’s Analytics JavaScript, which is hosted by Google).Images are uploaded to the EC2 machine, then resized and uploaded to S3. This is managed using thePython S3 API.
HAProxy Load-Balancing
Quora usesHAProxyat the front-line, which load-balances onto the distributed Nginx servers behind them.
Nginx
Behind the load-balancer,Nginxis used as a reverse-proxy server onto the web-servers.
To understand more about this setup I recommend reading “Using Nginx As Reverse-Proxy Server On High-Loaded Sites“.
Pylons And Paste
Pylons, a lightweight web framework, is used as their main web-server behind Nginx. They use the defaultPylon + Paste stack.
Pylonswas selected much like you would select a pumpkin for Haloween. They scooped out the insides, such as templates and the ORM, and replaced it with their own technology, written in Python. This is whereLiveNode and webnode2 reside.
MochiMediawas also one of the inspirations for using Pylons, since they were using it themselves.
Python
Coming from Facebook, it was a good bet that Charlie and Adam would choose PHP for their development language. As Adam points out, “Facebook is stuck on that for legacy reasons, not because it is the best choice right now“. From this experience they knew that choosing technologies, especially programming languages, for the long-run was very important. They also looked at C#, Java, and Scala.Discounting Mono, C# would be a choice of more than just the language. It would require them to build on-top of a Microsoft stack. Python won over Java because it is more expressive and quicker to write code than Java. Scala was too new. Adam mentionsspeed and the lack of type-checkingas drawbacks with Python, but they both already knew the language reasonably well. Where Python lacks speed for performance critical backend components, they opt to write them in C++. They saw Ruby as a close match to Python, but their experience with Python and lack of experience in Ruby, made Python the winner. Python2.6, to be precise.
Additional benefits for using Python are the fact that data-structures that map well to JSON, code readability, there is a large collection of libraries and the availability of good debuggers and reloaders. Browser-server communication using JSON is major component of what Quora does, so this was an important factor.
No IDEsare used for development as most use theEmacs text editor. Obviously this is personal choice, and would change as the team grows.
PyPy, a project that aims to produce a flexible and fast Python implementation, was also mentioned as something that might give them a speed-boost.
Thrift
Thriftis used for communications between backend systems. The Thrift service is written in C++.
Why would you write a Thrift service in C++?
Adam D’Angelo, I’ve written a lot of Python, in… (Sep 4, 2010)
Mainly if you want to keep data in memory between requests, and want to keep your Python code stateless. Writing a Python wrapper around a C library involves some memory management with reference counting that requires some understanding of the Python internals, but writing a thrift interface is simple. You also isolate failures this way – if the service goes down it won’t take the Python code down with it.
Tornado
TheTornadoweb framework is used for live updating. This is their Comet server, which handles the large volumes of open connections used for long-polling and pushes updates to the browsers.
Long Polling (Comet)
Quora does not display just static web pages. Each page will update will new content as questions, answers and comments are submitted by other others. As Adam D’Angelo points out, one of the best ways to do this currently is with “long polling”. This is different to “polling”. Withpollingthe browser will repeatedly send requests to the server saying “Any updates?” and the server will respond with “No”. A few seconds later it will ask again, “How about now?”. “No”. “How about now?”, “No, already!”. This puts the client (web-browser) in the driver’s seat. This is backwards because the client does not how long to wait before asking again. If the client asks the server too frequently then it will unduly overload the server. If it pings the server too infrequently, then server will be sitting on updates while it waits for the client to request them and the end-user will not see updates immediately.
Long polling, also known asComet, puts the server in control, by making the client wait for responses. The conversation between the client and server is the same, but instead of the client waiting before making another request, the server waits before it makes the response. The server can keep the connection open for a long period of time (e.g. 60 seconds) while it waits to see if any updates come in. When updates do come in it can respond immediately to the client. On receiving the update, the client then immediately sends a new request for more updates. The server, once again, delays responding until it knows something worth telling the client or enough time has past that it would be rude not to respond.
The benefit to long-polling is that there is less back-and-forth between the client and server. The server is in control of the timing, so updates to the browser can be made within milliseconds. This makes it ideal for chat applications or applications that want really snappy updates for their users.
The down-side is that you are going have lots of open connections between the clients and your servers. If you have a million users (Quora will soon) and, if only 10% of them are online on your site, you will need an architecture that can hold open at least 100,000 concurrent connections. This assumes they only have one tab open to your site. Right now I have 7 tabs open for quora.com in my browser. Each tab usually has multiple connections open to quora.com. In short, Quora must maintaina lotof open connections.
The good news is that there are technologies specifically designed for this. It costs very little to hold open connections in memory if you free up all the resources used for that connection. For instance, Nginx (Quora uses this for proxying requests) is a single-threaded event-based application and uses very little memory for each connection. Each Nginx process is actively dealing with only one connection at a time. This means it can scale to tens of thousands of concurrent connections.
Adam D’Angelo, Quora (Sep 29, 2010)
There is no reliable way to do this without having the client polling the server. However, you can make the server stall its responses (50 seconds is a safe bet) and then complete them when a message is ready for the client. This is called “long polling” and it’s how Quora, Gmail, Meebo, etc all handle the problem.
If you have a specialized server that uses epoll or kqueue, you should be able to hold on the order of 100k users per server (depending on how many messages are going). This is called the “c10k” problem. http://www.kegel.com/c10k.html
MySQL
Just like Facebook, where co-founder Adam D’Angelo previously worked, Quora heavily uses MySQL. In answer to the Quora question “When Adam D’Angelo says “partition your data at the application level”, what exactly does he mean?“, D’Angelo goes into the details of how to use MySQL (or relational-databases generally) as a distributed data-store.
The basic advice is to only partition data if necessary, keep data on one machine if possible and use a hash of the primary key to partition larger datasets across multiple databases. Joins must be avoided. He sites FriendFeed’s architecture as a good example of this. FriendFeed’s architecture is described by Bret Taylor in his post “How FriendFeed uses MySQL to store schema-less data“. D’Angelo alsostatesthat you should not use a NoSQL database for a social site until you have millions of users.
It is not only Quora and FriendFeed who are heavily using MySQL. Ever heard of “Google”? It is hard to imagine, since everything Google does has to scale so well, but inthe words of Google, “Google uses MySQL [...] in some of the applications that we build that are not search related”. Google has releasedpatchesfor MySQL related to replication, syncing, monitoring and faster master promotion.
Adam D’Angelo, Quora (Oct 10, 2010)
One option is to simulate some load. Write a script that mimics the kinds of queries your application will be doing, and make sure it can handle the amount of load you want it to be ready for (especially as the size of the dataset changes).
Memcached
Memcachedis used as a caching layer in front of MySQL.
Git
Gitisused for version control.
JavaScript Placement
If you look at Quora’s source you will see that the JavaScript comes at the end of the page. Charlie Cheeversuggeststhat this give the feeling of a quicker loading page, since the browser has content to display before the JavaScript has be seen.
Charlie Cheever Follows “14 Rules for Faster-Loading Web Sites”
Steve Souders, author of High Performance Web Sites and Even Faster Web Sites, lists the followingrules for making websites faster. This list is mentioned by Charlie Cheever, the co-founder of Quora, as one of the reasons for Quora’s speed.
“One resource we used as a guide is Steve Souders’ list of rules for high performance websites:http://stevesouders.com/hpws/rules.php”
Steve Souders’ 14 rules are…
Make Fewer HTTP Requests
Use a Content Delivery Network
Add an Expires Header
Gzip Components
Put Stylesheets at the Top
Put Scripts at the Bottom
Avoid CSS Expressions
Make JavaScript and CSS External
Reduce DNS Lookups
Minify JavaScript
Avoid Redirects
Remove Duplicate Scripts
Configure ETags
Make AJAX Cacheable
Conclusion
Quora is a great example of a modern tech start-up. They are very small team who understand the technologies they are using very well. They have made considered choices in the technology they have selected and have a good vision of which components would be better written from scratch. They seem keen to share these in-house technologies with the open-source community and I look forward to when they have the time to make this a reality.
I intend to keep following Quora and writing about themmore in future blog posts.
If you found this post useful then pleaseleave a commentorfollow this blog.
Resources
