One of the recurrent complaints about desktop search as compared to the standard set by Google is that finding one's own files is so much slower. If Google can index a couple billion pages and return the results in .03 seconds, why does a search of my hard drive take 30 seconds? We know that we'll put up with the search being 3 orders of magnitude slower because our own information is 3 orders of magnitude more important than the random fluff on the web, but how can we make it work better?
Well, Google's secret is no secret at all: It's RAM. Random Access Memory. Just like you've got in the machine you're using right now. Keep in mind, all the thousands of boxes in Google's racks do all their querying in RAM. Even with network latency and the time it takes to send a request to their data center and with the index encompassing terabytes of data, RAM is just that much faster. The downside, of course, is that keeping data in RAM requires redundancy, since it loses its contents as soon as the power is cut. Indeed, Google's people have said that one of their primary logistical concerns is the massive amounts of power needed for supporting and cooling all those RAM-hungry boxes. But if the couple hundred megabytes of information that you value were all available in RAM, your search times would be even faster than Google's.
So, how can this be done? Well, it's not hard. PCs have had a driver for RAM drives available since the days of DOS 3.0. But improvements in Windows' memory paging system made RAM drives obsolete. Still, Microsoft offers a sample RAM disk driver for Windows 2000, and even includes a version of the driver in XP, though you have to manually install it to activate it. Or you can download a free third-party version of the same driver, complete with installer.
(Side note to XP users, if you try to activate the driver: If you don't have any disks with the FAT file system, the driver won't work, as it uses FAT for the newly-created drive. You'll need to enable the fastfat driver to make use of it.)
I tried a few experiments, moving my browser cache to the RAM drive, or moving my temporary files there, or even moving my entire email archive to RAM, and it turns out to make a massive difference in performance. Nearly everything is instant. Running a copy of Photoshop installed to the virtual drive is astonishing the first few times you start the app. But it's not a solution that really works for any length of time, as various parts of Windows balk at being run from a drive that's not "real". And you don't want to have to reinstall your apps every time you reboot, even if XP only rarely needs to be rebooted.
So what good does this insight into Google and RAM drives do us? Well, it seems like there's an opportunity. Someone with just a little bit of time and ingenuity, who's interested in making tiny apps like Andre does with torrez.net could easily make an installer for a RAM drive that caches important, frequently-accessed data in RAM in a way that preserves its state by writing it to disk upon shutdown. Perhaps with a backup system that mirrors the RAM disk to the hard drive when the system is idle. The fact that RAM is so volatile could probably be compensated for, or even sold as a benefit. ("Your browser history is permanently erased every time your computer shuts down, for the ultimate in security!") And a My Documents folder mirrored to this storage area would be searchable with Googlesque speed.
I'd be willing to beta test it. I work on a laptop, so even if the power cuts out, my battery acts like an uninterruptible power supply, preserving my system's power source. And I'd pay, I dunno, $30 or something for the simple software, along with another couple hundred bucks for enough RAM to copy everything useful to memory. I bet there's a market.
]]>Of course, what happens to general Windows performance when N% of the RAM is taken up by the RAM drive? Does it start to get more sluggish?
RAM manufacturers should get behind this ... it would be a way to soak up supply.
]]>With RAM as cheap as it is these days, I'd add another 512Mb stick into my PC just for this app.
(that'd be a great way to upsell the shareware RAM drive - just throw a DIMM in with say, a $50 price, or get someone like Crucial to bundle your app with their upgrades)
]]>You could fake it, though. Create a RAM drive with whatever RAM Disk driver you prefer. Then use something like Mr. Mirror to ensure that the contents of that drive are constantly mirrored to a space on your physical disk.
At that point, all you need is a simple copy operation during Windows startup to copy everything from your mirror to the RAM drive.
]]>http://www.padring.com/soft/Utilities/DiskTools/RAMDiskNT.html
]]>Another smart move is setting your IE cache to 0 rather than just sticking it in RAM. The instant displaying that results is grand. The drawback, though, is that pages have to refresh when the back button is clicked.
]]>The buffer cache built into any operating system newer than MS-DOS 3.3 is *exactly* "a RAM drive that caches important, frequently-accessed data". If you want a demonstration, just load Photoshop twice; you'll see how the second time it's much faster -- that's because the files have been cached in RAM by the operating system! If using a RAM disk is still faster, then the OS is broken and its VM system needs fixing.
In any case, the reason Google is faster is because it's *indexed*. You can get desktop search applications that work much more quickly than Google does; the cost is that they'll need to "crawl" your data periodically (just like Google!) and spend a bunch of disk space on an index (just like Google!). That's disk space, not RAM. Once your data has been indexed, the queries will execute lickety-split.
Any query engine that takes 30 seconds to return is either monstrously broken, or (much more likely) doesn't have an index; it's scanning all your data sequentially. The e-mail search in Outlook, for example, is not indexed. Other e-mail systems are better about indexing. There are add-ons that do nothing but index your mail; see Zoe Interwingle, for example, which I promise will take less than 30 seconds to process a query. Back in the day, AltaVista used to sell a very fast desktop search add-on.
Maybe more desktop application data should be indexed, maybe there should be some kind of universal search interface that runs over it all -- that's a holy grail we've been chasing for many years now.
But please, please, stop telling people to use a RAM disk, that's just wrong. Operating systems are designed to move data between RAM and disk ("virtual memory" and "buffer cache") adaptively; overriding that is rarely a good idea and always a lot of hassle.
It *is* possible to optimize applications to run out of RAM (that's what the enterprise companies who accelerate databases and such do), but this really isn't the direction to be going in. Much better to optimize the basic OS caching algorithms (if necessary) and *build indexes*.
Whew.
]]>If you make a big RAM disk and dedicate it to your browser cache or Photoshop or whatever, you will find that your browser or Photoshop or whatever may be faster.
However, you've just subtracted a bunch of RAM from the rest of your system, which means that *everything else* will be that much more sluggish. On the whole it may be a lose, but you won't know that by benchmarking Photoshop start-up times.
It may sometimes be useful to second-guess the operating system's cache, but you have to be very careful about what you measure.
]]>That said, maxing out your RAM is a good thing. It might even help you get much better battery life, if your laptop never has to spin up the drive.
Rumor has it the next version of Windows will include a relational database filesystem that should make local search much more pleasant. And DRM features to guarantee that whatever you find isn't usable ;-)
]]>And you're absolutely right, indexing is critical. But applications like Outlook do index messages or files, and Office has long offered an indexing service, as has Windows itself. Yet none of these systems are as fast as a brute-force full-text search on data that's stored in RAM.
I don't doubt that the right way for these systems to evolve is to have smarter engines that index during idle time and cache more info more intelligently in RAM. But if a non-indexed, unintelligent search mechanism is orders of magnitude faster in RAM than a VM-aware, indexed "smart" search engine, why wouldn't we pursue the avenue that's producing better results right now?
]]>A well-constructed btree runs searches that run in O(log(n)) where n
is the number of items in the index. A brute force search will always
run in time O(n). Scanning RAM instead of disk provides a linear
factor speedup. The tree method wins out after only a very small
change in the size of n (specifically, your order of magnitude or so.)
Even in memory, a brute-force search is going to be limited by the
data rate that memory supports. A btree or generic index is going
to beat that simply by virtue of not needing to scan through most
of the corpus.
Sadly, on windows, you are hamstrung by poor implementations
(Microsoft Find Fast or whatever basically sucks) but at least in
Unix-land, there are good solutions. I have over a gigabyte of
archived mail, and using glimpse with an on-disk hash is still much
faster than simply putting everything in RAM and scanning the whole
thing.
Don't believe us? Install a *decent* third party search tool. Microsoft's are all really stupid and have confusing user interfaces that lead you to believe that things are indexed when they're really not. Hell, Zoe is freeware written in Java and it still runs rings around Microsoft's indexing; how's that for pathetic? And there are much better tools than Zoe. (I don't use Windows, so I can't tell you exactly which one is best right now.)
If you find that a RAMdisk helps you, then by all means use one, but understand that this is not a new and exciting principle for system design. System designers already spend lots of time carefully balancing disk and RAM -- at least, the good ones do. A RAMdisk is the clumsiest possible caching solution, and the fact that it helps you says more about Microsoft's technology than anything else.
]]>I guess what I'm advocating, if you separate out the implementation, is caching of information based on its importance, not merely the frequency with which it's accessed. For example, I have archives of old mail which I only touch once a week, or once a month, or sometimes less than that. But when I need a phone number or some quote out of one of those messages, I need it now, and even the best indexing systems that I've seen don't handle that properly yet.
So I want to be able to flag data as being important to cache, not merely frequently accessed and efficiently indexed.
Thanks to both of you for the advice, though. That's what I get for speaking on a subject based on observations instead of in-depth knowledge. Cool stuff!
]]>On OS X, the locate DB is a god-send in the terminal. It works like a champ. And if I leave the Sherlock indexing turned on, find by content in Aqua is super-sweet.
I think you're on Windows, Anil, but if you've got an OS X machine handy, drag those foldrs into the Sherlock window, do a Get Info on the original folder, and choose the drop-down triangle for Index Content. Next time you need to look in that folder (which I sounds like it's static, otherwise you ought to manually upate it regularly), the search by content is super-fast. I do exactly this for some old mail folders.
I believe the same thing exists in Windows, but I usually turn it off, there, too, and so have little experience with it beyond that.
Also, I don't know if you saw this, but it sounds interesting. Also, there are vague rumors that something like the BeOS live metadata file system will be implemented in the next major release of OS X. That kind of setup basically precludes real searching at all, if I understand it correctly.
]]>Not a Win/Mac thing, but Mac users should pay attention to indexing. For example, by default the Finder's preferences are set to use languages such as Danish, French, German, etc. to search files. Pop that down to English only (or, fewer languages, at least) and you'll not only decrease the size of the database, but you'll also increase the speed of your searches.
I also make a habit of re-indexing my PowerMail database from time to time ... It finds mail extremely quickly.
]]>A few other people recommending it: http://www.bigredswitch.com/blog/archives/2002/12/10/000093.html, http://www.amazon.com/exec/obidos/tg/detail/-/1558605703%3Fvi%3Dglance&e=6251
The accompanying software (http://www.mds.rmit.edu.au/mg/) was created to demonstrate the key concepts from the book and was originally written in C, but appears to be unmaintained now. However, there is a Java version (http://mg4j.dsi.unimi.it/) which looks interesting.
]]>Of course, mounted drives that aren't HFS+, or UFS drives (if you're that much of a Unix weenie) don't have the same data available, so the Finder's Find is much slower. That's where locatedb might come in worthwhile.
]]>I need something similar, to search the hundreds of resumes I have stored on my drive.
-robnit
There certainly are good indexing systems that handle that properly. Perhaps try an evaluation copy of dt-search desktop or Enfish or, for something a little more esoteric, Zoot.
With all of these you should see pretty much instant searches after they've indexed the relevant data.
]]>The Directory B-Tree will let you find files with the name "banana" quickly, but they still have to look at the contents of each file if you want to search the content, if it's not content-indexed as well.
]]>