We don't claim omniscience and part of this nextproposal is seeking out the wisdom of you our readers to help us make the best decisions. What we've listed below is a preliminary set of technologies / services / etc. we believe could be useful in our endeavors to create a better search engine. We'd love to hear your thoughts - feel free to make pull requests or shoot us an email at [email protected].
One of the historical challenges to starting and maintaining a search engine is the creation of an appropriate and updated index against which queries are run. We believe that Common Crawl will provide a great seeding source for nextsearch. It will not be the extent of our index but when our index does not have other, better results available Common Crawl may be a robust resource.
There are several robust search engine options currently available in the open source world. Among those we have found most attractive:
While we'll be using sources such as Common Crawl to seed our index we also expect to do indexing of our own. For this purpose, Apache Nutch seems a wise choice.
We think Discourse has created a robust gamification / authority system which is worth analyzing and adapting.
We believe that users should own their data and we hope that this data will be in a open and common format that can be easily ported by users between different search engines.
One step in this direction is the implementation of user owned storage. We are particularly interested in:
- Blockstack with its Gaia storage system.
- remoteStorage which already integrates with a number of popular data storage providers (including Google Drive).
- Solid which offers PODs and comes from none other than Tim Berners-Lee.
We are particularly interested in using alternative monetization methods such as:
- Brave / Basic Attention Token (BAT) which was founded by Brendan Eich, creator of JavaScript.
- Coil which is partnering with Mozilla and CreativeCommons to provide $100 million in funding to make a better web.
Heck, we figure if we mention cryptocurrency enough times its bound to attract attention. But seriously, we do think cryptocurrencies could play a role in our solution. One project we've followed for a long time is Stellar.
Throwing another buzzword in here but for serious reasons. We think that using the blockchain could be an excellent way to authenticate and validate web content and search rankings...more on this later.
As will be seen elsewhere in our proposal, authenticating identity will play an important role in the way the search engine operates.
Keybase looks interesting, not least because Chris Coyne is there and his previous work on OkCupid is quite inspirational.