Discussion Forum

Building Semantic Search Engine Using Grakn

Hi,

If one wants to build a search engine and exploit semantic information, how to build one using Grakn? How do you compare the Grakn to ElasticSearch/Solr? What should be the relevancy scoring?

Context:
I have built a search/recommendation engine using ElasticSearch. One of the challenging aspects is to encode business logic in the search query. Suppose in a hypothetical case of course recommendation based on user’s current knowledge and aspiration,
Example Query: “Create web application”
Example course documents:

  1. “Building reactive websites using Play framework”
  2. “Create an automated test case to stress test web application”

Although a vanilla ElasticSearch would be able to achieve this but with a proper document enrichment, we will be able to fine tune the relevancy. But one advantage of the VSM is that it will be still able to retrieve some document although they may be distantly relevant(As a fallback mechanism).

HI @rakshithbegane,

Sorry it took me a whole day to reply to your question from Slack.

The short answer: ElasticSearch and Solr will provide you with intelligent string search based on properties of an entity, but does not provide the ability to search across relationships of an entity.

For example in your sample query: search for “create web application”, ElasticSearch/Solr will return documents where the strings contained in these documents have a high hit rate to the string that you gave.

However, ElasticSearch/Solr could not return to you queries, such as: search for “action movies filmed in south africa in the 2000s”. Because in that query, you are searching for 1 type, movie, where based on 3 types of criteria: action genre, year between 2000-2009, and location in South Africa. Grakn can do this query.

There’s the key distinction. Graql (Grakn’s language) allows you to query for many relationships between entities, to filter your result, while ElasticSearch/Solr only provides intelligent string matching. And in the [near] future, we will actually be integrating ElasticSearch/Solr into our string-matching for every resource under-the-hood, anyways. So basically, you should be able to do all the intelligent string matching that you can do in ElasticSearch/Solr, plus you can query for relationships between entities to contextualise your result.

I hope that explains the relevance scoring, as well. Which basically means relevance will be contextualised based on relationships to other entity.

As to “how does one build a semantic search engine using Grakn?”. Well, first of all, we’ve built ont: Moogi.co, so you should check it out. :slight_smile: and test action movies filmed in south africa in the 2000s.
In terms of high-level steps to achieve it:

  1. Gather your data sources which you want to consolidate into one knowledge base to search over.
  2. Gather the schema of every data source
  3. Compile all schemas of every data source into one Grakn ontology, and load it into a Grakn database.
  4. Write migration scripts (using Graql migration language) to migrate all datasource into the Grakn database.
  5. Done.

Of course the 5 steps above is significantly simplified set of steps, as you can image. :slight_smile: But that’s generally the approach.

Let me know if my answer helps! :slight_smile:

I’m a bit confused: This does roughly the same query you did on Moogi.co, but on Elasticsearch. So, IMHO, we can do such queries in ES and even much faster.
What’s difficult, instead, on ES is a query like: who are the actors “with whom Bruce and Tom have worked together?”