The Foundations of Great Enterprise eCommerce Search Engine

Providing Relevant Results

It may seem obvious, and even an easy task, that a search engine should provide results relevant to an executed query. This is because we are accustomed to Google, which we use daily, and which is top-notch in relevancy. Google has universally set the standard of what a search function should accomplish. 

But keep in mind that Google uses an extremely complicated, ever-expanding and changing algorithm to be able to do that, backed by hundreds of developers constantly fine-tuning it – and it took them literal decades to arrive at today’s model.

Take a moment to think about what “relevant search results” actually means. For example, let’s say a shopper enters the query “cooking book” on Amazon.

What if there is a typo in the query? The shopper is still going to expect relevant results, and will only get frustrated with a “no results” page and the fact that they have to repeat the search.

Then there is the question of how do you weigh the importance of product attributes – hint: if the keywords appear in a title, it should be more relevant than if it’s only included in tags, but shipping to the shopper’s location is even more important, and so on.

As you can see, relevancy is a much more complicated question, especially if you have hundreds or thousands of products in your online store.

Reliability, Uptime and Speed

On the technical side, you also have to make sure that your search engine actually works: shoppers can run queries anytime, from any part of the world, and they get the results almost instantly. 

The average Google search, again, is a standard that we are used to, and most times it takes only a fraction of a second to complete. Even if you can’t compete with them, you should use an enterprise eCommerce search engine provider and hosting solution that makes it possible for your shoppers to get to the result page as soon as possible.

Must-Have Capabilities for Your Enterprise eCommerce Search Engine

1. Autocomplete

One of the most basic functions that shoppers expect is Autocomplete: this predicts shopper queries once a shopper focuses in the search box, or from the first keystroke. It suggests keywords, products, and categories that yield relevant results.

Predicting shopper intent  is only one aspect: what is even more useful is that it slightly guides shoppers down their path-to-purchase by suggesting searches that won’t end in zero result pages. 

There is not much more to this, except our advice: if a site search engine lacks Autocomplete, stay far away from it.

And autocomplete should also be…

2. Typo-tolerant

There aren’t many studies out there with numbers on the average amount of spelling mistakes, even fewer focusing on search engines because search engine providers don’t usually publish this kind of data. But we can take away some interesting points from a 2006 article by W. John Wilbur, Won Kim, and Natalie Xie titled Spelling correction in the pubmed search engine.

A typo-rate somewhere between 1 in 4 and 3 in 4 might seem extremely high, but with keyboards, fast typing, the prominence of typo-tolerant search engines and spell checking, it certainly can be possible, because we don’t really pay attention to spelling anymore.

If your site search engine can’t tolerate typos – as in, recognize them and recommend another search, or even better, show results for the correctly spelled keyword instead of returning no results – you run the risk of driving your customer away. 

3. Managing synonyms

Another important aspect: synonyms. In many cases, your shoppers are going to refer to products or categories, even certain attributes by a different name than you do. 

For example: if they are looking for a laptop, they may write “notebook” in the search field, luminosity and brightness might get confused with LED lights, someone who wants to buy an instrument panel may search for a control board.

And if your search engine can’t understand this, shoppers are going to end up on zero result pages, even if you do actually sell the product they want to buy.

Which, of course, is bad, because your products effectively remain hidden from shoppers with strong purchase intent, and you lose money on every one of them.

We recommend you choose a solution with built-in Synonym Management. 

You might think it would be easy to find a solution that includes this, maybe it’s just a matter of integrating a synonym database – but actually customizing a search engine this way is very tedious work and it’s not at all fast.

Which brings us to…

4. Customization

Out-of-the-box solutions are convenient, cheap, easy and quick to implement, but rarely satisfy all needs. They aren’t all flexible in terms of customization: usually you can’t expand on the existing functionality or performance.

Meanwhile, open-source search solutions can be customized as you want, but they are also resource-demanding – implementation, development, maintenance can take a lot of time and energy.

Before committing to any kind of site search solution, think about your unique business processes and the needs of your shoppers. 

Can the search provider meet both of these needs?

5. Personalization

Personalizing what your users see can be a great way of not only increasing your conversion rate, but also your average order value.

A few examples how you can personalize the result page based on the past behavior of your shopper:

Show them products they viewed (and didn’t buy) earlier, similar products to those, or products matching their previous searches. This way you may guide them to additional purchases, as they may see results they didn’t stumble upon before.

Overall, personalizing the result pages based on behavior can decrease site abandonment, increase engagement and conversion rates, and improve the user experience and customer journey of your webshop.

6. Dynamic filtering

Instead of providing a static results page when the shopper executes a query, you can improve user experience and increase your conversion rate by leveraging dynamic filtering.

What this means: the search engine displays relevant results and filtering options based on the initial query. The key here is that none of the filtering options will lead shoppers to a 0-results page. 

For a ‘real life’ example, let’s say you have an online book store. When a user enters “sci-fi novel”, they are presented by a number or results, and also a few checkboxes appear where they can filter by the most relevant authors like Isaac Asimov, Frank Herbert, Iain M. Banks etc. But they won’t see J. R. R. Tolikien’s name there, as he never wrote science fiction.

This also means that someone who searches for ‘romance novel’ won’t see the same filter options as those authors didn’t write romances. 

The search engine does this by analyzing the attributes of the results and applying relevant filters, which is why this is called faceted filtering. 

As the shopper starts checking boxes to filter results, the irrelevant filter options disappear. 

This not only guides shoppers to the product they want to buy, but it gives them a sense of control and improves their overall experience on your online store.

7. Natural Language Processing

Natural Language Processing (NLP) is a very complex topic as it encompasses both machine learning and AI, so we are going to concentrate on what it actually does when integrated into a search engine: it enables the search engine to better understand queries by, well, processing natural language.

NLP enables the search function to provide more relevant results, offer suggestions via Autocomplete, and learn from user behaviour based on the wording and language used. 

It also makes your search engine more language-independent, as NLP is not strictly tied to a specific language.

It is a resource-intensive feature to develop, and some solutions simply sell their advanced synonym management as NLP, so keep an eye on the validity of these claims. We can’t add much to the advice Shopify gives you:

When you don’t dig into the specifics [of the NLP feature], you might only be getting a demonstration of basic functionality like synonyms with a fancier label. That’s a problem if you’re expecting an automated system with sophisticated language parsing.

8. Ranking

Make sure that you have the option to customize certain ranking factors. While relevancy is the most important, if you want to be able to provide better customer experience, it might also make sense to highlight popular products on a result page for example. 

Ranking algorithms usually incorporate relevancy and popularity scores. However, search providers have dashboards to let their customers alter the boosting parameters to sell more of what they want.
These ranking algorithms should also be refreshed at least daily to ensure results remain accurate and shoppers get the best experience possible.

9. Analytics

Your on-site search engine is one of the most valuable resources of customer data. You can learn invaluable information about them including: how they search for procuts, what wording and language they use, what they’re searching for that you don’t currently sell/have in stock, if they are successful with their searches or need help, and much more.

If you have all of these, you can refine your customer journey, improve marketing messaging  and to achieve greater conversion rates!

10. Merchandising

Merchandising is a way to effectively promote the products you want to sell. These can be new products, products with a lot of inventory, or products on sale. 

You can customize the placement of these by adding banners, editing the products on the search results page, adding redirect rules, and disambiguation tiles.

11. Indexing

Indexing is the process by which the search engine creates a database with all the available data on your products and provides results based on this database.

With indexing the key is to choose a solution with a frequency that suits your business model. Many times the frequency with which the tool indexes products and attributes depends on how much you pay for the service, the tier you are in etc.

Shoppers on your site expect to find products instantly – this includes new products. Most online revenue is generated by shoppers who use the search function. If your search doesn’t have access to an updated product database, it can’t direct shoppers to the products they want to buy.  

Which means you miss out on money.

In-house search engine development or SaaS vendor solution?

When it comes to implementing site search in your webshop, you have three basic options to choose from:

• You can select an out-of-the-box solution.
• You can decide to develop your own site search engine.
• You can opt for a SaaS solution.

Therefore,  the rest of this article will focus on in-house development and SaaS search solutions: we’ll provide you the pros and cons of both options, so you can make an informed decision yourself.

In-house Search Development with Elasticsearch or Solr

Elasticsearch and Solr are currently the most popular open-source search engines that you can work with. 

The timeline for in-house development

Setting up an open-source search solution (1-4 weeks)

With Elastic or Solr you are getting a basic site search solution, for which the setup will take something between a week and a month. 

However, you should know that the time frame depends on your in-house capacity: the size of the dev team, their experience with search, how much time they have to focus on the project, and , the number of products and requests your online store receives. This means that if you have a small dev team (or one focused on other products) or a lot of products and shoppers, it can take longer.

Advanced Data Processing (2-8 weeks)

Your search engine must have all the data about your products – all the attributes, values should be standardized and processed, together with sentences from product descriptions, relevant keywords have to be extracted. 

This will provide the base for your engine to work, and continuously updated in the future. The time it will take to do it the frist time will depend on the expertise of your developer(s), but usually it will be around 2-8 weeks.

Configuring the Basic Features (4-8 weeks)

Synonyms, default facets, tokenizers, default stemmers and the rest of the core features get configured in this phase that will take somewhere between 1-2 months.

Configuring the Advanced Features (20 weeks)

NLP-based suggestions and autocomplete, custom filters, contextual dynamic facets and some of the more complex features: these can take up to 20 weeks to configure.

Custom Ranking (1-4 weeks)

As we have mentioned, customizing the result pages to fit your business model and marketing efforts is an important function. 

Determining and setting the basic factors for this should take 1 to 4 weeks, depending on complexity. 

(And a few days each month for updating these values.)

Custom Search Functionality (3 weeks, at least)

Depending on what kind of extra functions you and your online store require, it will take at least 3 weeks to set these up. 

Think: search assistants, voice search, personalization of the results pages etc. 

The more you want to customize, the longer it will take – this could possibly be extended by many months.

Post-Live Launch Bug Fixes and Algorithm Improvements (3-4 months)

This is the fine-tuning phase after everything is set up – which is, of course, something that will go on as long as your site is live. The first testing, bug fixing and improving phase should last at least 3-4 months after launching your search engine.

Advantages of In-House Search Engine Development

Customization 

Developing your on-site search engine with your own team means that you have complete control over the process and what to include, how to build it. This allows you to tailor everything to your business needs.

Flexibility in Terms of Making Priority Bug Fixes

If you have an in-house team, priorities can be set as you want: they can get to fixing critical issues right away and you don’t have to wait on a third party, on customer service and so on.

Results Can Be Leveraged Beyond Search

Your custom-built search engine can be useful in developing other important site functions which your developers can create based on this work.

Disadvantages of In-House Search Engine Development

Team Recruitment 

If core members leave your team, crucial knowledge can be lost with them, even if all the documentation and written know-how remains. This cannot replace their experience and the fact that they have been working together as a team, and new team members always have to be onboarded, introduced to customs and policies ang it generally could take quite a long time for them to get up to speed.

Team Management 

Your search team should be an independent one from your core dev team – otherwise multiple projects running parallel will take time and capacity from each other – consequently both basic development and building/maintaining your search engine will have less progress made in the same time.

It Takes Years to Fully Develop an Enterprise eCommerce Search

No matter what you do, basic development itself will likely take 1-1.5 years – and after that, adding extra functionality, customization, bug fixes, maintenance and updates will take years until your engine is fully formed. It’s essential that your team always stays up-to-date with e-commerce search best practices and continuously implements them.

Manual Infrastructure Setup and Maintenance

With no third-party partner or SaaS provider, you have to purchase, set up and maintain all the infrastructure needed for both development (tools and devices the team will use) and operating the search engine. In terms of resources, this means additional costs and time.

Continuous Performance-improvement, Monitoring and Optimization

You need to continually develop new features – the needs are going to be changing based on customer requests and competition, changing habits, alterations in your business model and a lot of other factors.

Lack of Up-to-Date Technology

Search engine development is a very specialized field, and most of the times you will not have relevant, fresh technology at your disposal to use – you have to do everything yourself, which will usually take months to accomplish.

No Support

With an in-house team, you can’t rely on outside expertise: if you hit problems, you will have to figure it out yourself or find the solution on dev forums – both of which can be time-consuming and frustrating.

SaaS Vendor Enterprise eCommerce Search Provider

If you don’t want to hire and maintain a full development team, and still want a high-quality,flexible solution, your best option is to use a SaaS provider.

But you are going to have to be careful about which one you choose (see the comprehensive list above for the most important factors to look at when choosing a provider)-, because search-as-a-service is a highly specialized field where every vendor, every solution has their specific strengths and specializations.

Quick integration

When using a search provider, some offer integration within weeks, which means you’re up and running in no time.

Some SaaS site search providers have integration times that take months, though. 

So if you’re looking to get started quickly, check the timeline of each individual provider, along with their integration process, to see which best fits your needs. 

Proven ROI before commitment

You not only get better search in a shorter amount of time, but also get the opportunity to commit to a service that has history: case studies, testimonials to back up its effectiveness. The core features are going to be constantly updated based on data not only from your webshop, but from all the shops and clients the vendor has, meaning best practices could be implemented with a much greater effect.

Detailed Analytics

You are going to get a dashboard and likely options to customize the analytics and reports you get, meaning you will be able to implement any useful data in your sitewide marketing efforts and business model from day one.

Dedicated Customer Support

A team of experts will be at your disposal as a part of the service, and you don’t have to pay their salaries or for the tools and devices they use. You can also be sure that they are experienced in search technology and will likely have the answers you are looking for.

Fully managed search

You and your team will not be the ones who have to make updates, develop features, manage scaling or hosting: you can reach out to the SaaS vendor and simply tell them your needs.

Freedom in Scaling Along With Your Business

SaaS solutions are traditionally scalable: as your online store and business grows, you can simply upgrade the service to fit your growing needs, from handling a larger amount of products to including more specialized, custom functions and options in the engine.

Predictable Cost

The expenses will be easy to integrate into your cash flow, and you are going to know exactly how much you are going to pay each month for the service – and if you need custom solutions, the price for developing and integrating them will also be set.

Disadvantages of Using a SaaS Vendor

No Real Control Over Development

As just one of the clients, you will have some say in what features have to be developed, what bugs are going to be fixed – but that say will be minimal. Their development team is going to focus on what the majority of clients and the market are demanding – which might not necessarily align with your personal needs

Operational Features to Consider

Free Trial

Before committing to any solution, you should be able to try it out beforehand. Always look for a free trial or A/B testing opportunity with an enterprise eCommerce search provider, because you are about to make a long-term decision. Also, make sure that your analytics is set up correctly, so you can track any changes in user behavior, conversion rates, exit rates, and all other important KPIs after launching the free trial.

On-going Optimization Support 

Don’t count on a moment happening when you can kick back and say, “well, we are done with site search”. As one of the most important elements on the site, intimately tied to user behavior and revenue generation, continuous optimization, fine-tuning, A/B testing and bug fixing will be a part of your routine. So make sure the solution you chose has the team to support these efforts and are available to help with whatever you need (in a quick manner!).

Analytics Dashboard 

A monthly PDF in your inbox will not be enough when it comes to site search analytics. Make it a point to have access to a live dashboard where you can not only see the most important data and changes of KPIs.

Data Centers, Uptime

Uptime is a critical factor: always check the location of your provider’s data centers along with their SLA to find out their policy on uptime and redundancy. 

The best option is to work with a provider who has their data center in the same time zone as your business, as this will help with potential latency issues.

What is their SLA?

Service Level Agreement 

Many important factors can be found in an SLA, for example, the availability they vow to provide. 99% is the standard benchmark, but keep in mind that this is not all that good a percentage – if your site search works 99% of the time, it means that it is offline 7+ hours a month.

Monthly Cost

Monthly invoicing instead of a year-long commitment gives you both security and flexibility: you can expect fixed costs while also having the freedom to opt-out of the service if necessary.

Occasionally, yearly invoicing comes with a discount – so be sure to check that out.

The Decision is Yours

Choosing an enterprise eCommerce search provider solution should not be a fast decision. There are a lot of factors to consider to give shoppers the best possible search experience – and also support any ongoing marketing efforts and your business model, to achieve the highest possible ROI.

If you have doubts, start with the comprehensive list above, and if you still have questions, don’t hesitate to contact us, we are always happy to help with your decision.

Do your research before making a long-term commitment, and keep in mind that what you go with will fundamentally decide the customer journey and shopping experience in your store!