Friday 25 October 2013

Waiting for a search crawl in Office 365 – plan search-driven sites carefully

HourglassHere in autumn/fall 2013, if you’re working with Office 365 you might notice that content changes (such as new pages and documents) take some time to appear in search results. I spent a little time thinking about this recently, as my team and I finished building a search-driven news site. On this project, we are mainly developing against Office 365 – we use local virtual machines also, but since O365 is the target we are deploying our customisations there frequently as we develop.

We noticed that “index latency” – the time taken for new content to appear in the search index – was poorer than we expected on Office 365. We run several tenancies on different subscription levels (e.g. SharePoint P2, Office 365 E3 etc.), and we experience the problem across all of them. Some days are good, some days are bad. One memorable (read, stressful) time, we had a “end of sprint demo” - our solution was provisioned 2 days before the demo, giving us lots of time to create test content in order to make the demo to the business users go well. We completed adding our pages, documents, pictures and videos a full 24 hours before the demo, and waited for our home page to “light up” as content was crawled in Office 365.

Unfortunately, only some of the content was indexed in time. The demo itself went well, but perhaps only because a bit of narrative helped the business users imagine the ‘full’ picture. Overall, it’s difficult not to feel that 24 hours is a long time to wait for content to be indexed in SharePoint! Business users these days have higher expectations, and most on-premise environments I’ve worked with have used incremental crawls with a frequency of 15 or 30 minutes.

How long is normal in Office 365?

The poor performance surprised us somewhat. My colleagues and I thought that we had originally read that a delay of up to 15 minutes was expected in Office 365, perhaps suggesting that SharePoint 2013’s “Continuous Crawl” is used. The Office 365 Service Descriptions – Search page now suggests that isn’t the case, but however it is managed in the back-end, we certainly weren’t expecting such long delays. Some further digging will lead you to this KB article:

Search doesn't return all results in SharePoint Online – KB2008449

“Search crawls occur continuously to make sure that content changes are available through search results as soon as possible. Recently uploaded documents may not immediately be displayed in search results because of the time that's required to process them. SharePoint Online targets between 15 minutes and an hour for the time between upload and availability in search results (also known as index freshness). In cases of heavy environment use, this time can increase to six hours.”

OK, so at least that’s something official, even if it’s not necessarily what we wanted to hear. But why are we sometimes seeing longer delays than 6 hours even? I raised a Service Request with Microsoft to find out..

The support line

In short, I didn’t get a 100% satisfactory answer from Office 365 support. Ultimately it sounds like this kind of thing is fairly normal in Office 365 right now. I asked if other customers were reporting this issue, and the answer was “yes, but we just ask them to wait another day”. Hmm, OK then! Of course, if your site deals with time-sensitive content (or you are just looking for fresh content to be shown in search in a reasonable timeframe) this isn’t a great situation.

Working around the issue

So if you need to consider other alternatives:

  • If you are dealing with search-driven functionality, could the same thing be provided with query rather than search (e.g. if you do not need to aggregate across site collections)?
  • If you are in a hybrid situation, could the functionality be delivered by an on-premises environment?
  • Do you need a solution right now, or can you afford to wait for improvements? (I personally am hopeful that upgrades to Office 365 will improve the situation in the future.)

For us, in fact all three are options we could use. In our situation the 2nd option could be the simplest if we need an immediate solution - everything we are building for this client can work be deployed to Office 365 or on-premises SharePoint. This requires quite a lot of careful engineering (not only in terms of the solution, but also deployment scripts/processes etc.), but results in a nice position to be in for a hybrid deployment.

In general though, let’s hope that Microsoft work on this in Office 365. I’ll keep you posted if we see improvements - and if anyone has any useful information in this area, feel free to share in the comments below.

Sunday 20 October 2013

Speaking at SharePoint Saturday UK 2013 – developing for Office 365

SharePoint Saturday UK

As usual, I’ll be speaking at this year’s SharePoint Saturday UK event – I think this is the 4th one, and also my 4th time speaking there. It’s a great event, and I’m looking forward to learning from the other sessions. For my part, I’ll be doing my best to share knowledge gained from recent Office 365 and SharePoint 2013 projects.

My session is mainly targeted towards developers and technical people, but hopefully there’s something in it for anyone looking at Office 365. Myself and some colleagues have spent an intense few weeks/months delivering a cloud project recently, and it feels like every day has been a learning day. My talk hopefully conveys some of this, and I’ll be evolving it for future conferences:

Developer decisions, tips and tricks - lessons learnt from Office 365 projects

As a developer or technical lead, your early Office 365 projects are likely to throw up some interesting questions. Should you avoid the sandbox? How should test environments be handled in the cloud? How should a site template be implemented? And just how do you provision Managed Metadata fields, when the on-premises techniques cannot be used?

This session walks through the “dev strategy” decisions we made at Content and Code, and why. Over the course of several demos, we’ll also discuss apps, automation scripts and also advanced techniques such as Continuous Integration for Office 365.

Event details

This year’s event is on 9th November 2013, and is held in Hinckley, near Leicester. If you’re interested, you can register here: