Sunday, 31 May 2009

More on optimization, HTTP 304s etc. – a solution?

In my last post Optimization, BLOB caching and HTTP 304s, I did a fairly lengthy walk-through on an issue I’d experienced with SharePoint publishing sites. A few people commented, mainly saying they’d noticed the same thing, but there have been further developments and findings I wanted to share!

Quick recap

Under certain circumstances some files in SharePoint are always re-requested by the browser despite being present in the browser cache (“Temporary internet files”). Specifically this is observed for files stored in the Style Library and Master Page Gallery, for anonymous users. Although SharePoint responds with a HTTP 304 to say the cached file can indeed be used (as opposed to sending the file itself again), we effectively have an unnecessary round-trip to the server for each file – and there could be many such files when all the page’s images/CSS/JS files are considered. This extra network traffic can have a tangible impact on site performance, and this is magnified if the user is geographically far away from the server.

A solution?

Waldek and I have been tossing a few development matters around recently over e-mail, and he was curious enough to investigate this issue for himself. After reproducing it and playing around for some time, Waldek discovered that flushing the disk-based cache seems to cause a change in behaviour – or in layman’s terms, fixes everything. To be more specific, we’re assuming it’s a flush of the BLOB cache which is having the affect – in both Waldek’s test and my subsequent validation, the object cache was also flushed as well:

FlushDiskCache

After the OK button is hit on this page, the problem seems to go away completely, so now when the page is accessed now for the first time as an anonymous user, the correct ‘max-age’ header is added to the files (as per the BLOB cache declaration in web.config) – contrast the ‘max-age=86400’ header on the Style Library files with what I documented in my last post:

AnonymousCorrectHeadersAfterFlushCache

This means that on subsequent requests, the Style Library files are served directly from the browser cache with no 304 round-trip:

SecondRequestNo304s

This is great news, as it means the issue I described is essentially a non-issue, and there is therefore no performance penalty for storing files in the publishing Style Library.

So what gives?

I’m now wondering if this is just a ‘gotcha’ with BLOB caching and publishing sites. I know other people have run into the original issue due to the comments on my previous post, and interestingly enough one poster said they use reverse proxy techniques specifically to deal with this issue. Could it really be that everybody who sees this behaviour just didn’t flush the BLOB cache somewhere along the way, when it’s actually a required step? Or is the testing that Waldek and I did flawed in some way? Or indeed, was my initial investigation flawed despite the fact others reported the same issue?

I am interested to hear from you on this – if you can reproduce the problem I’ve described with a publishing site you’ve developed, does flushing the BLOB cache solve it for you as described here? Leave a comment and let us know!

Good work Waldek :-)

14 comments:

Harold said...

Flushing the blobcache didn't solve the issue on my VPC. I have a VPC with the default "blueband" publishing site. After flushing the blobcache the max-age is still set to 0.

During testing I had some strange behaviour of fiddler though. I used Internet Explorer with an authenticated session and Chrome with an anonymous session. After I did a refresh of the page in Chrome I was suddenly authenticated!

Therefore I have used two other techniques to verify the max-age=0 issue. I have installed FireFox with firebug. In the network view of firebug all items in the style library had a max-age of 0.
Next I have installed wireshark on the SharePoint VPC to capture the "real" network packages. Then I browsed from my laptop to the site on the VPC with Internet Explorer. And wireshark still showed packeges with max-age=0....

Can you please verify that the max-age still isn't 0 when you're anonymous on the site? Or perhaps you were mislead (just as I was) by the strange behaviour of fiddler?

Kind regards,
Harold

Chris O'Brien said...

@Harold,

Very interesting - I'll do some more testing along these lines. One of the aspects which concerned me is that I'm constantly logging in/out to do the tests (all locally on the VM). I might try and get networking sorted to the VM (currently having Windows 7/VMware fun..) so that I can be anonymous on the host but can safely log in on the VM (e.g. to flush BLOB cache).

Will also nudge Waldek.

Thanks,

Chris.

Will du Preez said...

I have seen similar behaviour using Fiddler. It seems the authentication session is set up between the server and Fiddler. This means that as soon as one session running through Fiddler is authenticated, all sessions are.

I believe the way to get around this is the disable connection reuse (Fiddler -> Tools Menu -> General Tab). This means that Fiddler will make an new connection to the server on each request (thus acting more like a real proxy).

HTH!

Will

Chris O'Brien said...

@Will,

Hmm - could be that's what got me. I certainly wasn't aware of this behaviour with Fiddler.

Will take the steps you suggest when I get to retesting over the next day or two.

Thanks for this, much appreciated!

Chris.

Maxime Bombardier said...

Flushing the disk based cache (which only does it for that zone and server if you use the checkbox) may or may not temporarily fix this issue.

as I mentioned in the other post, it's a known bug that isn't currently in plan for fixing with MOSS 2007 (although that can change).

I'd deploy CSS/JS/XSL in a different library.

Maxime

Chris O'Brien said...

@Maxime,

Again, thanks for this. I was having difficulty verifying the test (either way) as per the previous comments, so it's good to get a more solid answer from someone in MS.

So we conclude there is a confirmed problem, the proposed solution is unlikely to be reliable, and my earlier post stands that we should look to store CSS/JS files in another library or the filesystem - in line with your recommendation.

Cheers!

Chris.

Anonymous said...

I am having Style Library caching issues too, but in my case it is authenticated users NOT anonymous users that are having issues (we do not allow any anonymous access).

The interesting thing is that the windows authenticated (default) zone IS caching the Style Library, but the forms authenticated (internet) zone is caching everything EXCEPT the Style Library. (The web.config blobcache directive is exactly the same in both the windows and forms sites).

We have looked for most gotchas (e.g. unpublished images won't cache, "Access to the path 'D:\BLOBCACHE\... is denied." in the application event log etc.) but so far no luck.

Can anyone confirm if the "known bug" relates only to anonymous users, or whether forms authenticated users are affected too? They seem to be.

Anonymous said...

I am fairly convinced that this bug affects not just anonymous users, but also forms authenticated (membershipprovider) users too.

I have tried endless suggestions and fiddles, including the one Chris just posted, but I cannot get the Style Library to cache for forms users.

Chris O'Brien said...

Interesting. Unfortunately I don't currently have access to a site which uses FBA and images in the Style Library to check. This would mean that it's even more important for static files to be stored outside of the Style Library (and we must test if storing in another library to ensure the problem doesn't occur there also).

Thanks for reporting this.

Chris.

Anonymous said...

Just to say that we eventually got the Style Library to cache for FBA users!

This was the exact order of events for us before it worked (not all steps may be necessary depending on your install):

1. Reset the blobcache back to "false" in both web.configs (default zone and internet zone).

2. iisreset.

3. Manually delete the blobcache directory.

4. Set the blobcache to "true" for the internet FBA zone only.

5. Go into the internet site as a site collection admin user, go to top level site settings > site collection object cache, select ALL the checkboxes, as per Chris's suggestion.

6. iisreset.

7. Browse to internet FBA site and it's now working!! (thanks Chris!)

8. Go back to the web.config for the default zone and set the blobcache back to "true"

A couple of things we found out along the way:

i. It's better to get your internet site blobcache working first if you have 2 zones.

2. Yslow (a free firefox addon) is a great tool for debugging what is/is not caching (just look at the components tab in the addon window)

Needless to say, it does all seem a bit fussy/flakey, and it took us multiple attempts before we got it to work. So to re-state what Maxime said - this "... may or may not temporarily fix this issue."

It definitely seemed to us like selecting all the checkboxes made the difference. The "reset disk based cache" checkbox on its own was insufficient.

Chris O'Brien said...

Wow, great write-up thanks.

Well done for getting it working. I can believe that process would work, but am slightly worried something might stop it all working down the line - the flakiness aspect you mention. Be interested to hear how you go over the next few days/weeks.

Many thanks for taking the time to write down your workaround!

Thanks,

Chris.

Anonymous said...

Hi Chris,

Just an update on the FBA site.

So far, despite our initial problems with getting things setup, the cache headers are still coming through just fine on Dev, Sys, and Uat platforms (currently undergoing load testing).

We have tested the site for a couple of weeks now, and the only other change we've had to make was to allow all of our FBA users to have restricted read access on the Style Library.

So the two main things seem to be:
1. Check all the boxes when you clear the cache, and
2. Make sure your users have (at least) restricted read access on the Style Library.

This seems to mirror your experience with anonymous users.

While Maxime indicated that there may be issues with the blobcache, and that it may be best to store the files elsewhere, this is not an option for us this late in the dev cycle. It's a shame she wasn't more specific about the exact nature of the problem.

Regardless, when I (and you as it happens) were at the Sharepoint Best Practices Conference, a colleague of mine specifically asked an MS Rep where we should be storing css and their associated images (as opposed to publishing images). We told him it was an FBA authenticated site with various levels of user permissions, and he stated that the Style Library would be the right place to put these. To be fair, caching wasn't mentioned to the MS Rep, but like I say, we are no longer having any issues and it's too late to change anything now...

Chris O'Brien said...

Thanks for the update.

I hope your environments continue to behave. I prefer the certainty which would come with storing the files elsewhere but am sure you'll be fine.

In terms of the conversation with the MS person - I think this issue has only really hit people's radar recently, so wouldn't necessarily expect every MS person involved with SharePoint to know the issue.

Cheers,

Chris.

Anonymous said...

Maxime has now blogged about this here:

http://blogs.msdn.com/maximeb/archive/2009/08/11/blobcache-style-library-and-anonymous-users.aspx