Implementing redirects when upgrading a site to SharePoint

Late last month we launched the best site i’ve ever had the pleasure of working on, Education + Training International (ETI). The pure size and scope of the site, particularly everything which drives it behind the scenes, is something which everyone who has worked on (and will continue to work on) it should be proud of. Having taken the site from inception to launch, the number of hurdles we came across along the way would be enough to fill this blog for the year. This post will touch on just one small aspect of it – ensuring that the URLs from the old site accurately redirected to those in the new.

I actually touched on this topic in some sense a couple of years ago in my post 301 vs 302 Redirects in SharePoint so it was good to get the opportunity to revisit it again, explore options and implement a solution. This post is quite long, so I’ve broken it up into headings outlining what is being discussed – feel free to skip ahead to what interests you.

Requirements and Caveats

There were 2 main considerations for requiring the redirects (and we wanted all URLs redirected – there wasn’t a ridiculous number we’d need to handle – around 130 or so). First and perhaps most importantly was user experience. The last thing I wanted was for our users to be consistently hitting a generic 404 page. The other consideration was for Search Engine Optimisation – ETI had previously invested in SEO on their existing site and we did not want to lose those search engine gains which had been realised.

There was a caveat to the decision making process (isn’t there always). The infrastructure team was not prepared to take on the burden of managing the redirects and had reservations anyway around the feasibility of managing them at the reverse proxy level.

I also personally wanted to avoid any manual deployment steps both out of principle and for disaster recovery and future maintenance reasons – those kinds of steps are often lost over time as people leave and documentation becomes outdated and ignored.

Finally, ideally the process would be able to be managed down the track, preferably through a SharePoint list within the site.

Rejected Solution #1 – custom web part on PageNotFound

My first investigation centred around the concept of placing a custom web part on the 404 PageNotFound page (having easy, editable access to this page from within SharePoint is a great improvement over previous versions). The idea was that a list would exist which managed the old URLs and the mapping to the new URL, if the ?requestUrl= found a match, we’d serve up a 301 redirect to the new URL, otherwise we’d provide the next best thing – perhaps a search within the site for some of the terms within the URL. The user experience was actually seemless with this approach, however a quick look at Fiddler showed that the 404 response was still served before my code hijacked it and served up the 301, meaning the search engines would assume the pages were gone. Back to the drawing board.

Rejected Solution #2 – storing URL rewrite mappings in a separate file

It was at this point that I gave up on my desire to have a user-editable list of redirects (perhaps prematurely). Our specific requirement was simply to redirect the old URLs to the new, future redirections were more of a nicety I was trying to provide. I knew a bit about managing redirects in IIS (I referenced Jeremy Thake’s post How we did it: 301 Redirects in IIS 7.5 on Windows Server 2008 R2 in my previous blog post) so figured I’d explore that track (I did have some reservations about blowing out the web.config particularly in relation to the 250kb file size cap, so I intended to use Ruslan’s Storing URL rewrite mappings in a separate file – I figured this would also allieviate the ‘adminstrative burden’ of managing them through IIS). The approach worked great – I could manage a set of old URL/new URL pairs in a separate configuration file, but to avoid the manual deployment steps I’d want it managed and deployed via our solution.

Unfortunately, that separate file had to exist along side the web.config in the IIS virtual directory for the site. I wasn’t able to reference an absolute path and deploy my configuration file somewhere within the SharePoint layouts directory. I investigated the ability to Deploy Files to SharePoint Web Application Virtual Directories At Feature Activation via Brian Jackett but the noted flaws in that system and the complexity around it quickly turned me off. Back to the drawing board.

Implemented Solution – writing URL rewrite mappings to web.config via feature

My next approach was to take a look at managing the redirects within the web.config itself. I did have concerns regarding blowing out the file and the maximum size limit for it – but tests soon proved that we wouldn’t even get close to that figure, so it wasn’t as much of a concern as I had made it out to be. So this approach worked fine too, the challenge was to remove the manual administration/deployment steps out of the process. Enter Using SPWebConfigModification to Update the Web.config in SharePoint 2013. I think i’ve explained how to do that reasonably well in that post (which i’ve also now updated to include some new information) so i’ll jump straight into some code snippets you can use in combination with that post.

            SPWebConfigModification modification = new SPWebConfigModification();
            modification.Path = "configuration/system.webServer";
            modification.Name = "rewrite";
            modification.Sequence = 0;
            modification.Owner = "ETIWebConfigModifications";
            modification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
            modification.Value = "<rewrite />";
            webApp.WebConfigModifications.Add(modification);

            modification = new SPWebConfigModification();
            modification.Path = "configuration/system.webServer/rewrite";
            modification.Name = "rewriteMaps";
            modification.Sequence = 0;
            modification.Owner = "ETIWebConfigModifications";
            modification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
            modification.Value = "<rewriteMaps />";
            webApp.WebConfigModifications.Add(modification);

            modification = new SPWebConfigModification();
            modification.Path = "configuration/system.webServer/rewrite/rewriteMaps";
            modification.Name = "rewriteMap";
            modification.Sequence = 0;
            modification.Owner = "ETIWebConfigModifications";
            modification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
            modification.Value = "<rewriteMap name='Redirects' />";
            webApp.WebConfigModifications.Add(modification);

            modification = new SPWebConfigModification();
            modification.Path = "configuration/system.webServer/rewrite/rewriteMaps/rewriteMap";
            modification.Name = "redirect-1";
            modification.Sequence = 0;
            modification.Owner = "ETIWebConfigModifications";
            modification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
            modification.Value = "<add key='/eti-overview/profile-of-education-training-international.html' value='/about-eti' />";
            webApp.WebConfigModifications.Add(modification);

            modification = new SPWebConfigModification();
            modification.Path = "configuration/system.webServer/rewrite";
            modification.Name = "rules";
            modification.Sequence = 0;
            modification.Owner = "ETIWebConfigModifications";
            modification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
            modification.Value = "<rules />";
            webApp.WebConfigModifications.Add(modification);

            modification = new SPWebConfigModification();
            modification.Path = "configuration/system.webServer/rewrite/rules";
            modification.Name = "rule";
            modification.Sequence = 0;
            modification.Owner = "ETIWebConfigModifications";
            modification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
            modification.Value = "<rule name='Redirect rule1 for Redirects' />";
            webApp.WebConfigModifications.Add(modification);

            modification = new SPWebConfigModification();
            modification.Path = "configuration/system.webServer/rewrite/rules/rule";
            modification.Name = "match";
            modification.Sequence = 0;
            modification.Owner = "ETIWebConfigModifications";
            modification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
            modification.Value = "<match url='.*' />";
            webApp.WebConfigModifications.Add(modification);

            modification = new SPWebConfigModification();
            modification.Path = "configuration/system.webServer/rewrite/rules/rule";
            modification.Name = "conditions";
            modification.Sequence = 0;
            modification.Owner = "ETIWebConfigModifications";
            modification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
            modification.Value = "<conditions />";
            webApp.WebConfigModifications.Add(modification);

            modification = new SPWebConfigModification();
            modification.Path = "configuration/system.webServer/rewrite/rules/rule/conditions";
            modification.Name = "add";
            modification.Sequence = 0;
            modification.Owner = "ETIWebConfigModifications";
            modification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
            modification.Value = "<add input='{Redirects:{REQUEST_URI}}' pattern='(.+)' />";
            webApp.WebConfigModifications.Add(modification);

            modification = new SPWebConfigModification();
            modification.Path = "configuration/system.webServer/rewrite/rules/rule";
            modification.Name = "action";
            modification.Sequence = 0;
            modification.Owner = "ETIWebConfigModifications";
            modification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
            modification.Value = "<action type='Redirect' url='{C:1}' appendQueryString='false' />";
            webApp.WebConfigModifications.Add(modification);

The only concerns I had left were around the performance implications of managing so many redirects within IIS/the web.config, but they were soon allayed via How to check for performance in URL Rewriting and IIS Rewrite Module rewrite map performance.

Was there a better option?

In the end I was pretty happy with the end result. I managed to take the burden off the infrastructure team (activating a web-app scoped feature is no significant task), managed to take the manual administrative steps (such as deploying files to specific locations which would need to be repeated if new WFE’s were added to the farm) out of the process and provided a seemless experience for the users while maintaining good SEO practices. The only thing I didn’t achieve was providing the user the ability to manage the redirects and I couldn’t help wonder if that was possible.

Thankfully I have the pleasure of working with someone who’s seen it all before and had a chance to try a few different approaches to this problem over time! Faced with this same challenge (but at a much larger scale), his solution was to create a custom IIS module which inspected the 404 requests going to the redirect page and converted them to 301’s, then a custom web part on that page did another 301 if it found an item in the old-new URL list, otherwise it wrote 404 to the response header for the nicely branded 404 error page.

The only downside to this approach I could see was the need to install the custom IIS module on each server – however is that really worse than having to install the IIS rewrite module on each server? Probably not. The approach also had a significant unexpected upside – by monitoring the analytics on the 404 page they were able to identify legacy links which were still being used across the web, and had the ability to redirect those to a new URL on the fly and provide ongoing functionality to redirect merged/renamed/moved pages on the ever evolving web site.

So hopefully this post has given you a few ideas regarding how to implement your redirects when upgrading a legacy site into SharePoint. As always there are pro’s and con’s to each approach but there are definitely options available to achieve a decent result to meet a number of different requirements.

Using SPWebConfigModification to Update the Web.config in SharePoint 2013

Seven months ago I wrote an article on Jumping the Hurdles of using SPWebConfigModification to Update the Web.config – that article was based on my experiences in SharePoint 2010 and having researched the topic thoroughly at the time I was interested to see how things fared in SharePoint 2013. Thankfully I had the opportunity to architect a solution from the ground up in 2013 which gave me the opportunity to once again adhere to the vow I made in Application Settings in SharePoint to never manually modify the web.config file.

While this post has not been as thoroughly researched at the time of writing as I usually would like, a quick search on SPWebConfigModification in SharePoint 2013 brought up few results and when I was looking into it for the project mentioned above, little had been written about it. A recent question on the OZMOSS mailing list showed that there were still some questions around how SPWebConfigModification did (or didn’t) work so I thought it would be useful to document my findings here for anyone wanting to ‘do the right thing’ and keep their hands off the web.config file.

For a bit of background for this post I’d thoroughly recommend you read the article I link to above – the purpose of this post is to compare the behaviour of the functionality between versions of the platform and it will make far more sense if you understand my (and others) previous findings.

However for those of you who care little about the past and just want to know what the deal is in 2013 – here is the quick summary:

SPWebConfigModification in SP2010 came with a number of issues however none which were completely insurmountable with a little work. The majority were already documented by others (and I link to them in that post) however one phenomenon had little written about it and even less in terms of a solution.

This problem was that even though the removal code ran successfully and even removed the entry from the modification collection, the element in the web.config file still physically existed. I came up with a workaround where  in the removal code I first applied another modification reverting the entry back to its original state, then ran the code to remove all modifications.

So how did this fair in SP2013? There was good news and bad news. On the plus side, this issue had been fixed! Removing the entry from the modification collection successfully removed the entry from the web.config file. On the negative side it came with a nasty side effect – if the modification you were removing was a change to an attribute on a pre-existing element then that whole element was removed from the web.config, not just your change.

This was clearly unacceptable behaviour particularly if the entry being removed was essential to successfully loading the website.

Once again however I managed to jump the hurdles that always seem to exist with this tricky class. The workaround this time was to remove all the customisations by default at the beginning of both the FeatureActivated and FeatureDeactivating functions and then add the customisation required. In FeatureActivated this would be your customised entry. In FeatureDeactivating this would be the original entry (if it existed in the web.config before your modifications). Again this solution isn’t full-proof and is prone to falling over in future iterations of the platform, however it is a solid workaround as things stand now. I’ve provided a code snippet to achieve this below:

public class WebConfigModificationsFeatureEventReceiver : SPFeatureReceiver
{
	// Due to what appears to be a bug or just and unfortunate side-effect in SP2013, using SPConfigModification on an existing element
	// replaces that element with a new one. When removing the customisation, that element is removed, hence meaning that the original
	// entry no longer exists. To counter this we will re-add the original values in the deactivating feature.

	public override void FeatureActivated(SPFeatureReceiverProperties properties)
	{
		SPWebApplication webApp = properties.Feature.Parent as SPWebApplication;
		RemoveAllCustomisations(webApp);

		#region Enable session state

		httpRuntimeModification = new SPWebConfigModification();
		httpRuntimeModification.Path = "configuration/system.web/pages";
		httpRuntimeModification.Name = "enableSessionState";
		httpRuntimeModification.Sequence = 0;
		httpRuntimeModification.Owner = "WebConfigModifications";
		httpRuntimeModification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureAttribute;
		httpRuntimeModification.Value = "true";
		webApp.WebConfigModifications.Add(httpRuntimeModification);

		httpRuntimeModification = new SPWebConfigModification();
		httpRuntimeModification.Path = "configuration/system.webServer/modules";
		httpRuntimeModification.Name = "add[@name='Session']";
		httpRuntimeModification.Sequence = 0;
		httpRuntimeModification.Owner = "WebConfigModifications";
		httpRuntimeModification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureChildNode;
		httpRuntimeModification.Value = "<add name='Session' type='System.Web.SessionState.SessionStateModule' preCondition='' />";
		webApp.WebConfigModifications.Add(httpRuntimeModification);

		#endregion

		/*Call Update and ApplyWebConfigModifications to save changes*/
		webApp.Update();
		webApp.Farm.Services.GetValue<SPWebService>().ApplyWebConfigModifications();
	}

	public override void FeatureDeactivating(SPFeatureReceiverProperties properties)
	{
		SPWebApplication webApp = properties.Feature.Parent as SPWebApplication;
		RemoveAllCustomisations(webApp);

		#region Revert session state

		httpRuntimeModification = new SPWebConfigModification();
		httpRuntimeModification.Path = "configuration/system.web/pages";
		httpRuntimeModification.Name = "enableSessionState";
		httpRuntimeModification.Sequence = 0;
		httpRuntimeModification.Owner = "WebConfigModifications";
		httpRuntimeModification.Type = SPWebConfigModification.SPWebConfigModificationType.EnsureAttribute;
		httpRuntimeModification.Value = "false";
		webApp.WebConfigModifications.Add(httpRuntimeModification);

		#endregion

		/*Call Update and ApplyWebConfigModifications to save changes*/
		webApp.Update();
		webApp.Farm.Services.GetValue<SPWebService>().ApplyWebConfigModifications();
	}

	private void RemoveAllCustomisations(SPWebApplication webApp)
	{
		if (webApp != null)
		{
			Collection<SPWebConfigModification> collection = webApp.WebConfigModifications;
			int iStartCount = collection.Count;

			// Remove any modifications that were originally created by the owner.
			for (int c = iStartCount - 1; c >= 0; c--)
			{
				SPWebConfigModification configMod = collection[c];

				if (configMod.Owner == "WebConfigModifications")
				{
					collection.Remove(configMod);
				}
			}

			// Apply changes only if any items were removed.
			if (iStartCount > collection.Count)
			{
				webApp.Update();
				webApp.Farm.Services.GetValue<SPWebService>().ApplyWebConfigModifications();
			}
		}
	}
}

So once again we’re left with an option to do things right which doesn’t quite act the way you’d expect, but as you can see, can still be used to achieve the end result desired. For all its ills in both versions of the products i’d still strongly recommend taking this approach to making changes to the web.config file over making manual modifications any day. I hope this post has made doing so a little easier moving forward.

UPDATES

We shouldn’t be surprised, but over time new hurdles have presented themselves which need documenting (and I apologise, the first one I mention I should have covered a long time ago when I encountered it!).

The approach documented above worked fine in my single-server development environment. As soon as we tried activating the feature in a multi-server farm test environment all sorts of issues occurred. This has been documented however and you can read about it in Jeremy Jameson’s post Waiting for SharePoint Web.config Modifications to Finish and Simon Doy’s post PowerShell to Detect Web Configuration Modification jobs.

The code changes required include a few functions to be added and a slight modification to the ApplyWebConfigModifications call as indicated in the below code.

        private static bool IsJobDefined(SPFarm farm)
        {
            SPServiceCollection services = farm.Services;

            foreach (SPService service in services)
            {
                foreach (SPJobDefinition job in service.JobDefinitions)
                {
                    if (string.Compare(job.Name, jobTitle, StringComparison.OrdinalIgnoreCase) == 0)
                        return true;
                }
            }

            return false;
        }

        private bool IsJobRunning(SPFarm farm)
        {
            SPServiceCollection services = farm.Services;

            foreach (SPService service in services)
            {
                foreach (SPRunningJob job in service.RunningJobs)
                {
                    if (string.Compare(job.JobDefinition.Name, jobTitle, StringComparison.OrdinalIgnoreCase) == 0)
                        return true;
                }
            }

            return false;
        }

        private void WaitForOneTimeJobToFinish(SPFarm farm)
        {
            float waitTime = 0;

            do
            {
                if (!IsJobDefined(farm) && !IsJobRunning(farm))
                    break;

                const int sleepTime = 500; // milliseconds

                Thread.Sleep(sleepTime);
                waitTime += (sleepTime / 1000.0F); // seconds

            } while (waitTime < 20);
        }

        /*Call Update and ApplyWebConfigModifications to save changes*/
        webApp.Update();
        WaitForOneTimeJobToFinish(webApp.Farm);
        webApp.Farm.Services.GetValue<SPWebService>().ApplyWebConfigModifications();

The second issue is one we’ve encountered more recently. The issue was in regards to the inability to successfully extend a web application which already had the web.config modifications applied. There’s 2 ways to approach this one – if you know before the fact, then make sure you extend your web application before applying any of the web.config modifications. Otherwise, hope that you’ve implemented decent deactivation code and be sure to deactivate the features before extending, and then reapply after the fact.

Integrating Facebook into SharePoint

As mentioned in my post Integrating Twitter into SharePoint I was recently faced with creating a stream from both Twitter and Facebook to display on a public-facing SharePoint website. While the Twitter experience was quite simple and enjoyable, that which was experienced when jumping across onto the Facebook side of the fence was far less so. In the end it actually wasn’t too difficult to implement, however the road there was full of potholes and speed humps.

My first stop was attempting to find an example of one which had already been implemented and the options were surprisingly limited. The closest I came was 3PillarLabs’ Facebook Webpart for SharePoint – Part 1 which looked like it would have at least given me the code samples I needed to get my own implementation working – if I was approaching it around the same time frame. The problem was that I continually (usually on the 2nd load) received an error: The remote server returned an error: (400) Bad Request. Viewing a number of recent comments on their code page seemed to indicate this was not just isolated to me.

The main issue was that in late December Facebook changed how their authentication tokens worked. You can read more about this both on Facebook’s own Removal of offline_access permission page and Randy Hoyt’s Changes to Facebook API: Access Tokens and offline_access. Long story short, 2 things seemed to now be the case – you couldn’t simply store a Facebook application code as per the example highlighted above because it seemed to last all of 2 minutes, and you couldn’t get a user token that would allow offline access which would never expire.

I was left with two options – I either needed to work with a page access token instead or refresh the user access token (which, after extended, would last up to 60 days). My original intent was to pull back Facebook statuses and unfortunately the page access token simply wouldn’t work for this purpose – I continually received the error (OAuthException – #102) A user access token is required to request this resource. Now this threw me off track a fair bit – as it turns out, you can actually use the page access token to retrieve a number of other things that would suit a similar purpose – namely feeds and posts. If you are happy with this approach you can skip the next paragraph and flick ahead, as this was how I finally implemented the solution.

It’s possible however that you may be faced with a situation where you need access to something that requires a user access token and therefore the brainstorming I did to solve this issue may still be of use. Essentially what I planned on doing was having a CustomAction in the SiteActions group which would make 3 calls – the first to access the application code via the URL


https://graph.facebook.com/oauth/authorize?client_id=CLIENTID&redirect_uri=CustomActionURL&scope=read_stream

the second to access a short-term user access token via the URL


https://graph.facebook.com/oauth/access_token?client_id=CLIENTID&redirect_uri=CustomActionURL&client_secret=CLIENTSECRET&code=CODE&scope=read_stream

and the third to extend the short-term user access token into a long (60-day) one via


https://graph.facebook.com/oauth/access_token?client_id=CLIENTID&client_secret=CLIENTSECRET&grant_type=fb_exchange_token&fb_exchange_token=USERTOKEN

This long term token would then be stored in a property bag and accessed by the web part. This process however would need to be run within every 60 days – it’s possible you could instead use a timer job to automate that process.

Before continuing on it’s important to point out that the CLIENTID and CLIENTSECRET values are easily retrieved similarly to how Twitter’s were – all you need to do it sign in to the Facebook app development page and create a new app.

The final piece of the puzzle was to find an SDK to make life a little easier when programming against the Facebook API. Unfortunately, while SpringSource have a Facebook SDK for their Java framework, the ported Spring.NET equivalent for .NET (which I used for Twitter) did not. I ended up using the Facebook C# SDK for ASP.NET. Unfortunately though when using their code samples I ended up with an error as shown below: Dynamic operations can only be performed in homogenous AppDomain.

dynamic-operations

A little searching turned up that this occurs when trying to use dynamic properties with a web.config entry of <trust level=”Full” legacyCasModel=”true” /> – interestingly enough something Corey Roth pointed out was changed in SharePoint 2013 in his article New level of trust in SharePoint 2013 Preview. Rather than adjust that value which seemed fraught with danger (and was recommended against by one Microsoft employee, Humberto Lezama, in an MSDN forum post Getting SharePoint 2013 and MVC 4 to co-exist), I simply adjusted the code to work without dynamic properties.

The end result – finally – was a success.

facebook-feed

var oAuthAppToken = GetOAuthToken();
var client = new FacebookClient(oAuthAppToken);
var me = client.Get(FacebookId, new { fields = "posts.limit(10).fields(message,updated_time)" }) as IDictionary<string, object>;
var posts = (IDictionary<string, object>)me["posts"];
var data = (JsonArray)posts["data"];

List<FacebookStatusDisplay> statusDisplays = new List<FacebookStatusDisplay>();
foreach (var theStatus in data)
{
    var status = (IDictionary<string, object>)theStatus;

    if (status.ContainsKey("message"))
    {
        string message = (string)status["message"];
        string updatedTime = (string)status["updated_time"];
        statusDisplays.Add(new FacebookStatusDisplay(message, updatedTime));
    }

    if (statusDisplays.Count == 2) break;
}

if (statusDisplays.Count > 0)
{
    StatusRepeater.DataSource = statusDisplays;
    StatusRepeater.DataBind();
}
else
{
    FacebookFeed.Visible = false;
}

Note above that it is important to check for the message before trying to add the post, because we’re not dealing with statuses directly it’s possible that a message won’t exist within the post.

string url = string.Format("https://graph.facebook.com/oauth/access_token?client_id={0}&client_secret={1}&grant_type=client_credentials", OAuthClientID, OAuthClientSecret);
WebRequest request = WebRequest.Create(url) as HttpWebRequest;

using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
    StreamReader reader = new StreamReader(response.GetResponseStream());
    string retVal = reader.ReadToEnd();
    oAuthToken = retVal.Substring(retVal.IndexOf("=") + 1, retVal.Length - retVal.IndexOf("=") - 1);
}

It’s also worth noting that like the Twitter integration I used a SharePoint list for configuration, had the necessary error handling present and used the PrettyDate function identified to retrieve the correct time-lapsed date.

So overall with this knowledge at hand it would be relatively simple to knock out another web part which read posts from – importantly – a publicly accessible Facebook page. There were just so many little issues that had to be overcome to make this an enjoyable experience, namely the changes Facebook have made to their authentication. I feel for Facebook application developers and am glad this should be the extent that I need to interact with Facebook within SharePoint in the near future!

Integrating Twitter into SharePoint

It’s been over a year since I wrote about Integrating WordPress into SharePoint as one of my first (and most visited) posts on this blog, so it seemed only fitting that early into the new year I found myself faced with the task of integrating both Twitter and Facebook into SharePoint 2013. As it was when it came to WordPress, the integration to be discussed here is light – simply displaying a feed of the latest tweets on the site – but the hurdles encountered are likely to be helpful and relevant with any level of integration you’d be capable of achieving with a bit of study of the Twitter API and third party SDKs.

Thankfully there was a lot of information available on the net in regards to programming against Twitter, the first source being the Twitter API itself. It wasn’t long after that I learnt that there were mature and decent SDKs available to make programming against Twitter even easier – one of the .NET ones recommended by Twitter themselves was Spring.NET Social for Twitter which proved easy to learn, integrate into the solution and use.

The first thing to know about creating an application for Twitter is that you must register it. Creating a new application will give you the Consumer key and Consumer secret you’ll need to identify your app, and creating an access token will give you the Access token and Access token secret to authorise your application to interact with your feed. I chose to store these values as configuration items using the list approach identified in my post Application Settings in SharePoint.

The second thing to know is that the Twitter API is rate limited (180 accesses per 15 minutes for user feeds) and therefore needs to be considered. This is easily countered by caching the results which are returned – obviously not ideal considering the results returned would not be up-to-date to the minute however necessary if you believe the site will be accessed that many times within a 15 minute period.

So the next step was to code up a proof of concept, but as we all know, approaching any task within the confines of SharePoint tends to lead to unexpected hurdles that need to be overcome. The first issue I encountered was ‘No connection could be made because the target machine actively refused it’. This turned out to be a bit of a red herring – the target machine had nothing to do with the problem, it was in fact a replica of the proxy issue I came across when integrating WordPress and was resolved in the same way. The error encountered and the web.config entry to resolve it can be seen below.

proxy-error

<system.net>
  <defaultProxy>
    <proxy proxyaddress="http://your-proxy-address:port" bypassonlocal="true" />
  </defaultProxy>
</system.net>

Thinking that was all a bit too easy and looking forward to seeing the end result I was immediately fronted with another issue: ‘The remote certificate is invalid according to the validation procedure’ which could be further narrowed down to ‘The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel’. This one took a little longer to get over as there were a lot of posts on the net which proposed unresponsive or unattractive resolutions to the problem. There were however a few ways in the end that this one could be resolved.

certificate-error

Firstly, it was possible to set a web.config entry to get SharePoint to ignore the problem altogether – however this did not seem like the most security conscious way to approach it.

<system.net>
    <settings>
      <servicePointManager
          checkCertificateName="false"
          checkCertificateRevocationList="false"
      />
    </settings>
</system.net>

Secondly, it was possible to get SharePoint to ignore the problem via code – basically the same approach as above and similarly unattractive.

ServicePointManager.ServerCertificateValidationCallback = new System.Net.Security.RemoteCertificateValidationCallback(delegate { return true; });

Finally, and probably the most appropriate and security conscious approach to take, was to register the SSL certificate as a trusted certificate in SharePoint. This is where it took a bit of trial and error to find the right one – most guides suggested that trusting the right root certificate from VeriSign would do the trick however it wasn’t until I exported the appropriate certificate from https://api.twitter.com that I got it working – it’s important to note I needed to export the VeriSign Class 3 Secure Server CA – G2 certificate, not the root. You can read up on how to export the certificate in Sean Wallbridge’s post SharePoint 2010 and Cert Trust – Could not establish trust relationship for the SSL/TLS secure channel and either use his Central Administration instructions listed or the following PowerShell commands:

$root = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2("(location)\VeriSign Class 3 Secure Server CA - G2.cer")
New-SPTrustedRootAuthority -Name "VeriSign Class 3 Secure Server CA - G2.cer" -Certificate $root

So after all that I had a working twitter integration – however the feed only returned the CreatedAt date and not the relative time lapsed since the tweet was made. I used a couple of posts to implement that feature, namely Sam Allen’s C# Pretty Date and a thread in StackOverflow Calculating relative time.

The end result and some code snippets to achieve it can be seen below – overall the experience of creating an integration application with Twitter was relatively pain free and easy to accomplish which is more I can say for my experience with integrating Facebook!

twitter-integration

 ITwitter twitter = new TwitterTemplate(ConsumerKey, ConsumerSecret, AccessToken, AccessTokenSecret);

 IList<Tweet> tweets = twitter.TimelineOperations.GetUserTimeline(2);
 foreach (Tweet tweet in tweets) tweetDisplays.Add(new TweetDisplay(tweet));

 // Cache the result for 20 minutes (Twitter limit access for a certain number of calls per 15 minutes)
 CachingHelper.AddToCache(cacheKey, tweetDisplays, new TimeSpan(0, 20, 0));

 TweetsRepeater.DataSource = tweetDisplays;
 TweetsRepeater.DataBind();

Jumping the Hurdles of using SPWebConfigModification to Update the Web.config

In my Application Settings in SharePoint post earlier in the year I vowed that on any new project I would ensure that the web.config file would never be (ab)used with manual modifications to store application settings. I’m happy to say that I stuck to that goal, and this post is a reflection on my experience since of avoiding any manual intervention with the file. As I discovered there are a number of potential issues and gotchyas with using the SPWebConfigModification class including one particular bug which will be discussed later – it is important that one has read the numerous informative and valuable posts on the topic before endeavouring to ‘do things the right way’ when making updates or changes to the web.config.

You could be excused for thinking that with the myriad of issues that seem to exist when using SPWebConfigModification that it couldn’t possibly be a suitable method for making updates to the web.config. It is however generally considered a best practice and there is good reason for its use. Anyone who has studied for and sat the application development exam (70-576) would know that it is clearly positioned as the right way to modify values in the configuration file. It is listed as one of two available techniques in Managing Web.config Modifications, mentioned in the patterns & practices SharePoint Guidance in regards to adding entries into the web.config and referenced as one of the 10 Best Practices For Building SharePoint Solutions by having SharePoint Manage Custom Config Settings.

Rather than take it as gospel though there are some great posts out there that explain exactly why it should be used. One of the best is Charlie Lee’s What every SharePoint developer should know about the web.config file which sheds some light on how the web.config is pieced together while stressing the fact it should not be manually modified. Mark Wagner’s How To: Modify the web.config file in SharePoint using SPWebConfigModification, while being written for SP2007 and being a great resource in outlining exactly what SPWebConfigModification and its properties do, also discusses the advantage of having the class propogate your changes across the farm. Finally, Jonathon Le exposes some of the issues with making manual modifications in his post SPWebConfigModifications and Manual Config entries don’t mix.

But as I also mentioned in my Application Settings post, the internet is littered with issues and problems with using SPWebConfigModification. While this shouldn’t preclude its use, it’s important to understand everything that may go wrong and the things to pay particular attention to so any potential issues can be avoided. The first port of call here is Reza Alirezaei’s SPWebConfigModification’s Top 6 Issues. It highlights some of the common pitfalls that occur when trying to use it and how to work around them. One important thing to note here is a lot of the best resources around this topic were written for SharePoint 2007 and hence are not all still applicable in 2010, however a vast number of them still are.

So how did I implement it myself? As per numerous others I used a Web Application scoped feature with FeatureActivating and FeatureDeactivating event receivers to run the necessary code. MSDN’s How to: Add and Remove Web.config Settings Programmatically page proved reasonably useful however it’s important to note that the two code samples use different methods of applying the modifications – I adjusted the adding code to use the web application format present in the removing code. I also adjusted the removal code to ensure all modifications from a particular owner were removed, not just the most recent.

I was quick to learn however that using the Web Application scoped feature came with its own side effect – as soon as the solution file was deployed the feature was activated across all web applications automatically – not just the one I intended it for. A more in depth explanation and the resolution to this issue can be found at Waldek Mastykarz’s post Inconvenient SPWebConfigModification development with Visual Studio 2010 SharePoint Developer Tools and Nancy Brown’s SPWebConfigModification 2010: Change, Caveat, and Code – the latter also has a neat code sample on how to back up the web.config files before applying changes.

That wasn’t the only issue I ran into though. At one stage I was getting duplicate entries resulting in a configuration error when trying to load the site, and while I managed to resolve the reason for this on my own, was interested to see that Nancy had already documented the exact phenomenon in her article SPWebConfigModification Tool and Explorations under the How to Make the Removal Fail (or “How This Can Make You Nuts”) heading. This is another post worth reading as it includes a lot more valuable information within it.

The most intriguing issue overall however was the fact that my removal code didn’t seem to be taking effect although it was running without error and successfully removing the change from the modification collection. Essentially, it just wasn’t being reflected in the web.config. The change I had made was to add an attribute to an existing element within the file and while it added it fine it just wouldn’t revert back to its previous state after being ‘removed’. I stumbled across a couple of Technet forum posts; WebConfigModification Issues and Remove the web.config entries programmatically not working in sharepoint 2010 in which a Microsoft staffer, Rickee, indicated it was actually a bug in SharePoint 2010 and one which was not due to be fixed. He mentioned within those threads that it may be possible that while it visibly existed in the web.config file it may not have actually been taking affect – possibly a satisfactory conclusion to some but not something I was content with (as a side note, I never did end up testing whether that theory was correct – I wanted that attribute gone!).

I ended up coming up with my own solution – in the removal code I first applied another modification reverting the entry back to its original state, then ran the code to remove all modifications (including the one I had just run). This resulted in a clean modifications collection and a web.config entry which mirrored the initial value. While its not a perfect solution (that value could easily be set to something different in a subsequent version of SharePoint!) I figured it suited my immediate needs just fine.

As you can appreciate from this article there are a number of things you need to consider when modifying the web.config using SPWebConfigModification and you must take great care when doing so, lest you end up in a situation described in Michael Nemtsev’s post How SharePoint manages web.config via SPWebConfigModification! What’s important though is that you understand the risks and pitfalls but more pressingly understand the importance of using it to apply your configuration updates.

(Update: To see how SPWebConfigModification performs in SP2013, take a look at Using SPWebConfigModification to Update the Web.config in SharePoint 2013)

Application Settings in SharePoint

One of the most commonly neglected features in SharePoint I find across a number of different projects is the manner in which application settings are managed. Far too often it is left to manually modifying the <AppSettings> section of the web.config file for the relevant web application. Perhaps this is due to legacy behaviour of ASP.NET developers moving into the SharePoint sphere, perhaps its due to environments being small and the risks of manually modifying the file not being exposed. It may be due to organisational process or procedure, or (most likely) it may be because on face value it appears to be the most simple, quick and easy option to maintain some configuration values.

It’s ironic that in projects laced with features and solutions to deploy resources ‘the right way’ that configuration is overlooked and remains a manual process. Aside from wanting to avoid any manual modification of files directly on the server by principle, dangers exist in terms of ensuring the web.config files are the same on all web front end servers. Maintaining application settings this way also requires access to the web.config file (and hence server) itself which wouldn’t be considered best practice, and the act of changing a variable in the web.config directly would cause the application pool to refresh and cause the site to load slowly on the next hit.

I write this post with a tinge of regret – I too have been guilty on multiple occassions of maintaining the status quo and leaving (and even adding to) configuration values maintained manually in a web.config file. So not only should this post serve as a source of information regarding the best practice methods to maintain application settings, but it should also serve as a motivator to myself and others to practice what I/we preach on all future greenfield projects.

Anyone who has studied for or taken the application development exam (70-576) would have noticed that 4 main options consistently get presented for application configuration: web.config, lists, property bags and the hierarchical object store. I’ll discuss each option briefly and provide some situations where the method would be appropriate.

Web.config

Now it may seem like i’m beginning to contradict myself here – I previously mentioned that manually maintaining application settings in the web.config was a BAD idea. It is. The beauty of SharePoint though is that it provides a mechanism to maintain application settings in the web.config file automatically via the SPWebConfigModification class. If you’re wanting to store them in the web.config, then a feature should be created which on activation uses the SPWebConfigModification class to modify the web.config for you. This ensures that the same values are written to the web.config files on all front-end web servers and even ensures they’re consistent amongst new servers added to the farm.

You’re likely to want to use this option if you just can’t pull yourself away from the web.config for variable configuration. At least its a safer method. There are some valid reasons however – it’s reasonably simple to incorporate this functionality and then easy to program with seeing you’re probably used to it already. It ensures values are consistent across all front-end web servers and are available and consistent across the entire web application. It’s also fairly useful for ‘set and forget’ the values – if you’re unlikely to need to change them in the future, then you won’t run into the application pool refresh issue I mentioned previously and will get the performance benefits of the cached settings.

Before running off and using this method however, be warned that the web is littered with issues and problems with using SPWebConfigModification. I’d probably want to avoid making any ‘unnecessary’ changes to the web.config especially for the purposes of setting configuration values considering there are other options available.

SharePoint lists

It almost seems too easy and obvious. SharePoint provides us with these useful things called lists. Most of us would be familiar with or have used database-driven variable configuration mechanisms before. Lists, on face value, are reasonably similar to database tables. Why don’t we use lists for variable configuration? It’s not actually that silly a proposition and is in fact one of the more reasonable options. One of the beauties of using lists is that you get all that SharePoint listy goodness. Versioning, security and an out-of-the-box UI for example. You’re likely already familiar with list-based programming, so that isn’t too much of an inconvenience.

Using SharePoint lists is definitely one of the preferred methods. It’s probably easier to outline some of the reasons why you may not want to use it; a site with lax security restrictions would probably be wise to avoid this method – you wouldn’t want any user being able to modify or delete configuration items or the list itself. If you wanted the application settings to be available to multiple site collections/web applications then you’d potentially end up duplicating the list values across these sites.

There’s another huge advantage to choosing this method. An MVP named Chris O’Brien has written a Config store framework available on Codeplex which makes the option even more attractive and erases some of the drawbacks from choosing it. I’d definitely recommend reading a bit about it in his post A better Config Store for SharePoint sites.

Property bags

This option is perhaps a little less obvious but similarly powerful. A number of SharePoint objects contain a Properties property which stores a collection of key/value pairs. These obviously lend themselves towards being used for storing application settings.

This option is possibly the best one to harness when scope is an important consideration for configuration. It lends itself to being able to maintain numerous variables with different values on a per-scope basis (per web, per web application and so forth).

Like the SharePoint list option, this also benefits from addons which improve the usability and effectiveness of the option. The SharePoint Guidance Library provides the Application Setting Manager which is an API to read and write application settings. Codeplex also has the SharePoint Property Bag Settings project which provides a GUI in Central Administration to manage the settings.

Hierarchical Object Store

This is a farm-scoped store similar to a property bag. It requires custom code to create and is far more complex than the other options, however has the benefit of being able to store more complex types if that’s neccessary. Generally for the purpose of variable configuration though, in my opinion i’d go for one of the other options.

You’d possibly choose this option if you were looking for a global storage option for values or required complex values to be stored. If the values are simple key/value pairs however a farm-scoped property bag would probably be easier to use and do the trick.

WebPart properties

I know I said before there were 4 main options, but I wanted to briefly cover this one too. In the event that your custom web parts require configuration values specific to the web part then it will often make sense to include custom properties editable in the web part itself – they are then able to be set by modifying the settings of the web part.

My initial concept for this post was to include an array of advantages, disadvantages and code samples on each option to more completely highlight where each could be used and how they’re used, however during my research for this post came across an MSDN article written by David Mann titled Managing Custom Configuration Options for a SharePoint Application – it essentially covers in reasonable detail those points and I would strongly recommend reading.

Overall I’d have to say I prefer the SharePoint list method the best. I’m somewhat surprised, I remember thinking a while ago that using lists for application settings would be a bit primitive and that surely the other options which have been provided (obviously for the purpose of variable configuration?) like SPWebConfigModification, SPPersistedObject and the Properties collections would be the best practice method to use. I guess sometimes the easy option actually is the best, and it’s sandbox-friendly to boot.

Integrating WordPress into SharePoint

The title here may be slightly misleading. The majority of this post will be applicable to any .NET application integrating with WordPress – however my experiences were derived from creating the WordPress functionality within the Career Centre portal. The requirements were for an extract of the latest post to appear on the front page and for any relevant categories to be presented on the Career Connect page. The featured article functionality was initially intended to come from WordPress but ended up being list-driven as there was no obvious way to mark the WordPress blog as a feature (In hindsight I may have created a ‘Feature’ category in WordPress and use the API to bring it back that way, but as it stands it’s relatively static).

This actually turned out a little more challenging than I was expecting. There appeared to be a lot of information in regards to WordPress APIs as it related to plugin development, but the information I was searching for wasn’t as obvious to find. I eventually found out that WordPress supports both the Atom Publishing Protocol (AtomPub) and XML-RPC for client development and that currently WordPress provides more features and capabilities via XML-RPC than AtomPub. XML-RPC seemed the way to go.

This snippet of information actually made life a lot easier. Turns out that the information WordPress provides on this subject is actually pretty useful. Firstly there’s the XML-RPC Support page which identified that 3 APIs were supported; Blogger, metaWeblog and Movable Type. There’s also an extension to the Movable Type API specific for WordPress (conveniently called the WordPress API) that should be used as a first resort if possible. That API is pretty well documented too on the XML-RPC wp page.

It seemed pretty promising – getCategories? Perfect. There didn’t seem to be an obvious way to get the latest blog post though from this API so the search continued. Reading through the alternate API’s metaWeblog seemed the easiest to use and appeared to have the functionality I needed. Unfortunately I found it to be pretty poorly documented – it seemed that no one wanted to make this task easy! I did stumble across one page; Eric’s WordPress XML-RPC – MetaWeblog API. Without this page I think I would have had a much harder time getting everything to work – getRecentPosts? Perfect.

The only thing left to do was to code it up in .NET. Turns out there’s an excellent library for this purpose by Charles Cook called XML-RPC.NET. The download includes the DLL you’ll need to reference (CookComputing.XmlRpcV2.dll) and a bunch of examples to get you started.

Time to pull out a few code snippets. Firstly you need to create an interface with the functions you intend to call:

public interface IStateName : IXmlRpcProxy
{
    [XmlRpcMethod("wp.getUsersBlogs")]
    Blog[] GetUsersBlogs(string username, string password);

    [XmlRpcMethod("metaWeblog.getRecentPosts")]
    BlogEntry[] GetPosts(int blog_id, string username, string password, int numberOfPosts);

    [XmlRpcMethod("wp.getCategories")]
    Category[] GetCategories(int blog_id, string username, string password);
}

The arrays are public structures with the properties you’re wanting to pull back – note that casing is important for the properties – make sure it matches the documentation exactly. Next is the code to retrieve the most recent post, and then all the blog categories:

IStateName proxy = XmlRpcProxyGen.Create<IStateName>();
proxy.Url = XMLRPCUrl;

BlogEntry[] blogEntries = null;
BlogEntry blogEntry = new BlogEntry();
Blog[] blogs = proxy.GetUsersBlogs(Username, Password);
if (blogs.Length > 0)
{
    Blog blog = blogs.FirstOrDefault(s => s.blogName == BlogName);

    if (blog.blogid != null)
    {
        blogEntries = proxy.GetPosts(int.Parse(blog.blogid), Username, Password, 1);
        blogEntry = blogEntries[0];
    }
}

 

IStateName proxy = XmlRpcProxyGen.Create<IStateName>();
proxy.Url = XMLRPCUrl;

Category[] categories = null;
Blog[] blogs = proxy.GetUsersBlogs(Username, Password);
if (blogs.Length > 0)
{
    Blog blog = blogs.FirstOrDefault(s => s.blogName == BlogName);

    if (blog.blogid != null)
    {
        categories = proxy.GetCategories(int.Parse(blog.blogid), Username, Password);
    }
}

The XMLRPCUrl is your wordpress address /xmlrpc.php (ie http://spmatt.wordpress.com/xmlrpc.php), the BlogName is the name of your blog (ie SPMatt) and your Username and Password is pretty self explanatory – use the account you use to author/administer your blog (I believe some of the actions you’re able to perform are dependant on the credentials of the account you use – using the administrator account is obviously the bad practice / guaranteed success way of getting it working).

That’s pretty much it. There’s one last potential gotchya and that’s to do with organisational proxies on your SharePoint servers. If the server requires a proxy to access the internet then there’s a fair chance you’re going to end up getting an error page like this (click to expand):

What’s required is to find the proxy line in your web.config and change it to the following:

<system.net>
  <defaultProxy>
    <proxy proxyaddress="http://your-proxy-address:port" bypassonlocal="true" />
  </defaultProxy>
</system.net>

And that should be it! You’re good to go. Pretty basic integration I know, but for most public facing sites this is all you’ll ever need. Dig in to the APIs and see the true power you can get interacting with WordPress through the APIs (anyone feeling like authoring blogs in SharePoint and publishing them to WordPress and vice-versa?).

One last point I should mention – I’ve yet to link in this blog to a man who seems to be a bit of a guru in this stuff – at least it seems that way from the research I did. His name is Joseph Scott and you can view a slidedeck of a presentation on the WordPress APIs here. If you’re scouring the net for decent advice I’d suggest looking out for his name would be a good start.

Follow

Get every new post delivered to your Inbox.