Is This Right

clip_image002Last week, I was talking about accuracy. This week, I learned that there is something more important than accuracy – confidence. We have been working on the design of a SharePoint solution that is both simple and complicated at the same time. Of course, we are concerned about accuracy, but the people using our system will have to feel confident that they have selected the information that they need. This isn’t a challenging SharePoint problem; this is a design issue, a usability issue and ultimately an adoption issue.

Our project involves a contact list. That’s simple, but it’s really about 10 contact lists, and since they need to be exposed on both an internal and an Internet-facing server, it’s really 20 contact lists. The “master” list is a group of people who serve on a variety of committees, and that’s the only list we want to maintain. The people come and go and the committee membership changes from year-to-year. We need to know who is on each committee today, we need to know who was on each committee 2, 3, 5 and perhaps 10 years ago, and we need to know this from multiple angles. For example, I might simply want to be able to contact the members of Committee A. On the other hand, if I’m about to meet one of the people on the list, I might want to know every committee he or she serves on. In yet another scenario, if a person is retiring, I might want to know every committee the person ever served on. We’ve cracked the list design issues, and with the help of HarePoint Workflow Extensions, we’ve cracked how to keep the lists in sync on the Internet-facing server. OK, so what is this confidence issue?

Let’s think about those scenarios again. In the first one, I want to see the names of the people who are currently serving on Committee A. Obviously, I could simply filter the list on a few columns to distill the contents down to right group. That would work, but while most people know how to do this, they don’t want to have to do it every time; that’s what views are for. Views, now this is where we start to get into trouble with usability.

Views are a fantastic feature within SharePoint, but neither lists nor views scale very well. That’s not to say that you can’t put a ton of items in a list, or create a bunch of views, but sooner than later the results are untrustworthy. Once you get a couple of pages of items in a list, you have to resort to sorting or filtering to make it useful. Similarly, once you have 15 or 20 views on a list, the selection is equally unmanageable; in fact, a large selection of views might even be worse than a large collection of items. When I am setting sort and filter parameters, I know what I am doing; the trouble is that I have to be able to imagine the results as I proceed. When I choose a view, I am relying on someone’s ability to name a list intelligently. If I make the user figure out how to get the information they want, I put the burden of accuracy on them and it becomes a confidence issue.

There’s a very fine line between wanting my users to understand SharePoint and forcing them into a situation where they are uncomfortable. When they get to that point, they are going to ask someone else (maybe me) to get them the information they need, and at that point SharePoint and I have failed.

We have to find a way to tame a large list that can be rendered in a large number of ways. I can’t just substitute a bunch of input fields for a search or data view web part, that’s really no different than asking them to configure the raw list. I need to give them links and simple binary selections, coupled with a standard output format so they remain confident that they are getting what they want. Links like “Committee A Members” with a choice for “Current members only” or “Most recent five years” or “2000 – Present”. The output should be simple: Contact name (with a link to details), his or her role on the committee and the company they work for. I would argue that this kind of solution is the poster child for “perfect is the enemy of good enough” – making this process more elaborate, or the making the results fancier will only erode the confidence of the average user.

The simplest contact list that we have ever prepared, is rendered from our annual Policyholder Meeting survey, it’s a dirt-simple list: “Who’s Golfing on Thursday”, and it’s an absolute favorite among my coworkers.

The Market Will Save Us

clip_image002I just finished reading a blog post by Sahil Malik which refers to a blog post by Marc Anderson and other posts and comments on the subject of the soon-to-be-gone-but-not-forgotten design view in SharePoint 2013. I have followed this story from the beginning when Marc was hinting that the design view was missing, up to and now well beyond, the seemingly official death sentence that was published by the Microsoft SharePoint Team on their blog. That the design view is going, going, gone seems certain at this point, but whether that will result in a wave of power users learning the art and craft of development or just skyrocketing demand for existing developers is not so clear. I can only say with clarity what will happen in my little world – nothing.

I should rephrase that “nothing will happen right away” and by “nothing” I mean SharePoint 2013. It won’t land here at least until this time next year because activity is underway that relies on the work that members of my team are performing in SharePoint Designer, using design view. Those projects will be completed and an annual event that depends on some custom Data View Web Parts that we assemble will be supported on SharePoint 2010 as it has for the previous two years. Switching canoes in the middle of this stream isn’t appealing and it won’t happen. This isn’t going to hurt Microsoft, (unless SharePoint 2013 marketing success depends on posts in this blog) because we have SharePoint under Software Assurance.

While people are lining up to prognosticate as to which way companies will go (developer of just plain SharePoint) I have faith that good old marketplace economics will save the day. Somebody will build a tool or an add-on tool for something like Dreamweaver, to once again empower the end-users and designers to push SharePoint beyond the limits of its dumpy box. If that doesn’t happen, then other market forces will bear down on the situation and other platforms may benefit from this decision. Again, I can’t speak for the market, or the broad content management / collaboration community, but if I have to turn developers loose on an ECM solution, I’m going to take a hard look at Nuxeo before I start buying more copies of Visual Studio.

Of course, I probably don’t even have to go that far. Before watching Marc Anderson deftly build a Data View Web Part using j-this, j-that and SPServices, I was using Ruby and Dreamweaver to put the information I wanted in a Web Part. I certainly prefer the approach Marc taught my crew, but I can always revert to other options. In fact, I may not have to revert to Ruby; much of what we display in Data View Web Parts can be rendered by SQL Server Reporting Services, another technology my crew is adept at using. Then again, the market may save the day in other ways. When I was using Ruby, I coupled it with Dreamweaver because someone had written a Dreamweaver extension for working with RHTML. I’m willing to bet that there are way more SharePoint power users than there are Ruby developers so I wouldn’t be surprised to see a solution become available for building the kind of solutions that we continue to need in SharePoint from a different development starting point.

Microsoft may want SharePoint to look like SharePoint, and as I have said before on this blog, I support that to a degree. The more SharePoint solutions share a common look and feel, the easier it is for people using SharePoint to move from one solution to another. Still, sometimes what you need isn’t in the box. When intelligent and enterprising people are confronted by the absence of the things they need, they find them elsewhere or they figure out how to build them. When enough people start building things, vendors bring tools to the market. Maybe Stanley will come out with the Fat Max Editor for SharePoint – I would buy that in a heartbeat.

HarePoint

clip_image002One of the things that I like most about managing a small technology shop is the agility with which we can operate. We are constrained by the usual suspects, i.e. the limits of technology, budgets and time; but most decisions are easy. If we like something, we can look at it. If we like it after we look at it, we can buy it (we usually don’t look at the stuff we can’t afford) and if we have to install something to a server or change a database, we can do that too. Long term readers of this blog know that I like to write about the good people we work with, whether they have developed an add-on product or have rolled up their sleeves and helped us to build something. One of the companies I have wanted to write about is HarePoint, but I wanted to wait until we could use their product to solve one of our most vexing problems – that day has come!

I learned about HarePoint several months ago when I was looking for better ways to work with dates in SharePoint workflows. While poking around their website, I discovered this list of Workflow Extensions. Some of the extensions looked pretty cool, including the ability to work with arrays, and those for manipulating lists, libraries and individual documents and images. By the time I read about the ability to execute a SQL Query from within a workflow, I was starting to drool a little. All of the extensions seemed cool, but one that really piqued our interest was the ability to move a document to a different library. That feature may sound like kid stuff, but it’s not as simple as it appears. We wanted to copy a document, along with some of its metadata from a document library on our internal farm, to a library in a site on our Internet-facing farm. We asked the people at HarePoint if their extensions could do that and they thought that they could. Unfortunately, our first attempt failed.

Some of the best vendors we work with have distinguished themselves when things didn’t work – HarePoint is now a member of that club. We told them what we were trying to do. We told them that it didn’t work, and we anticipated being told that, in retrospect, the feature wasn’t designed to move documents between farms. Instead, the support crew at HarePoint told us that they thought this would be a good feature to have, and they worked with us to make it work. It took a couple of attempts, but last week we were able to create a workflow that moves a document from a library in our engineering site to a library on our Internet-facing SharePoint server.

This isn’t just an interesting technical challenge; this is the final piece of an intricate puzzle that was mostly assembled over a year ago. You can read a ton about that project by searching this blog for ‘inspection’, but the short story goes like this: When our engineers write an inspection report, a series of SharePoint workflows marshal the reports through various reviews, updates a variety of metadata and stores a final copy of the report in a Records Library. The final step was always supposed to have been to create a copy of the report for our customers to access in a SharePoint site we provide for them, but that has remained a manual process – until today. We successfully tested moving one of these inspection reports from our library to a target library on our test farm. The problem seems to have been solved, and the implementation couldn’t be easier:

clip_image004

I want to publically acknowledge the technical support and development groups at HarePoint. The company has a great product, and these people went the extra-mile to make it even better. I’d also like to acknowledge the members of my staff who did battle with the ever present nemesis in SharePoint (permissions) to put these awesome capabilities to work. If you’re looking for ways to extend the functionality of your workflow-driven SharePoint solutions, you might want to take a hard look at HarePoint.

Head Bone Connected to the Neck Bone

imageRemember that song, Dem Bones? All those connections, from toes to head and back again; it was a fun way to think about our skeleton, and as long as I don’t hear my doctor humming it, it’s a fond memory. As it turns out, connections have continued to play an important role in my life. With respect to my skeleton, it’s the connections that caused the problems that sent me to physical therapy. With respect to people, it’s the connections that allow me to enjoy life, and get my job done. With respect to SharePoint, well, sometimes connections in SharePoint help us do remarkable things, and sometimes they send me looking for therapy.

Recently, we have been experimenting with the various ways we can connect SharePoint to the structured data in our SQL Server. There are two primary reasons for wanting to exploit this capability. One, we can enhance the value of our content-centric applications on SharePoint when we can add bits of data to the picture. Two, sometimes, processes in SharePoint grow until they reach the threshold of being an application; nudging them over that threshold saves us the effort of developing another application. Here are two examples of the connections we are working on right now, and why:

One of our most complex content-centric applications is the solution that helps our engineers manage the documents associated with their loss-control inspections. The solution relies on a few supporting custom lists. For example, one list includes, among other things, the URL of the document library in the customer’s site on our Internet-facing server so we know where to put their copy of the report. Recently, a manager in our underwriting department asked if the underwriters can be notified when inspection reports are distributed. Of course, they could set alerts on these libraries, but they only want to know when one document reaches its final stage in the process. Since there is a workflow running, and a lookup-list, we could easily add the underwriter to that list and shoot him an email. Then again, underwriter assignments change, and those assignments are already being maintained in our policy rating system. Instead of manually duplicating that information in the look-up list, the workflow, with a little help from a function library that we are testing, can look up the underwriter in the rating system. If this library doesn’t work, the workflow can look up the underwriter in an External List rendered from the table in question; that would be duplicate data but not duplicate work, SharePoint would be doing the heavy lifting.

The second example features one of our earliest (and ongoing) workflow driven success stories, a time tracking solution that our attorneys use. Our first step was to automate the time entry, which relies on a series of related custom lists. For example, we use a custom list to let the attorneys identify which projects are active. By toggling that status, the pick lists in the entry forms are modified to only show valid choices. The evolution of this solution included a hand-crafted set of Data View Webparts to let the attorneys enter their time from their iPads and integrated SSRS reports to provide the summary data to the accountants. Of course, the accountants need to turn that summary data into journal entries. The time-honored method has been to read the report and type the entries into our General Ledger system. Starting next month, SharePoint will be making those journal entries, courtesy of some fancy script work by Marc Anderson, aided by the woman on my team who has guided this solution through its various stages. The last stage was not without a certain amount of drama. We ran into trouble trying to update an External List from the script. Perhaps the capability doesn’t exist, perhaps the feature is broken, perhaps we still didn’t manage to get the permissions right. However we can update the External List from a workflow. So, Marc put his journal entries into a SharePoint list and a workflow puts them into SQL Server via the External List. It sounds like a kludge, but it works well and it doesn’t require any additional input during the process. I do hope that at some point, Marc describes the details behind his middle-tier magic over on his blog; I’ll leave it at “it works, and we love it!

Web Parts, Data sources, External Lists, related SharePoint lists; it’s not as easy as “the toe bone connecting to the foot bone”, but it’s actually not that hard. As I mentioned last week, we keep these solutions moving forward by taking small steps – sometimes we fail and learn, but sometimes we succeed.

Letting SharePoint Shine – Part 1

clip_image002We have been spending quite a bit of time lately working with the various ways that SharePoint can interact with a SQL Server for displaying, adding and editing rows of structured data. Initially, we were hoping to extend SharePoint’s reach and save ourselves some time by quickly developing SharePoint-based solutions for tasks that would otherwise require a hand-crafted fat-client application. After we began studying these capabilities, two trends emerged: one familiar and one unexpected.

The familiar trend involved a series of frustrating setbacks – things that should have worked simply were not working. When I say “should have worked,” I mean things that involve documented features of SharePoint, External Lists and Business Connectivity Services. When I cay “familiar,” I am referring to errors which are difficult to determine, debug and correct but which somehow have something to do with permissions. While I can’t say for sure, I’m guessing that the demonstrations I’ve seen of these features involved a solution running with the permissions that most of us would never grant to an end-user. Two other members of my team have decided to slog through these issues, so I embarked on the unexpected journey.

Since SQL Server access from a Data View Webpart using a Data source works well, I decided to proceed with our effort to replace an aging payment system. The current system is a fat-client deal that stores information about vendors, allows for the creation of check requests and, once approved, feeds a system that prints checks and updates various accounts. The process is manual, up until the point of inputting the transaction data necessary to produce the request and print the check. In addition, the approval is manual (i.e. go fetch signature), the current storage of the supporting material is analog, and just about every step of the process requires the person performing that step to be in our office. We knew we could improve this process, we were planning to improve this process, but I don’t think we were planning to improve it enough. I don’t think we were planning to give SharePoint a chance to shine.

When I drew this process out on a clean slate, it became clear that only a small amount of the data involved actually needs to be stored in SQL Server. Obviously, we need to know how much money was involved, who the money went to and what account the transaction should be charged to. The other data; the vendor information, the approval chain, the information that would support analysis and help people prepare budgets, none of that needs to be locked up with the transactions. We needed to consider what we could do if we kept that information in SharePoint. For starters, we could handle the entire human part of the transaction in SharePoint. Maintaining vendors, entering payment requests, allocating the requests to the appropriate accounts, submitting, reviewing and approving can all be performed against SharePoint list items. The good news is that if we build that process in SharePoint, we can start, review, submit and approve it from anywhere. In addition, the Data View Webparts we would use to drive this process run well and look great on our iPads, in fact, they run well and look pretty good on my iPhone. Even better, if we store that data in SharePoint lists, people can review it themselves, without the aid of someone who can write SQL or who has access the back-end server. Of course we have to take care of some details, but I think we can handle those too. As far as I can see, we have to:

  • Control access to the process
  • Upload, attach and store invoices and supporting documents
  • Select vendors, departments and valid accounts from predetermined sources
  • Provide a mechanism for requests to be submitted, approved (or rejected) and perhaps approved (or rejected) by a second person
  • Notify the appropriate person (people) when action is required
  • Prevent duplicate processing and/or alterations after processing has occurred
  • Perform some basic math and alert people to errors
  • Write specific bits of valid and approved transactions into the backend SQL Server

We can do all of those, we can do them in SharePoint, we can script it all from within SharePoint Designer, and we can give our users an experience they have never enjoyed with that fat-client solution. We can personalize the page for them. We can show them open items, let them review recent items or historic items. We can let them sort and filter those items by vendor, by date, by account and we can track the process of those open items if they like. In other words, we can leave the information people want in a place where they can easily gain access to it, and we can move the critical data where it belongs. At this point, we are working on proof-of-concept test pages. Things are going well, but suffice it to say, this topic begs a Part-2.

Taking the Next Step

imageIf you follow this blog, or even if you drop in from time to time, you know two things about our SharePoint implementation – we have had some success, and much work remains. Given that situation, this might not seem like the time to be more aggressive with our SharePoint goals, but I think it is. I want to build on the success we have had; I want to extend our reach into other areas of our business, while also expanding the number of ways that we can take advantage of SharePoint.

Lately, we have been working to find something in between the boring out-of-the-box look and feel of SharePoint and a totally custom solution. We want to avoid recreating, as Owen Baern said: “The Problem SharePoint Solved”, but we are trying to give our users the experience they deserve. I am also trying to save time and money by extending SharePoint into the realm previously dominated by fat-client desktop applications. During the past 10 months, we have had some good success building off the lessons we learned during a training session last year with Marc D. Anderson, but as the title suggests, it was time to move the bar. Last year, we learned a lot about working with Data View Web Parts and script, and we used that knowledge to build a couple of very cool summary analysis pages (dashboards if you prefer buzzwords). This year, we wanted to get started with Marc’s SPServices library, so we had him take us to school again.

I have seen a large number of shout-out’s and thank you’s to Marc on Twitter about SPServices, but we really hadn’t come up with a reason to use it, until a couple of months ago. Having split our Internet-facing server into several site collections, we wanted a way to intelligently share items from a master list at the root level of the farm. The list contains employee contact information, including relational items so we can assign them to the sites we insure, the committees of our Board of Directors, and the business partners we deal with. All of these assignments are made by role, so an insured knows who the underwriter is, who the engineer(s) are and who to call for technical support. We have done this in the past with a separate master list in each site collection, using views or DVWPs, but we wanted a single list, so the people who maintain the contact information didn’t have to bounce around between sites. We also wanted the list to be easy to load on each/any page, and we wanted it to look good.

Marc quickly put us on the road to success, with a simple extract of data from the master list, displayed in a Content Editor Webpart that had that familiar SharePoint look and feel. We were impressed, but then Marc put us on the highway, making the list look a little better and driving its contents dynamically from the page it was on. I dropped that link on a test page, in a different site collection, and it sprang to life with the appropriate data. Next, Marc put us on a Trans-AM course, when he used his library’s feature to return a JSON array, so that we could group the results in a way that made sense to our customers, as opposed to the way they were returned by the query. image

Some of our insureds operate multiple reactor sites and there may be numerous underwriters and/or engineers assigned to the account. In those cases, we wanted the account staff to be grouped by insured site, but we wanted the technical staff only listed once. For single unit locations, we wanted the team listed without any grouping or at least with the single group opening in an expanded state.

As Marc worked his middle-tier magic, we got the clear impression that he could put us on a Formula-1 course if we wanted to go that far – we didn’t. We want a good user experience, but we still wanted our users to know they were in SharePoint. This is the point when I am glad we teamed up with Marc. He didn’t come here with a solution in search of a problem; he didn’t walk in trying to sell us the same approach he used at his last client, or his favorite approach, in fact, he ended up trying something new. He came to teach, and he taught us how to do what we wanted to be able to do. If we ever need help building a solution, he’s the first person I would call. This time around it was a learning event, and it was a total success.

What She Said

clip_image002What if SharePoint had a party and nobody came? This is a little more than a silly hypothetical question. After we upgraded to SharePoint 2010, we noticed a problem with the main page of our internal server. Our attempts to fix this problem have been frustrated by a reference to the old server that we know should not be there. I am convinced that this is the cumulative effect of our migratory path from a 2003 WSS site to SharePoint Portal Server, then to SP 2007 and now to SP 2010. We have never paid much attention to the main page, focusing more on the sub-sites. Along the way from 2003 WSS to our current mix of Farms and Sites, we have only made the most minor attempts to draw people into the front room, as it were.

This past week, our attempts to fix this page resulted in an error when the page loads. We reset the page layout several times, returning us to the recalcitrant home page, and another shot at its repair. On one attempt, we created an error situation that we seemingly cannot return from. My Systems Administrator instantly offered up options ranging from restoring from our robust backup options to hacking bits and pieces of the file structure, while I am advocating starting over. As we battle what seems to be the Hydra of SharePoint, our users have been surprisingly quiet. Are they being kind and understanding, or do they simply not care that the homepage is gone? Perhaps they haven’t even noticed…

Our users own, and use the sub sites. They come to SharePoint to accomplish tasks, and they usually come by way of a link to the specific page they need to visit. We have tried to put useful information on the front page, in an attempt to route them through our content on the way to theirs, but we have met with only limited success. Probably the most popular home page attribute is the Daily Dilbert. I can’t help but be reminded of a family clash of culture from my childhood:

One day, my father asked my mother if she wanted to “go to the mall” – this was like asking a politician if he wants money. My father explained that he needed a saw blade, my mother didn’t care, and she didn’t require a reason to shop. We drove to the mall, parked near and entered through Sears’ tool department. My father purchased his saw blade and we promptly left. Despite my mother’s protests, we ignored the rest of Sears, not to mention the other 101 stores.

I sided with my father back then, and I’m not sure I don’t side with our users today. Do I really care if they visit the main page? I don’t think I do.

Of course, we will fix the main page, but I am taking specific guidance from the muted reaction of our users to its absence. First, when we get it back up, I am going to focus on getting people from there to their ultimate destination as fast and as easily as possible. Second, I am going to make sure that the stuff I want them to see on the main page is visible throughout the sub-sites. In other words, I’m going to do what I have always advocated, I am going to “give ‘em what they want.”