Jake Kappus.comCertified Geek, Dad, MMA fan, and ocassional Pro Wrestler

You may be tempted to wing it
Use a hardcoded link submit it
But performance will suffer
When you’re left to your druthers
Should have Autodiscovered
Then all would be well
The Autodiscover Song
http://blogs.technet.com/b/exchange/archive/2008/08/08/3406026.aspx

I usually don’t like to write how-to articles.  Don’t get me wrong, I love helping people and that’s why I love consulting, but I leave the how-to articles to my friends Scott Feltmann, Elan Shudnow and Jeff Guillet.   Me on the other hand, am an entertainer, always have been.   I like to hit the topics that no one usually wants to touch.  Like our friend the Exchange Autodiscover service.

Over the past few months, I’ve been called in by several clients to clean up the work of other “Exchange experts” who underbid me (well, the company I work for), and then pass off the work as a job well done.   Then, when they leave and move on to their next work of art, the clients call me back and ask them to fix the laundry list of “out of scope” issues the “expert” left behind.

More times than most, one thing fixes 90% of these issues.  Sing it with me kids….A-U-T-O-D-I-S-C-O-V-E-R.

What Does Autodiscover do?

Why, I’m so glad you asked!    Here is the official Technet description of the service.  (Yes, I am the MASTER of cut and paste!)

The Autodiscover service does the following:

• Automatically configures user profile settings for clients running Microsoft Office Outlook 2007 or Outlook 2010, as well as supported mobile phones. Phones running Windows Mobile 6.1 or a later version are supported. If your phone isn’t a Windows Mobile phone, check your mobile phone documentation to see if it’s supported.
• Provides access to Exchange features for Outlook 2007 or Outlook 2010 clients that are connected to your Exchange messaging environment.
• Uses a user’s e-mail address and password to provide profile settings to Outlook 2007 or Outlook 2010 clients and supported mobile phones. If the Outlook client is joined to a domain, the user’s domain account is used.

Now, that’s pretty self-explanatory, right?   From what I’m seeing in the field, I think most people are reading the first sentence and clicking the next link.  (…and I thought I had a severe case of ADD!)

AUTODISCOVER DOES MUCH MORE THAN PROFILE CREATION PEOPLE!

Let’s read the second bullet, you know, the one they didn’t read!

• Provides access to Exchange features for Outlook 2007 or Outlook 2010 clients that are connected to your Exchange messaging environment.

Now, first a little bit of side info that’s pretty valuable for this topic.  Exchange 2007 and 2010 uses Exchange Web Services (EWS) to serve clients like Outlook 2007, 2010, (2011 for you Mac users), OWA, ActiveSync, and even that waste of silicon the Blackberry.

No really, I may end up voting Republican if Obama doesn’t ditch the BB and get a real phone.

Outlook 2000 or 2003 does not use the Autodiscover service.  But that doesn’t mean you shouldn’t deploy it if you use those clients exclusively.  Remember those ActiveSync clients, they need some Autodiscover love too!

If you are still on Office 98 call me, please. I’ll steal some 2007 or 2010 licenses for you somehow.  (Kidding!)

With EWS, these clients get the much improved Availability Service (Free/Busy for those of you still suck on 2003), Out of Office, MailTips, Offline Address Book and even access to the new, and seriously cool Exchange Control Panel, securely and efficiently! (read: No WebDav!)

Besides, when you take advantage of that new kick-ass cross-site DAG you just implemented, how are your Outlook clients going to find their mailbox after your basement datacenter just took in several hundred gallons of the Minnesota River, and you (thankfully) failed over to the backup datacenter?  Magic?  A quick desktop support staff?  Nope, Autodiscover.

So here’s how it works:

When you first setup the Outlook or ActiveSync client internally from a domain joined client, the client will query the Autodiscover Service Connection Point for the list of CAS servers. The client sends an HTTP POST command to the Autodiscover service. This command includes XML data that requests the connection settings and URLs for the Exchange services that are associated with the Outlook provider. The service will gather the locations of the different Exchange Web Services (EWS) from Active Directory which will then present the client with the results in an HTTP request formatted in XML.

What if something changes or breaks?

The Outlook client queries Autodiscover periodically (1 hour).  If there are changes in the Exchange architecture, Autodiscover will change the profile automatically. If Outlook fails to connect to the Exchange server, it will attempt to connect to the URL’s it was presented every five minutes.

Now when the underlying network layer disconnects, after the first initial reconnection, Outlook’s MAPI layer will attempt to query Autodiscover every six hours.

Note:  For the client changes to take effect, you still need to restart the client or do a manual “Repair” under Account Settings.

The Service Connection Point

Let’s talk about the SCP Connection point.  This is AUTOMAGICALLY created by setup when you install the Client Access Role.   Setup will register this connection point under the CN=Services, CN=Microsoft Exchange, CN=Administrative Groups, CN=”AG NAME”, CN=Servers, CN=Servername, CN=Protocols, CN=Autodiscover container.  The Internal domain joined Outlook client will query AD for these connection points.  AD will return a list (one for every CAS server in the organization and another set of lists with the CAS servers in your site and outside your site), and will usually take the first one in the closest or same AD Site on the list.

Now before you go all ADSIEdit on your NTDS.dit, hold yer horses!  You do realize you have options that do not involve an authoritative restore after you accidentally bump the delete key on the root container, right?   In the Exchange Management Shell (oh come on, Powershell isn’t that bad!), you can use your carefully planned out Autodiscover URL to set the Autodiscover Internal URI value.

Set-ClientAccessServer -Identity “CAS-01″ -AutoDiscoverServiceInternalUri https://cas01.contoso.com/autodiscover/autodiscover.xml

Do this on every CAS server in your organization and it will return a properly formatted list to you every time.   Well, maybe.  Unless you neglected to plan for Autodiscover properly and did not include the server (or another hostname like Autodiscover) name on your brand new and very carefully crafted SAN certificate.

If this happens, you get the dreaded certificate error!  Plan, plan, plan, I say!

Certificates are another thing that gets a lot of “experts” into trouble.    Microsoft recommends using a SAN (Subject Alternative Name) Certificate, also called a UC (Unified Communications) Certificate.    A lot of them try get away with using a single name certificate, but it really is more trouble than it is worth.  I never recommend their use.

Make sure whatever name you use for your Autodiscover url, there is a valid certificate imported and assigned in Exchange with that name in the SAN field!

The names Microsoft recommends are:

CN (Common Name): *owaname*. Domain.com
SAN: Autodiscover.domain.com
SAN: legacy.domain.com (for legacy coexistience to Exchange 2003, if needed)

Note:  Personally, I like to add the local CAS server names into the cert as well, just to cover for any static client issues or testing.  It is not required though by any means.  With properly configured URL’s, you do not need to add them.  It’s just my personal preference after some late nights of troubleshooting some wacky environments. YMMV J

I could go much deeper into this subject, but I’m out of…ahem, beverages.   In Part II of this thrilling series on Autodiscover, I will go into external Autodiscover configuration, OCS/Lync dependencies, and how your phone finds Exchange from anywhere in the world.

For more information:

White Paper: Exchange 2007 Autodiscover Service
http://technet.microsoft.com/en-us/library/bb332063%28EXCHG.80%29.aspx

Understanding the Autodiscover Service: Exchange 2010 SP1
http://technet.microsoft.com/en-us/library/bb124251.aspx

And if you’re really geeky:
[MS-OXDISCO]: Autodiscover HTTP Service Protocol Specification
http://msdn.microsoft.com/en-us/library/cc433481%28EXCHG.80%29.aspx

Yesterday’s definition update for Forefront Endpoint Protection and Security Essentials apparently wasn’t tested by Microsoft as the engines flagged a temp file in the update as Adware:Win32/Hotbar. I experienced this as well, but just removed after some investigation, I just let FEP remove the file and all was fine.  It was a temp file anyway and the definition was already installed and fine as is.

Before going to bed I checked, and sure enough at 9:30pm CDT, the updated definition files were installed by Windows Update.  I could see where this could be a problem for large FEP environments though.  I would think though that you can use SCCM to suppress the warnings for the user and then just deal with the reports and alerts from there.

This was posted on the Microsoft Malware Protection Center:

On Jul 21, 2011 05:35 AM UTC, an update caused a temporary file related to the next Microsoft definition update to be incorrectly detected as Adware:Win32/Hotbar. At 08:51 PM UTC, Microsoft released a new signature to address the issue. Signature versions 1.109.92.0 and higher include this fix.

I’m hearing a lot of upset customers screaming at Microsoft for this.   Really?  No sense in getting wound up over a simple mistake.  It wasn’t even a virus, but just a typo in signature file that was caught.  Things happen, people make mistakes.  Exchange volumes were not formatted twice over because of it… move on.

I think… hmmm… client didn’t install any of the roll-ups since SP1 .   So I look at Windows Update’s Check Updates screen.

Windows Update says:

Google time!   Microsoft says this is the build number for UR2:

I knew at this point something was up.  Then I remembered a post I read on Technet a while back, but never bothered to file back into the inner caverns of my brain and decided to try another method.  I knew that this command:

GCM exsetup |%{$_.Fileversioninfo}

Would pull the correct file version number from AD and voila!  You get the correct build number.

Sneaky bastages!

Now what bothers me is that Microsoft still mentions the incorrect way in the Technet article:

http://technet.microsoft.com/en-us/library/hh135098.aspx

But then at the very bottom:

Why do they make this so difficult for us?

Didn’t think I would make it this year, but I made it to TechEd 2011 in Atlanta. This year I even smartened up by not lugging my beast of a laptop and dealing with a four hour battery life. This year I’ve armed myself with a shiny new iPad! Not to mention, I’ve finally gone with just a carryon instead of checking luggage, but that’s another post.

This morning I’m sitting with my coworker Scott Feltmann taking in the Mastering Exchange 2010 High Availability pre-conference seminar. A little early on a Sunday to be talking Exchange, but hey, it’s TechEd!

Now I just need to find some good southern BBQ here in Hotlanta. How many miles is it to Fat Boys in Montgomery AL? The south’s best BBQ!

I digress, again….

On my way home from Chicago this weekend I was scanning the radio for things to listen to.  I was almost to Eau Claire when I hit WCCO in Minneapolis and they were airing a call-in show that had a local “computer expert” talking and helping people with their home computer issues.

I normally just take these shows with a grain of salt and move on, but this guy really struck a nerve with me for some reason.   He did have some good recommendations, for example he recommended to a caller, as do I, to use  Microsoft Security Essentials for home anti-virus.   However, I’m pretty sure this guy owns stock in Google because every other word out of his mouth was promoting some Google app.

His first recommendation was for everyone to organize their files into folders (which I agree with), but he said emphatically that this would “greatly speed up your computer”.

Really?  Let’s debunk that one right now.

Sure, you’ll be able to find that picture that your camera named “pic20382172934.jpg” easier by organizing them in folders (which Windows Photo Viewer does by default anyway), but with a little research about NTFS, which is the file system that Windows XP and newer uses by default, you’ll know that it does not significantly improve disk or Windows performance.

NTFS uses a relational database to keep track of where files and folders on your disk.  When you create, delete, rename, move, whatever, a file, it just updates the entry in the MFT (Master File Table).

First, a little info on how disks actually store your data.  When you create a new Word document called “Document1.docx” , hopefully if there is room, Windows will try to write the file intact to a place on the hard disk (or multiple disks if you use RAID), if not, it will fragment it and squeeze it into places where it can fit the pieces into.  NTFS will document the locations on the disk where the file pieces are located in the MFT.  It’s much more complicated than this, but I’m trying to keep this article on level 100 here.

Now let’s draw this out a little bit:  (Assume we are on Windows XP)

When you access filename.doc in C:\Documents and Settings\Jake\My Documents (wait, that’s a pretty good folder structure right there!) , Windows will look for the location(s) of the file in the MFT and begin the process of putting the pieces back together in RAM (physical or virtual) and present to to the application (Word).

If I put that file into a folder, all that is happening is the entry in the MFT is being updated.  And technically, if you consider there are change logs and junction points to consider in all of this, if you were to suddenly reorganize that 1000 picture gallery into photos in a weekend, technically you could slow Windows down for a little while until the Change Journal fills up and then purges, or even while the indexing service catches up with your changes.

But, I suppose it will feel faster if you sort those files into a pretty folder structure.  I’m saying you shouldn’t organize your files, because you should just for your own sanity and others who may access your systems.  But to think that it will in any way speed things up is ludicrous.

His second recommendation was to move everything to the cloud, INCLUDING private personal financial data.  He went as far as recommending a service that I had never heard of just because “they encrypt your data”.

As the “Awesome” one says.  So, this expert is telling people to upload your budgets, or maybe the doc about your 401K, or maybe even the files with your passwords to your credit card websites, to “the Cloud”.  I’m down with the cloud for some things.  Music, video, gaming.  But like hell can I ever recommend you put your ultra personal files to someone else’s datacenter in god knows what country.   It’s the same idea as me collecting the copies of your birth certificates and saying “they’ll be fine here in my bottom drawer… I lock it.”   If you trust that, then I’ve got a new business idea!

Did this guy ever bother to read the Terms of Use for Google Docs?

4.1 Google has subsidiaries and affiliated legal entities around the world (“Subsidiaries and Affiliates”). Sometimes, these companies will be providing the Services to you on behalf of Google itself. You acknowledge and agree that Subsidiaries and Affiliates will be entitled to provide the Services to you.

4.3 As part of this continuing innovation, you acknowledge and agree that Google may stop (permanently or temporarily) providing the Services (or any features within the Services) to you or to users generally at Google’s sole discretion, without prior notice to you. You may stop using the Services at any time. You do not need to specifically inform Google when you stop using the Services.

4.4 You acknowledge and agree that if Google disables access to your account, you may be prevented from accessing the Services, your account details or any files or other content which is contained in your account.

And, if you sign up now, you also get:

1.3 You understand that Google, in performing the required technical steps to provide the Services to our users, may (a) transmit or distribute your Content over various public networks and in various media; and (b) make such changes to your Content as are necessary to conform and adapt that Content to the technical requirements of connecting networks, devices, services or media. You agree that this license shall permit Google to take these actions.

So, not only can Google take away your access to your docs, but they can also transmit them over public networks and make any changes they want to it.

Now… how’s that cloud looking now?    His idea of moving all of your pics to Picasa was equally as short sighted.  With the Feds having access to Google’s datacenter, do you really want those bathtub pics of the kids (come on, we all have them) up where someone could take those pics the wrong way and suddenly your in deep doo doo over them.  (And yes, it’s safe to click that link, it’s not pics of my kids in the bathtub.  It’s a link to an ABC News story about a family that was terrorized by Child Protective Services over bathtub photos. )

I’m not saying don’t use Google Docs or that Google is evil because I use their services too.  But I keep my important data in places that I always have access to.   I use Google Docs for quick access to some things, but nothing involving personal data.

My point in all of this is that with technology, no matter what it is, is so deep an wide.  No one knows everything about anything in this field and it takes a lot of work to get good at a very very small subset of this technology thing.  A lot of people will project themselves as experts or professionals, and either know just enough to be dangerous, or really will be an expert.  Unfortunately, there are FAR more of the dangerous type out there than the true experts.

How do you weed the dangerous clowns out?   Get multiple opinions, check credentials, look to see if the person quotes official  documentation and best practices when making a recommendation.  If they tell you “trust me, I’ve been doing this for X years”, run away.  Look for someone who has done it for X amount of years AND has the credentials and documentation behind them to prove it.     And don’t believe everything you hear in the media or god forbid Facebook.  If I hear another virus hoax or viral post about using HTTPS on Facebook will prevent someone from hacking your account, I’ll go Stone Cold Steve Austin on everyone.

Cost of Exchange Consultant to help attempt to recover your database = $124,000

Cost for Kroll OnTrack to write custom software to recover your damaged database = $120,000

Reliable and verified Exchange aware backups: Nice

Implementing high availability using best practices: Priceless

I laugh when people rejoice about the iPhone coming to the Verizon network.  I usually just think, good, now they can feel the pain that us AT&T users have been feeling for years.  And maybe it will take some of the strain off of AT&T’s network when we finally get HSPA+.  I have an iPhone and haven’t had an issue yet other than in high concentrated areas of cellular users (TechEd in New Orleans or in Manhattan).   I will never go back to an Android phone either, but that’s another blog post.

This article goes in depth into why the Verizon iPhone offering may not be as cool as everyone thinks…

Verizon’s iPhone Full of Tradeoffs, Good and Bad

.

For some reason I woke up this Saturday morning with a crazy idea to simplify my work life.  I have no idea why I think of these things, I mean why can’t I wake up and think of ways to simplify my home life?  Anyway…  I started writing a Powershell script to automatically install all of the pre-reqs required to install Exchange 2010 SP1 on Windows 2008 and Windows 2008 R2.  Then I remembered that I am a Powershell n00b.  After some research I found out that someone had already thought of this and I was really wasting my time.  I mean really, who gets up on Saturday morning and decides to write Powershell scripts for the hell of it?

The folks over on the Microsoft UC blog already wrote a pretty nice script that I’m definitely adding to my collection.  It even includes installing the Office 2010 Filter Pack, the iPDF filter and will even set the NET TCP Port Sharing service to Automatic for you.  And, if that wasn’t enough, if you act now it will even disable IPv6 the right way for you!

http://www.ucblogs.net/blogs/exchange/archive/2009/12/12/Automated-prerequisite-installation-via-PowerShell-for-Exchange-Server-2010-on-Windows-Server-2008-R2.aspx

That, combined with a script to automagically register the Filter Pack iFilters, makes for a lot of saved cycles that could be spent playing Angry Birds on your iPhone.

Grab that script here:  http://technet.microsoft.com/en-us/library/ee732397%28EXCHG.140%29.aspx

Now, I just need to find a script to install the required updates for Exchange 2010 Sp1 so I can go find a real hobby!

Ran into this one this morning and it made me think a bit.   This particular Exchange install has had it’s share of oddities, random reboots, WMI crashes, permissions automagically changing overnight and other fun issues that make for some late nights.  I blame a certain firm’s monitoring software for most of the problems, but I won’t drop any names. 

I digress…as usual.

This customer has a two node DAG in one datacenter (for now, another DR datacenter to come in a few months).  They have 20 db’s with 10 to be active on one with the other 10 to be passive on the other.  Nothing too crazy here, right?

Check this out:

Normally, I would just blame that on EMC wackiness and just bounce IIS, or close and reopen the EMC.  But the Shell confirmed this to be actually true!

To fix this, I went into ADSIedit and found my way to the database in question.  There I found the active copy (ATL02MB02), the passive copy (ATL02MB01) and another with ATL02MB01 followed by a strange series of characters which I assumed was a GUID.  I simply removed the misbehaving entry followed by a mount /remount of the active database.  This looked to solve our issue.

In perusing the event logs, I found that one of the LUNs that the database was attached to had been disconnected/reconnected for some reason (this kind of wackiness has been happening a lot lately).   I’m going to assume that possibly when I created the database copy on MB01, that the seeding didn’t finish properly or was interrupted by the LUN disconnecting/reconnecting, thus causing the diverged database.  I did notice a SeedDivergenceCheck file sitting in the database directory before I remounted the database.

I’m sure there will be more blog posts coming from this project.

Still trying to figure out how Exchange 2003 with a valid SMTP connector tried to route a message destined for the connector’s specified address space through both the connector and out through the internet connector.  The user got both a sent receipt AND an NDR, causing all sorts of mass confusion.   That one was a 14 hour ordeal that ended with a 6 hour call with PSS.  The fix?  Blow away the connectors, routing group and recipient group for the domain in question.  Did I mention that the Exchange 2003/Windows 2003 R2 server had not been rebooted in over 2 years?

Whatever…

I’ve all but given up on the great state of Wisconsin.  Don’t get me wrong, this is my home and I love this state, I really do.  From the good ‘ol Green Bay Packers, to Leinenkugels beer, to the beautiful scenery north of highway 29, I can’t think of a better place to live and raise a family in.

But there are no jobs here!

The Wausau area seemed to be booming about four or five years ago.  Industry was growing and I was busy as an IT consultant could be.  Every consulting firm in the area wanted a piece of this area.  Heartland moved in, CDW was a strong presence already, and even non-local players like Inacom, Tushaus, Logicalis, and Camera Corner all had or was attempting to make a presence here.  Me and my colleagues were doing all we could to fight off the recruiting calls from all of these companies and more, trying to build a customer base from our already existing relationships in the area.

Then sometime in about early 2009 it just died.  Blame the economy, blame competition, blame Bush, blame Obama, blame someone.  But the Wausau market just dried up.  Big industries like Greenheck, Wausau Window and Wall, Footlocker stopped hiring, laid off workers and stopped doing upgrades or replacements with consultants like me.  IT budgets froze, and so did the local consulting market.  Market values for a network administrator dropped along with those budgets.

I once interviewed with a great real estate investment firm in Northern Wisconsin.  They had been one of my best customers for years.  I watched them start out in a small office with 10 people and turn into a company with over 100 employees in six states.  Their financials were very very good.  They were a very liquid company with little to no debt and millions in profit every year since they started.  Their infrastructure had grown out of their small broom closet into over 20 servers, hub and spoke WAN to their 6 locations, Exchange, Cisco Wireless, Voice and Firewalls.  I had implemented Microsoft OCS  and an advanced Microsoft Dynamics CRM and Accounting system.  When it came time to hire a network admin to manage all of this, they called me first.  I was of course open to the idea of managing this infrastructure due to my great relationship with them and well, I had built most of this environment almost from the ground up.  It was a perfect fit.  Then we talked money.  Turns out they wanted to pay only about 40K per year to manage all of this.  Their actual job description and newspaper ad specifically required the CCNA, CCVP, MCSE, MCTS: OCS and a CWNA certifications for the position with a minimum of 5 years experience, all for 40K.  There was nothing I could do but shake my head and wish them luck.   They ended up with an entry level guy who had none of these certifications and/or experience, and I think they may be going out of business as of this writing.

With no major projects in the pipeline, the big non-local firms all either pulled out of Wausau or dramatically reduced their staff and footprint on the area.  Heartland, CDW, and RMM Solutions are prime examples of that.  I unfortunately worked for all three of them and saw it all first hand.  I was lucky with CDW and got out before things got crazy there, but not so lucky with Heartland and RMM.  I don’t blame either of them for the bloodletting (though their methods and communications could have been done with a little more morality in my honest opinion), but when the area does again pick up some steam, these firms will again experience difficulty finding advanced talent to design and implement these solutions.

So what’s a senior level Cisco/Microsoft/VMWare guy making good money as a consultant to do?  No one is hiring, and no one is paying the salaries that we have worked so hard to advance to. Sure, there is Green Bay, the Fox Valley and Milwaukee or Madison.  But what’s to say that the market is any better over there?  I haven’t seen any improvement in those cities either.  Madison probably is the best of the bunch.  Ministry, Marshfield Clinic and Aspirus are hiring, if you like working for lower pay and high blood pressure.  There’s a reason they have such good medical benefits!  So I’m stuck flying around the world doing Exchange and Active Directory migrations with a Quest vendor.  Not exactly the best scenario for a family of eight.

So, after a lot of discussions with my family and friends, we’ve decided to make our way back to Minnesota.  I just feel that if something were to happen in my current position, I have a better chance of landing something in the Minneapolis/St Paul area than anywhere in Wisconsin.  I never again want to experience the feeling of working your tail off and being let go due to something completely out of your control, and then having no options other than being a road warrior or putting my tail between my legs and working at one of the hospitals.   I want that control again over my career and be able to get back on the ladder towards my goals.  Moving to a better market will be the first step.

Kaylee and I will head out first with the family following after the next school year.  On a personal note, this has been such a difficult year for us on so many levels.  The job situation was just a small sliver of what was a horrible, horrible 2010.  Here’s looking at 2011 and things going positive for us again. 1/1/11 can not come fast enough for us.

Finally, I promise to never, ever become or let my children become Vikings fans.  No matter how many former Packers migrate over here to this clusterfuck of a football team.  The Twins, Gophers and Wild are a different story though.