Prevent Extended Exchange Outages

Small problems can turn into large ones in Exchange environments that aren’t regularly monitored, causing the system to deteriorate quickly to the point of outage or even total failure.

There are a few areas to watch to prevent outages in single-sever Exchange environments and large enterprises with multiple servers. Here are three of the most common causes of extended Exchange outages.

Failed/Incomplete Disaster Recovery Plan

A failed recovery is the most common cause of extended Exchange outages. It may sound like circular logic, but if the Exchange environment is down for multiple days, the root cause of the failure is no longer relevant. Don’t let indecision and untested processes fuel a crisis. Every Exchange shop needs to have a detailed plan to recover each of the following: single mailbox, single database, single server and the entire environment.

While there are a number of third-party products that handle disaster recovery, tools and processes included with Microsoft Exchange and Windows Server are good options because Microsoft offers support and documentation for different disaster scenarios. Microsoft provides guidance on how to restore the following — a single mailbox from a database backup, an Exchange Server, a DAG Member Server and dial tone portability, which can solve failures of a mailbox database, server or entire site.

Use these procedures regularly to understand the process and test backups. The processes to restore a database and a single mailbox are not invasive; administrators can perform these procedures on live servers. It’s best to perform these on weekends and after hours to minimize the potential effect on end users.

For the rest of the article, please follow this link to my complete article at Tech Target.

 

Reconnected!

Recognition is always appreciated and last month Microsoft honored me by including me in the MVP Reconnect Program. Its a privilege to be included in the group once more and to be reconnected with so many brilliant and talented people again.

dsc00051

20 Years of Exchange

It was an honor and privilege to be included in a small gathering last night celebrating the 20thdsc00024 Anniversary of Microsoft Exchange. The product group, MVPs and the writers/Rock-Stars were all there to commemorate the landmark achievement.

There were folks in that room who gave me opportunities many years ago to publish articles and chapters in books. Because of their trust and support I was able to branch off and write many more articles, to speak at events and to expand my own horizons within the field. In fact as I looked around the room and shook hands I was humbled by the realization of how many of them have helped me over the years.

This event was not about strategizing the future of email, our careers or even the future of Exchange. This gathering was a pure trip down memory lane and an opportunity for us to reacquaint and reconnect. I don’t know if there will be another version of Exchange but I will continue to help the Exchange community as we continue our migration into the cloud.

Office 365 eDiscovery bolsters an admin’s compliance arsenal

Many organizations contemplating a move to Office 365 have major concerns about security and compliance. Some may even abandon plans to move to the cloud because of compliance worries. Many of the legal and regulatory tasks administrators perform with on-premises Exchange Server can now be replicated with the updated Office 365 eDiscovery tool and other compliance utilities.

Microsoft recently shifted the location of Exchange management tools, and it put most of the retention and compliance utilities in the Office 365 Security & Compliance Center. In fact, the Office 365 eDiscovery feature has more power and versatility than similar tools in on-premises Exchange Server 2013 and 2016, and the Office 365 Exchange administrator console.

Last week, TechTarget published my entire article with walk-throughs, screenshots and lost of details. Follow this link for more details: Office 365 eDiscovery

Exchange 2016 upgrade considerations

For more detailed explanations, scripts and recommendations please follow this link to my article on TechTarget.

It’s tough to leave a good thing. Let’s face it: Exchange 2010 was a solid release. Not only did it bring native support for public folders but it also had direct Remote Procedure Call connections with the option of HTTPS, built-in antispam tools and great third-party support for just about anything we wanted — fax, antivirus and even BlackBerry Enterprise Services.

Unfortunately, Exchange 2010 and the organizations that depend on it are on borrowed time. Microsoft ended mainstream support in early 2015 and extended support is not an option for most of us, so it’s time to start planning an upgrade.

There are well-known compatibility and migration issues that can be solved in advance. With good preparation and planning, Exchange administrators can make the upgrade to Exchange 2013 or 2016 practically invisible to end users. If you follow my list of top five items to handle, then the switch should be fairly painless.

5. Load balancing

Load balancing is not an issue for smaller shops but there are specific differences between Exchange 2010 and later versions that need attention, depending on whether you use F5, NetScaler, or some other hardware or software options.

4. DNS namespaces and certificate planning

What we are really talking about is the names used for Outlook Anywhere (OA), Outlook Web App (OWA), Exchange Control Panel (ECP), ActiveSync (AS), EWS, Offline Address Book (OAB) and Autodiscover. It sounds like a lot but most administrators combine the namespaces for many of these.

3. Third-party compatibility

An organization that relies heavily on a particular add-on should talk to its vendor as soon as possible to see if it can offer a transition plan. Vendors that provide fax, compliance, e-discovery, mobile synchronization or security services, antivirus, backup and recovery, unified messaging and other services do not always have a clear path to newer versions of Exchange.

2. Exchange public folders

I have rated public folders so high because I have seen companies struggle with the transition work in this area for more than a year and severely delay the move from Exchange 2010. The process of moving the folders is cumbersome but even more difficult is the effort needed to identify and determine if a folder and its contents can be removed instead of migrated.

1. Exchange clients

Client software gets top billing in this list. Exchange 2013 and 2016 do not support direct RPC connections for MAPI so you will have to use Outlook 2007 or newer. Also, make sure the Outlook clients are patched.

This is the short list from the entire article but hopefully it helps you with your planning.

Exchange Costs : Cloud vs. On-Prem

Moving your Exchange services to Office 365 seems to be a fairly simple decision for smaller companies; it just makes sense. Most of my larger customers however remain on-prem due to security concerns and higher estimated costs. In addition, there are often political challenges in eliminating local servers and eliminating jobs within IT departments and sometimes irrational fears of a global cloud shutdown or problem. (Blackberry/RIM problems come to mind.)

The slower adoption rate of Office 365 for larger entities is a huge topic and I would need some help in covering that completely, but I did find an interesting article today with price comparisons for small shops. It’s the first time I have seen calculations and a nice chart that shows the costs between the two options for various small company sizes. This article suggests that the price advantage with Office 365 starts to wither once you reach 1,000 mailboxes.

Anyway, check it out.

Comparing Cost for Exchange Online to On Premise for Small to Midsized Businesses

 

 

 

Blueprint for an Exchange Service Level Agreement

Introduction

This article is intended to detail the mechanics behind an Service Level Agreement between a fictitious Company ABC and an outsourcing/hosting vendor. You may not have the luxury of getting an SLA with the vendor you have chosen but you may want to think about asking for one. Time to recovery, time to restore and up time guarantees are all just a few of the critical facets of IT we need to manage.

Several specific requirements have been assumed and documented in this article.Feel free to copy this content and edit it for your own use. My goal was to product a template that other companies could use to begin work on their own SLA.

There are two major parts to an SLA: the governing document and the process.

  1. The SLA Document is usually legally binding between a company and an outsourcing vendor(s). The document describes the exact services and service levels, with details about all agreements.
  2. The SLA Process represents the methods that the outsourcing vendor will use to support the SLA document. The methods of supporting the SLA document are usually left to the outsourcing vendor to identify. These processes should be discussed and possibly identified during SLA contract negotiation. It is important that both parties understand the processes and methods of support as well as the management and reporting tools.

The SLA process represents a third of the total solution. It is up to the hosting vendor and your company to ultimately choose the correct people to manage the systems and the best technology for implementation. The people involved in managing the process must also manage the technologies and understand the importance of reporting and monitoring the entire system.

System management and service desk automation technology can provide a supporting environment for tracking, escalation, and management of service metrics. End user satisfaction surveys can also provide input that will help target appropriate service levels and cost controls.

Service Level Agreements are often categorized in the following manner:

  • Basic: A single level service agreement is in place. Metrics are established and measured, possibly requiring manual collection of data for management reporting. Objective is to justify the technical support operation.
  • Medium: The automation of metrics data enables more comprehensive less labor intensive reporting of service level achievement. Introduction of cost recovery that maps to market rates and supported by service level reporting. Possible multi-level service agreements by cost per services rendered. Objective is to match service and cost levels with long term goal to increase service levels while decreasing costs.
  • Advanced: Service levels are embedded in overall service desk processes enabling dynamic allocation of resources either external or internal to meet changing business conditions. Goal is to provide a seamless mix of services, costs and service providers at better than competitive rates. Often enterprises at this level are ready to extend services to the open market.

The Scenario

For purposes of discussion, the remainder of this paper examines the considerations of a company evaluator who must complete an SLA document of the support requirements for Exchange/Outlook systems. We will call this company “ABC Company.” The evaluator works with one or more outsourcing companies to negotiate the final agreements recorded in the document. Recommendations and suggestions, which are based upon industry standards and project management experience, are provided throughout.

Objectives

The primary objective of the SLA document is to correctly identify the support requirements for Company ABC in regards to supporting the Outlook/Exchange infrastructure.

The ABC company evaluator alone cannot determine the appropriate details for the SLA. The outsourcing vendor’s industry experience and project management capabilities will provide required information and guidance. In many cases, it will be required for Company ABC and management within Company ABC to conduct workshops on the issues to determine specific objectives.

Moreover, we should all use our best judgment in collecting ideas and suggestions from the appropriate people. For example, for specific questions regarding helpdesk requirements, the outsourcing vendor may need to be involved in order to correctly identify a requirement unknown to the Evaluator.

Service Level Agreement Document

The processes in creating the SLA are broken down to ease the management of the project. The first four sections require the input of ABC Company management and, in some cases, end-user surveys. The next group of tasks may require the input of the current out-sourcing company in order to ensure all requirements have been identified. Next, theevaluator assembles the data in a document that can be easily read and understood.

image002

The last sections take place during the negotiations with outsourcing vendors. Usually, a legal instrument will be created to bind both parties to a final Service Level Agreement. While the final SLA will be based upon the evaluator’s SLA document, it is likely that sections will be added or removed as negotiations dictate.

Contract Specifics and Context

Management of the SLA is a critical part of supporting end users. Before we can determine if objectives have been met, we must first identify metrics and the specifics of the contract.

Contacts and Role assignment

First, name the key contact to the Service Level Agreements and delegate SLA management tasks to others. Other contacts for the SLA include:

ABC Company:

Management

Exchange connection into other corporate systems

Management of 3rd Party Outlook/Exchange Development

Application Development

Remote and Mobile Access

Reporting

The frequency and detail of reports must be identified as well. Reporting can then be further broken down into two techniques:

  • Automated system reporting should be implemented in order to provide current and historical data. This data should be made available to the above named contacts on a regular basis. The methods for providing reports to the named contacts may include a secured website or electronic mail attachments. Hard copies of the reports may also be requested. The reports, for these contacts should be fully detailed reports with data analysis and a trend summary for the month. Moreover, historical data should probably be included.
  • It may be necessary for regional and divisional managers to receive a summary report/graphic depicting uptime and overall system performance once a month, similar to the graph depicted to the right.

ABC Company may also require that an automatic mechanism be put into place to notify the named contacts when critical performance thresholds are met. Specific thresholds are discussed later in the document.

Questionnaires and end-user canvassing methods should also be performed by ABC Company and/or the Outsourcing Company as part of an overall customer service initiative.

Finances

Payment terms and contract length are negotiated with the outsourcing vendor. ABC Company may prefer a contract length of six months, but will consider contracts as long as one year. Renewals can be handled in many ways including automatic six month extensions. Both ABC Company and the outsourcing vendor should be able to request a formal renewal meeting to update the SLA with riders and to negotiate new terms.

There are two types of terminations possible:

  1. Contract Termination- Indicates that either ABC Company or the outsourcing company elects to terminate the contract. A “Technology Transfer” and associated fee would probably be required in order to shift the maintenance and support to another group.
  2. Technology Termination- A termination in technology would occur when the support requirements are no longer required due to a shift in ABC Company technologies. This form of termination may or may not require a formal “Technology Transfer.”

Termination Options are described as follows:

  • ABC Company may reserve the right to cancel the contract for either termination option with 60 days notice to the outsourcing company. ABC Company understands that there may be financial penalties for “Contract Termination” if the SLA objectives were met by the outsourcing vendor. These penalties often reflect the fee for one month of support.
  • The outsourcing vendor may reserve the right to “Cancel Termination” with 180 days notice to ABC Company. A “Technology Transfer” fee would be charged to cover labor costs associated with transferring the knowledge and technology to another group.

 Review Process

There should be a formal review to evaluate the performance and customer service levels as well as staff reviews. A quarterly review is sometimes formalized in order to include discussions on SLA fulfillment, staffing and future projects that may affect the SLA.

Change Management

Service Level Management is accomplished by negotiating a change or additional to an existing Service Level Agreement. Out-of-scope or new projects need not be discouraged. A change process occurs during every review process and can also be instigated as needed. Several things could require a change or addendum to the existing SLA:

  • A change in the process workflow
  • Additional services
  • Missed performance or customer service thresholds
  • Additional third-party applications

Changes are not made directly to the SLA. Instead, contract riders are appended to the SLA until such time that the SLA is rewritten to incorporate the addendums. The SLA can only be written during a renewal cycle with both parties present.

Financial Incentive Plan

Most groups believe that the total cost of ownership (TCO) is more a function of cost of service and support of the system than a function of the cost of hardware and software. SLAs can drive down TCO by identifying damages for missed service levels.

In the case of ABC Company, a third party may be asked to provide evaluations to determine if service level objectives have been met. The costs associated with the third-party evaluations are the responsibility of the party requesting the evaluations.

Penalties and bonuses for SLA performance guidelines could be “paid” quarterly. Performance objectives are met based on a +10/-10 (percent?) allowance. Penalties are paid as a deduction of regular costs for the pay period immediacy following the review cycle. Bonuses are paid with four weeks of the review cycle and do not require a separate purchase order from ABC Company.

Performance Level Guidelines

Inter-site Message Transfers

Because the outsourcing vendor may have little control over the stability of the hub servers, ABC Company may not require guaranteed delivery times for mail originating from, or addressed to, any mailbox outside of the ABC Company’s site.

However, inbound Internet email with legitimate addresses should not get returned as undeliverable from the Exchange systems within the supported (environment?). The outsourcing vendor should remedy any internal Exchange process that returns mail.

Intra-site Message Transfers

ABC Company requires that intra-site Exchange mail be delivered to the recipient’s server-based mailbox within 15 minutes of delivery to a server within the supported site.

Remote Synchronization Performance

Off-line Address book

Remote users who replicate the Offline Address Book should never wait more than thirty minutes for a complete refresh to transfer even over slow connections.

Mailbox Replication

You should probably define the mailbox limits. In many cases, mailboxes are classified into two or more categories. For example:

  • ClassA Users have a 1GB limit on mailbox size
  • ClassB Users have a 5GB limit on mailbox size

Directory update frequency

Many companies configure directory replication so that the directory is current within a forty-eight hour time period. For example, a mailbox that is added at 3:00 p.m. on Tuesday must appear in the directory and Offline Address Book before 3:00 p.m. on Thursday. This should be defined in the SLA.

System Changes

Administrative tasks, such as Exchange/NT username add/remove/change, should be able to be handled within one business day. The primary and backup responsibility may be divided up among the server support team and the helpdesk.

An matrix of administrative task groups and responsibilities should be created in order to identify the ability of each group, including:

  •  Add/Remove/Change of mailboxes and distribution lists
    • One business day or less
  • Lync Account Creation
    • One business day or less
  • Exchange Connector settings
    • Two business days per request
  • Updating permissions and security settings a Group Mailbox
    • One business day or less
  •  Assigning permissions to a SharePoint Site
    • One business day or less
  •  Distribution List creation
    • Two business days or less
  •  Distribution List modification
    • One business day or less
  •  Mailbox restoration (from backup or snapshot)
    • Three business days or less

The outsourcing company may want to define the maximum number of one-day requests that can be filled per business day. Additional requests will roll to the next business day and will take priority over new requests.

Note: Public Folder applications that require or use automation, such as routing or scripting, are considered a separate project. New projects, which are likely to incur extra costs,  are not part of the SLA.

Uptime Requirements

System availability can be an expensive requirement. It is important that we identify the specific requirements from a resource access standpoint and not necessarily on a server by server basis. The specifics dictate the availability of the servers.

Network and remote access

Network connectivity between sites and for users should be defined clearly as to the required uptime. Redundant links may be required based on the connectivity requirements.

Mailbox Access

This specification details the amount of time a user cannot access his/her mailbox on an Exchange server in the supported site. Many companies define at least two types of mailbox classifications:

  • Class A users can be without access to their mailboxes for no more than six business hours. This group usually contains managers and key people within ABC Company
  • Class B users can be without access to their mailboxes for no more than 24 business hours. This group represents the bulk of the ABC Company Exchange users.

Service Availability

Information gathered from the previous specifications dictate the level of availability that is required. The services are then classified using the following availability classes.

SYSTEM-TYPE

 

AVAILABILITY CLASS UNAVAILABILITY MINUTES/YEAR SYSTEM AVAILABILITY
Unmanaged 1 50,000 90%
Managed 2 5,000 99.0%
Well-Managed 3 500 99.9%
Fault-Resilient 4 50 99.99%
High-Availability 5 5 99.999%

While the table represents system availability, it is important to note that the figures represent unscheduled down time. It is critical that “windows” are allowed for scheduled maintenance and upgrades. The down time is always be scheduled on the same day every week over the weekend. Many companies detail acceptable times during the weekend such as 11 p.m. Saturday to 2 p.m. Sunday. The specific time needs to be negotiated.

The maximum allowable scheduled down time per week for the Exchange systems should also be defined. For example, you could specify an eight-hour maximum window during the week and 25 hours one weekend per month.

You may want to send out a user survey in order to determine the best time for scheduled maintenance.

The outsourcing vendor must balance the uptime requirements with the inevitable cost. The foundation for a mission-critical architecture has specifications for server availability, data accessibility, data protection and disaster tolerance.

Equipment Support Requirements

Access and Security

ABC Company should require that named contacts be permitted physical access to the equipment at any given time. Moreover, overall access to the equipment must be secured and restricted. Access to the equipment must be available 24 hours a day, 7 days per week for the vendor’s support personnel.

Disaster Recovery Preparation

Backups/Snapshots

Many companies require that the clients be able to request a “recovery of deleted items” for up to 30 days of deleted items. Moreover, backups and snapshots must be maintained and also retired based on company retention policies.

In some cases, there may be a need to recover items from backup. The outsourcing vendor should honor such a request from any of the named contacts on the SLA. The outsourcing vendor may accept requests from the user community for restores, but should then verify the request with the named ABC Company contacts.

Monitoring

ABC Company does not have any specific requirements in regards to the types of systems (hardware and/or software) used to monitor the equipment.

Staffing

Certification and Experience

ABC Company requires that at least one person supporting the systems have current MCSE status. At least two of the support personnel must be certified on the current version of Microsoft Exchange Server.

Exclusive/Nonexclusive use

Your company may have specific requirements in respect to Security and Privacy which could mandate the need for exclusive use of equipment. You may therefore need to add a line that specifies that the resources that support the ABC Company Exchange systems be exclusive to ABC Company and not used for non-ABC Company projects or tasks.

Equipment

Brand/Vendor

While ABC Company has no requirements as to the brands or types of equipment used for the Exchange Server environment, ABC Company does require that the equipment be included on the Hardware Compatibility List for the current version of Microsoft BackOffice.

The outsourcing vendor is responsible for the requisitions and costs associated with all equipment necessary to support the ABC Company Exchange Systems.

Spares for testing/recovery

ABC Company requires that at least two entire servers be allocated for spare equipment and testing/recovery. The servers must match the current servers in production so that parts can be swapped and/or replaced. ABC Company further requires that the test equipment be updated as the production equipment and online at all times for testing.

Summary

There are not many templates like this out there since most hosting companies mandate the terms to the customer. However, there are Exchange shops out there that balk at a commodity hosting solution due to privacy and security concerns. My intention with this article was to lay some of that groundwork out and save some research and headache with SLA planning.