Technology used to create this post: Veeam Backup & Replication 9.5, VMware vSphere Web Client, Microsoft Windows Server 2016
The vast majority of corporations use Active Directory (AD) in their environment for centralized management and authentication. Active Directory provides multimaster replication, which means that you can set up several domain controllers that will synchronize data between every domain controller in a domain when a change has been made to any of those domain controllers. The Active Directory Clock is an incremental sync when it replicates between domain controllers. Veeam Backup & Replication notifies the other domain controllers that a restore is happening, and that the clock will be turned back to allow the newest changes can be synchronized between all of the domain controllers in the domain. To accomplish this, Veeam Backup & Replication uses native VSS integration to back up Active Directory, including NTDS Writer and DFS Replication Writer. That integration allows for a complete backup that can be restored properly without damaging other data in the AD JET database.
If you’re curious what VSS Writers are on one of your servers, this script will get all VSS Writers from <localhost> and output a grid:
$VssWriters = Get-VssWriters | Sort "WriterName" $VssWriters | FT -AutoSize # Displays the data on screen $VssWriters | Out-GridView # Displays the data in GridView
One challenge facing administrators/engineers is the AD backup and recovery process. I’ll outline the process that I use, the reasons for my choices, and the gotchas I’ve encountered along the way in this blog post. Some things to consider before you design your AD backup strategy:
- Your domain must be using DFS-R for SysVol replication. If you’re not sure you’re running DFS-R in your domain, you can execute this PowerShell command to check:
Get-Item -path "HKLM:\System\CurrentControlSet\Services\DFSR\Parameters\SysVols\Migrating SysVols"
If the returned key value is 0 (like what’s shown below), FRS is still being used. If the returned key value is 3, DFS-R is being used.
- You should understand which domain controllers hold your FSMO Roles.
Import-Module ActiveDirectory $Domain = "CONSTCO.COM" Get-ADForest $Domain | format-list SchemaMaster,DomainNamingMaster Get-ADDomain $Domain | format-list PDCEmulator,RIDMaster,InfrastructureMaster
- You should understand which domain controllers are acting as a Global Catalog. The Global Catalog is a distributed data repository stored at a domain level, which contains a searchable partial representation of every object in a multidomain Active Directory forest. The Global Catalog is distributed through multimaster replication. To understand which servers are acting as Global Catalog servers in your environment, run this PowerShell script:
Import-Module ActiveDirectory Get-ADForest | format-list -Property GlobalCatalogs
- If you are installing on a Windows Server 2008 R2 virtual machine that uses the VMXNET 3 network adapter type, be sure to install the patch mentioned in this VMware KB article. If you don’t install that patch, SureBackup and Instant Recovery will not work correctly and you might end up with a Local Adapter #2 or your IP address settings might not be retained when you use SureBackup or Instant Recovery.
Sequential Backup of Active Directory
Lets talk about one of this issues that you might see during a cluster backup, or any backup really. Your target might go offline, or encounter a problem during the backup. This is a major challenge with backing up Active Directory, because with VSS triggering at the same time as the backup things get tricky. AD is a type of cluster, so I wouldn’t recommend that you trigger the backup of your domain controllers at the same time. I’ve seen or heard about issues with broken SYSVOL DFS-R replication because the domain controllers were all backed up at the same time.
Thankfully, Veeam fixed that issue in Veeam Backup & Replication 7. As long as you’re running Veeam Backup & Replication 9.5 or above, which you should be, you won’t run into this issue. If you want to avoid backing up your domain controllers and AD simultaneously just to be safe, you have two ways to make that happen.
- You can disable Parallel Processing and put the VMs in the same job. (Parallel Processing is a job setting, so this will slow your backup job down considerably.)
- You can put the domain controllers in different backup jobs and schedule them at different times. (Staggering or chaining the jobs would accomplish this, but chaining is not best practice.)
For object-level recovery, you only need one copy of the NTDS.dit file. There’s no real need to backup all of your AD domain controllers every hour if you want to use them for item-level recovery.
Enabling Application-Aware Imagine Processing
The most crucial part of backing up Active Directory with Veeam is enabling Application-aware Image Processing in the backup job that contains your domain controllers. This will allow Veeam to trigger the VSS framework, which makes the backup consistent and puts everything into place to successfully restore your domain controller VM. If you do not use Application-aware Image Processing, when you restore data from your backup, old data can sync back to an existing domain controller, which is a very bad thing. Veeam will contact the guest VM one of two ways:
- Using Direct RPC calls to the VM. Veeam will try to contact the VM over the available network.
- Using VIX or the VMware Tools API. This is useful if you’re backing up a machine that is isolated in a DMZ network that cannot be contacted via the network, but is in your vSphere environment. (This method is only available when backing up VMware, because Hyper-V’s integration services do not have an API as of this writing.)
In any case, Veeam needs administrative credentials to the guest VM. If you’re using the VIX method and User Account Control (UAC) is enabled on the target machine, you need to specify an administrator with a well-known SID ending (*). An example of that could be <DOMAIN>\Administrator or <LOCALHOST>\Administrator. You can find more information here: http://www.veeam.com/kb1788.
When Veeam triggers a backup on the target machine, it will add runtime components that trigger the VSS framework. When it’s finished the backup, the runtime components are removed and the virtual machine is kept in a “clean” state. You can find this in your Event Logs, with the following Event IDs:
- 7036: The VeeamVssSupport service entered the stopped state.
- 1917: The shadow copy backup for Active Directory Domain Services was successful.
- 1104: The DFS Replication service successfully restarted replication after a backup or restore operation.
Recovery of AD Objects Using Veeam Explorer for Active Directory
I know we’ve gone over this before, but we’re doing it again because it’s important. AD is a multimaster database, which means that nodes can read, write and update at the same time. That means that if an attribute is changed on two domain controllers before replication occurs, a stamp containing the version, a date, and the domain controller will be used to resolve the conflict. If one domain controller claims to have a higher version, that version will win and the other changes will be overwritten.
What happens when you delete an object? Well, the simple answer is AD addresses this with tombstones. When you delete an object, it isn’t really gone. Here’s what happens in Active Directory when you delete an object:
- There’s an attribute on each AD object named isDeleted, and the value of that attribute is set to True.
- The object is moved to a hidden AD container called Deleted Objects.
- The Distinguished Name of the object is changed so that other objects will not cause a conflict.
- Most of the attributes are stripped from the object.
- The AD garbage collection process runs every 12 hours on each domain controller, and will remove deleted objects that are over 180 days old.
If you havent busted that 180 day window, objects can be restored through a process called tombstone reanimation. It’s a bit of a messy process, and only some of the attributes and values for the reanimated object will be restored. With a Windows Server 2008 R2 domain, the AD Recycle Bin was introduced. If your domain has the AD Recycle Bin enabled, deleting an object is now a two step process.
- The object is marked as deleted, and not stripped of its attributes.
- When the deleted object lifetime has expired, the object will go into a recycled state similar to the tombstone state, ready to be picked up by the garbage collection process.
Enabling the AD Recycle Bin feature requires a functional level of Windows Server 2008 R2, and once enabled the schema will be altered to the point where the AD Recycle Bin cannot be disabled. A small price to pay for this feature is that your database will be larger, as deleted objects will keep their attributes for a time. If you want to read more about the AD Recycle Bin and best practices, you can do that here: http://blogs.technet.com/b/askds/archive/2009/08/27/the-ad-recycle-bin-understanding-implementing-best-practices-and-troubleshooting.aspx
Restoring a Tombstoned Object Using Veeam Explorer for Active Directory
Veeam Explorer for Active Directory was introduced with Veeam Backup & Replication 8. This explorer simplifies object-level recovery for Active Directory, allowing you to pick and choose objects to restore, entire OUs to restore, and everything inbetween. This tool doesn’t just restore the domain controller VM in a virtual lab, but rather starts a file-level recovery and automatically mounts the NTDS.dit file I mentioned earlier. This file, together with the logs from AD, form the AD JET database. How do I restore objects from Active Directory, you ask?
- Launch the Restore Wizard from the Inventory section of the Veeam Backup & Replication console. Click Next.
- Select the radio next to Application items. Click Next.
- Choose Microsoft Active Directory from the list. Click Next.
- This should narrow your search to domain controllers, but if it does not you can search for one and select it. Click Next.
- Choose your restore point. Click Next.
- Choose the mount point for your temporary VM. Click Next.
- Enter a reason for the restore, or leave this page blank. Click Next.
- Click Finish. and the Veeam Explorer for Active Directory should launch and automatically mount a temporary VM and browse the AD database.
Once your database is visible and you can see your AD structure, you’ll be able to search for the object you want to restore by name, or by browsing to it directly. You can even narrow your search using LDAP filters, and by showing only the changed objects in your Active Directory. If you’re unsure about the object you want to restore, you can use the Compare with Production button and you can compare the version of the object that exists in your backup with the version of the object that exists in your production Active Directory. Restoring objects is pretty self explainatory.
- Click on the object or container you’d like to restore.
- Click the Restore Objects button on the Objects bar at the top of the window.
- Click Restore to… from the menu.
- Select your target Active Directory Domain Services server from the list, or provide a server name manually. To specify Global Catalog server, click the Cogs button on the right. If necessary, select Use SSL when connecting to the server.
- Specify the account to connect to the LDAP server. You can use the current account under which you are running Veeam Explorer for Microsoft Active Directory, or specify another account in the <Domain>\<Username> format.
- Click Next. Veeam Explorer for Microsoft Active Directory will verify the connection. If validation fails, an error message will be displayed. Check the connection settings and try again.
- Accolades
Scott is an experienced professional and recognized leader specializing in the design and implementation of enterprise virtualization, enterprise storage, server infrastructures and operations. A self-starter able to work both independently and in a team-oriented environment, Scott has a proven ability to create and automate solutions to improve productivity, reliability and performance.
Throughout his professional career, Scott has proven successes implementing technology and service improvement initiatives and has a demonstrated ability to think strategically about business, create technical definition around objectives in complex situations, develop solution strategies, motivate and mobilize resources, and deliver end-to-end technology solutions.
Scott has achieved multiple industry recognized certifications in virtualization, networking, cloud, storage, converged infrastructure, hyperconverged infrastructure and other technologies, and is honored to be a VMware vExpert in 2018-2022, and vExpert EUC 2020-2022.
Achievements
VMware vExpert EUC 2020-2022
VMware vExpert 2022
VMware vExpert 2021
VMware vExpert 2020
VMware vExpert 2019
VMware vExpert 2018
Double VCP – Data Center Virtualization & Desktop Mobility
Double VCP – Data Center Virtualization & Cloud Management
Double VCP – Data Center Virtualization & Network Virtualization
Double VCP – Network Virtualization & Desktop Mobility
Double VCP – Network Virtualization & Cloud Management
Double VCP – Cloud Management & Desktop Mobility
Dell Technologies Proven Professional Exam Developer – 2021
Certifications
VMware Certified Associate – Digital Business Transformation 2020
VMware Certified Professional – Desktop and Mobility 2020
VMware Certified Professional – Data Center Virtualization 2020
VMware Certified Professional 6 – Desktop and Mobility
VMware Certified Professional 6 – Cloud Management and Automation
VMware Certified Professional 6 – Network Virtualization
VMware Certified Professional 6 – Data Center Virtualization
VMware vSphere 6 Foundations
Dell Technologies Cloud Platform Administrator 2020
Dell Technologies Virtual Desktop Infrastructure 2019
Dell Technologies Specialist – Infrastructure Security Version 1.0
Dell Technologies Specialist – Systems Administrator, VxRail Appliance Version 2.0
Dell Technologies Specialist – Systems Administrator, VxRail Appliance Version 1.0
Dell Technologies Associate – Converged Systems and Hybrid Cloud Version 2.0
Dell Technologies and VMware Co-Skilled Associate – Converged Systems and Hybrid Cloud
IBM Enterprise Design Thinking Practitioner
IBM Enterprise Design Thinking Co-Creator
SolarWinds Certified Professional
