Wednesday, February 1, 2017

VDP (vSphere Data Protection) 6.1.2 fails to backup after upgrading vCenter 5.5 from Update 3c to 3e

Problem

You’ve recently upgraded vCenter 5.5 from Update 3c to 3e and immediately noticed that your existing VDP 6.1.2.19 appliance backup jobs fail and the closest KB you can find related to this issue is the following:

825Communication fails in vSphere Data Protection with vCenter Server 5.5 Update 3 (2146825)https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2146825

However, attempting to download VDPHotfix.sh, uploading it to the VDP appliance and applying the hotfix does not appear to work:

login as: root
Using keyboard-interactive authentication.
Password:
Last login: Wed Jan 25 12:20:34 2017
root@vdp1prd:~/#: cd /home
root@vdp1prd:/home/#: cd admin
root@vdp1prd:/home/admin/#: ls
0.0           autorestart  cert.pem  getnodelogs  gsan.out  key.pem    truncate
VDPHotfix.sh  bin          cgsan     gsan         hfsclean  timerange
root@vdp1prd:/home/admin/#: .VDPHotfix.sh
-bash: .VDPHotfix.sh: command not found
root@vdp1prd:/home/admin/#: ./home/admin/VDPHotfix.sh
-bash: ./home/admin/VDPHotfix.sh: No such file or directory
root@vdp1prd:/home/admin/#: ls
0.0           autorestart  cert.pem  getnodelogs  gsan.out  key.pem    truncate
VDPHotfix.sh  bin          cgsan     gsan         hfsclean  timerange
root@vdp1prd:/home/admin/#: ./home/admin/VDPHotfix.sh
-bash: ./home/admin/VDPHotfix.sh: No such file or directory

image

Attempting to upload the full 2146825_vdp-hotfix.zip file onto the appliance and using the tar -zxvf command to extract the file does not work either:

admin@vdp1prd:/tmp/#: unzip 2146825_vdp-hotfix.zip
Archive:  2146825_vdp-hotfix.zip
  inflating: 2146825_vdp-hotfix.tar
admin@vdp1prd:/tmp/#: tar -zxvf 2146825_vdp-hotfix.
2146825_vdp-hotfix.tar  2146825_vdp-hotfix.zip
admin@vdp1prd:/tmp/#: tar -zxvf 2146825_vdp-hotfix.tar

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
admin@vdp1prd:/tmp/#: sh VDPHotfix.sh
sh: VDPHotfix.sh: Permission denied
admin@vdp1prd:/tmp/#: sh VDPHotfix.sh
sh: VDPHotfix.sh: Permission denied
admin@vdp1prd:/tmp/#: ls

image

Solution

Prior to calling VMware support, the only solution that worked for this environment where I encountered the problem was the workaround mentioned in the following forum post:

VDP 6.1.2 appliance can't connect to vCenter after initial configuration
https://communities.vmware.com/thread/542344?tstart=0

The workaround involved modifying the following file in the directory /usr/local/avamar/lib:

mcsutils.pm

image

… and adding the following line highlighted in red in between the two lines in black:

. "-Dfile.encoding=UTF-8 "
. "-Dsecurity.provider.rsa.JsafeJCE.position=last "
. "-Dlog4j.configuration=file://$mcsvar::lib_dir/log4j.properties "; # vmware/axis

image

Once this workaround was applied, the backup jobs began to run and complete.

Knowing that this workaround probably wasn’t the best solution, we opened a support call with VMware then was asked to upgrade the VDP appliance from 6.1.2 to 6.1.3 and noticed that the upgrade does indeed fix the issue.  I hope this post helps anyone who may come across this issue as I did not see an official KB article indicating upgrading was the solution.

Tuesday, January 31, 2017

Task Scheduler Scheduled task fails with: "Last Run Result (0x1)”

Problem

You’ve created a new scheduled task in Task Scheduler to execute a batch file but notice that the task does not complete successfully and the Last Run Result is (0x1):

image

Reviewing the History tab of the scheduled task shows the following log entries:

Level: Information

Task Category: Action completed

Operation Code: (2)

General:

Task Scheduler successfully completed task "\Microsoft\Daily Certificate Expiry Notification" , instance "{ae87616f-a564-4c34-ab21-0a7f7cc2a99c}" , action "C:\Windows\SYSTEM32\cmd.exe" with return code 2147942401.

image

Level: Information

Task Category: Task completed

Operation Code: (2)

General:

Task Scheduler successfully finished "{ae87616f-a564-4c34-ab21-0a7f7cc2a99c}" instance of the "\Microsoft\Daily Certificate Expiry Notification" task for user "NT AUTHORITY\NETWORK SERVICE".

image

Solution

One of the possible causes to this issue is if the Start in (optional): of the configured action is not filled out. To correct this, open the properties of the scheduled task, navigate to the Actions tab and then edit the action:

image

Notice that the Start in (optional): field is not filled in:

image

Fill in Start in (optional): with the path to the batch file:

image

With the above in place, the job should now run with the Last Run Result as The operation completed successfully. (0x0):

image

Filtering out certificates with blank Common Name when using the Certificate Expiration Alerting command tool

I’ve found that many of my clients with services that rely on Microsoft Certificate Authorities deployed within the internal network have frequently asked me whether there was a way to monitor the expiry of these issued certificates and the answer to that is yes, with the Certificate Expiration Alerting tool found here:

Certificate Expiration Alerting
https://blogs.technet.microsoft.com/nexthop/2011/11/17/certificate-expiration-alerting/

The next common question that usually pops up shortly after testing the tool is whether there was a way to filter out issued certificates that have blank common names as shown in the following screenshot:

CertExpAlerter.exe -c "cert01\Company-CA" -d 312

image

Note that the command above queried for certificates that expire in 312 days and 3 certificates were returned where 2 had blank common names.  The way to filter the common name as described in the TechNet article is with the use of RegEx and the only reason why I am familiar to it is because I used to work with Lync Enterprise voice quite a bit which forced me to learn it for creating translation rules. The RegEx expression we’re interested in is the following:

^(?!\s*$).+

What the above RegEx command matches is any string that contains at least one non-space character which results with the exclusion of blank common names:

image

Hope this helps anyone who is unfamiliar with RegEx and is looking for the expression to filter out blank common names.

Monday, January 30, 2017

Attempting to delete an Exchange Server 2016 mailbox database previously used to migrate public folders from Exchange 2010 throws the error: “This mailbox database is associated with one or more active PublicFolderMailboxMigration requests…”

I recently had to assist a client with migrating their Exchange Server 2010 public folders to Exchange Server 2016 and ran into a situation where the mailbox database storing the public folder mailboxes would throw the following error message when I attempt to delete it via the GUI or PowerShell:

This mailbox database is associated with one or more active PublicFolderMailboxMigration requests. To get a list of
all PublicFolderMailboxMigration requests associated with this database, run Get-PublicFolderMailboxMigrationRequest |
?{ $_.RequestQueue -eq "<Database ID>" }. To remove a PublicFolderMailboxMigration request, run
Remove-PublicFolderMailboxMigrationRequest <Recipient ID\Request Name>.
    + CategoryInfo          : InvalidOperation: (empfdb01:DatabaseIdParameter) [Remove-MailboxDatabase], AssociatedMRS
   RequestExistsException
    + FullyQualifiedErrorId : [Server=BMEXMB01,RequestId=65decc2b-2925-48cf-91d0-aba27ab1329f,TimeStamp=1/30/2017 1
   :20:19 PM] [FailureCategory=Cmdlet-AssociatedMRSRequestExistsException] A32E1F9E,Microsoft.Exchange.Management.Sys
  temConfigurationTasks.RemoveMailboxDatabase
    + PSComputerName        : bmexmb01.contoso.com

After ensuring that there were no Public Folder Mailbox Migration requests by executing the Get-PublicFolderMailboxMigrationRequest cmdlet, I did a quick search on the internet and found the following post:

https://social.technet.microsoft.com/Forums/office/en-US/54fd0db4-11d3-421c-92e8-d4050338a907/trouble-removing-2016-mailbox-database?forum=Exch2016Adm

… where a person indicated that there may be a lingering object in the Configuration container that is not returned by the PowerShell cmdlets.  I then went ahead and launched ADSIedit, connected to the Configuration container then browsed to:

Configuration > Services > Microsoft Exchange > ExchOrganization > Mailbox Replication displayed the following:

image

The node that appeared to be out of place was the one named:

CN=PublicFolderMailboxMigrationRequestsCNF:11714980-d0a3…

Browsing into that node displayed the following items:

image

Opening these items showed that they were objects that corresponded to the day I had started the migration batch for the public folder migration so I went ahead and deleted these items in the folder then forced replication via repadmin /syncall /AdeP on a domain controller and was then able to delete the mailbox database.

Sunday, January 29, 2017

Attempting to upgrade VMware vSphere Data Protection from 6.1.2 to 6.1.3 displays: “To upgrade your VDP appliance, please connect a valid upgrade ISO image to the appliance.”

Problem

You’re attempting to upgrade your VDP appliance from 6.1.2 to 6.1.3:

image

… but notice the following message after mounting the upgrade ISO and navigating to the Upgrade tab:

To upgrade your VDP appliance, please connect a valid upgrade ISO image to the appliance.

You’ve confirmed that the upgrade ISO is not corrupted and can see the files in the file when using utilities such as WinRAR to browse it.

Solution

A quick search on the internet appears to suggest that this has been a problem since version 6.1 of the VDP appliance and the way to get around this issue is to manually mount the ISO via commands through an SSH session.  The following are two posts that I found helpful:

VMware vSphere Data Protection – Upgrade 6.1.2 to 6.1.3 ISO not detected
http://www.stephenwagner.com/?p=1107

ISO Package Not Available During VDP Upgrade From 6.1
http://www.virtuallypeculiar.com/2016/12/iso-package-not-available-during-vdp.html

My first attempt to resolve the issue was to use the first post but while using VI to edit the /etc/auto.mnt file to mount the ISO worked to get the ISO mounted for the install, it was too cumbersome to repeat during the install when you had seconds to remount the ISO because it gets dismounted.  The single line command supplied in the second post was much easier.  The following are steps to perform the upgrade:

Step #1 – Backup VDP Appliance

It is extremely important to backup the appliance before performing the upgrade appliance as every failed upgrade I experienced rendered the appliance unusable so begin by shutting down the VDP appliance and change ALL the disks aside from Disk 1 to Dependent – Dependent disks are included in snapshots:

image

Once the disks have been changed to Dependent mode, snapshot the virtual machine to create a rollback point.

Step #2 – Power on VDP Appliance and Mount ISO

Ensure that the upgrade ISO is uploaded to a datastore to avoid an upgrade failure due to mounting the ISO through a remote console and either the desktop you’re upgrading with crashes or vCenter gets restarted causing the ISO to disconnect from the VM.

SSH to the VDP appliance’s IP address and execute the command:

df -h

Note the lack of the device /dev/sr0 in the screenshot below:

image

Mount the attached ISO by executing the command:

mount /dev/sr0 /mnt/auto/cdrom

Note the confirmation:

mount: block device /dev/sr0 is write-protected, mounting read-only

Executing df -h will now display the line item:

/dev/sr0 5.1G 5.1GB 0 100% /mnt/auto/cdrom

Browsing the directory with the commands:

cd /mnt/auto/cdrom

ls

Will display the contents of the upgrade ISO:

vSphereDataProtection-6.1.3.avp

version.properties

image

Step #3 – Start the Upgrade

Viewing the VDP Upgrade tab on administration console at https://<VDPapplianceIP>:8543/vdp-configure/ will now display the following:

Package verification in progress. This may take a few minutes…

image

The page should display the upgrade package after a few minutes:

VSphereDataProtection7280

Status: ready

Priority: normal

Version: 6.1.3.70

image

Start the upgrade:

image

image

image

Step #4 – Prepare to remount ISO at 71%

This step is extremely important because it takes quite a bit of time for the upgrade process to reach 71% which will eventually dismount the ISO.  Failure to remount the ISO within the 10 second span would cause the upgrade to fail and the appliance to be unusable requiring you to revert back to the previous snapshot.

Copy the following command to prepare to remount the ISO:

mount /dev/sr0 /mnt/auto/cdrom

Monitor the progress bar and as soon as it reaches 71%:

image

… execute df -h to determine whether the ISO has been dismounted and as soon as it has, execute the command to mount the ISO.  Note that the command may appear to fail the first few times but keep repeating the execution and you will eventually mount the ISO:

mount: mount point /mnt/auto/cdrom does not exist

image

The installation should continue to 72% if you have mounted the ISO in time:

image

At around 84%, the console would go back to the login page allowing you to log back in showing services are stopped and ISO not attached in the Upgrade tab but executing df -h in the SSH you have opened indicates it still is.  Reviewing the console of the VDP doesn’t show any changes:

image

Wait a bit longer and you will eventually get kicked out again and this time logging in will show more services started with the ISO attached again.  vCenter should show tasks performed on the as well:

image

From this point, give appliance maybe 10 more minutes just to be safe and proceed to reboot it.  You should be able to successfully connect to the appliance via the vSphere Web Client and see the version is now 6.1.3 in the console:

image

image

image

Step #5 – Verify VDP and delete snapshots

Proceed to verify that VDP appliance to ensure that it operates as expected, then proceed to delete the snapshots and change the disks back to Independent – Persistent.

Saturday, January 28, 2017

Attempting to generate a certificate request (CSR) on the Avaya Session Border Controller for Enterprise fails with the error: “The Subject Alt name field can not be empty.” or “Subject Alt Name is not properly formatted. See here for more information.”

Problem

You attempt to create a CSR on Avaya Session Border Controller for Enterprise 6.3 000-19-4338:

image

image

image

… but notice that you are unable to generate the CSR if the Subject Alt Name is left blank:

The Subject Alt name field can not be empty.

image

Attempting to fill it in with the Common Name fails with:

Subject Alt Name is not properly formatted. See here for more information.

image

Solution

The reason why the above 2 errors are thrown is because the entry is not supposed to be the FQDN as most Windows administrators are used to but rather an IP address and SIP domain.  The Avaya Session Border Controller should have 2 interfaces, 1 for internal and 1 for external so depending which interface this CSR is for, the format should either be:

IP:1.1.1.1,DNS:external.domain.com

… or:

IP:10.1.1.1,DNS:internal.domain.com

**Note that the certificate for the public interface should be have the public address accessible from the internet and not the internal NAT address.

The CSR should complete once this has been corrected:

image

image

Thursday, January 26, 2017

Attempting to execute Create-PublicFolderMailboxesForMigration.ps1 to create target public folder mailboxes on Exchange 2016 fails

I’ve noticed that many of my clients and colleagues have been calling me about a specific step during the migration of Exchange 2010 Public Folders to Exchange 2016.  Given the frequency of the calls I’ve gotten, I thought it would be a good idea to write this short blog post about it.

Problem

You’re in the process of migration from Exchange Server 2010 to 2016 and have gotten to the public folder portion of the migration.  The following TechNet article is what you are using for the migration:

Migrate public folders from Exchange 2010 to Exchange 2016
https://technet.microsoft.com/en-us/library/mt463355(v=exchg.150).aspx

You’ve been able to execute all of the steps but notice that as you approach Part 4 and attempt to execute Create-PublicFolderMailboxesForMigration.ps1 to create target public folder mailboxes on Exchange 2016:

image

… the cmdlet fails with the following error:

[PS] C:\PFMigration>.\Create-PublicFolderMailboxesForMigration.ps1 -FolderMappingCsv PFMailboxes.csv -EstimatedNumberOfC

oncurrentUsers:100

C:\PFMigration\Create-PublicFolderMailboxesForMigration.ps1 : Existing Public Folder deployment is not locked for migra

tion. The script cannot continue unless all Public Folder mailboxes are deleted first. Please, make sure the existing m

ailboxes have no data before deleting them.

At line:1 char:47

+ .\Create-PublicFolderMailboxesForMigration.ps1 <<<< -FolderMappingCsv PFMailboxes.csv -EstimatedNumberOfConcurrentUs

ers:100

+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException

+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Create-PublicFolderMailboxesForMigrati

on.ps1

[PS] C:\PFMigration>Get-PublicFolder

image

Solution

One of the most common reasons I find administrators encounter this error during the public folder migration process is because they are executing the Create-PublicFolderMailboxesForMigration.ps1 cmdlet on the Exchange 2010 server.  All the steps prior to Part 4 is usually executed on the Exchange 2010 server so some administrators forget that they need to execute this cmdlet to create new Exchange 2016 public folders on the new Exchange 2016 Management Shell instead:

image

Executing the Create-PublicFolderMailboxesForMigration.ps1 cmdlet should complete successfully and output the following:

PS] C:\PFMigration>.\Create-PublicFolderMailboxesForMigration.ps1 -FolderMappingCsv PFMailboxes.csv -EstimatedNumberOfC

oncurrentUsers:100

Do you want to run software from this untrusted publisher?

File C:\PFMigration\Create-PublicFolderMailboxesForMigration.ps1 is published by CN=Microsoft Corporation, OU=MOPR,

O=Microsoft Corporation, L=Redmond, S=Washington, C=US and is not trusted on your system. Only run scripts from trusted

publishers.

[V] Never run [D] Do not run [R] Run once [A] Always run [?] Help (default is "D"): A

Creating a new session for implicit remoting of "Get-OrganizationConfig" command...

Public Folder mailbox updates.

Creating 5 Public Folder mailbox(es) and updating 0. Total mailboxes to serve hierarchy will be 1. Would you like to

proceed?

[Y] Yes [N] No [?] Help (default is "Y"): Y

Total mailboxes created: 5. Total mailboxes updated: 0. Total serving hierarchy: 1.

Here is a list of Public Folder mailboxes created:

Name IsServingHierarchy IsMigrationTarget

---- ------------------ -----------------

Mailbox1 False True

Mailbox2 True True

Mailbox3 False True

Mailbox4 False True

Mailbox5 False True

[PS] C:\PFMigration>

image

Quite obvious when you realize it but we all tend to get a bit lost when we’re too immersed into a deployment at times.